IR-CM: The Fast and General-purpose Image Restoration Method Based on Consistency Model
Authors: Xiaoxuan Gong
Jie Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Image restoration
Image enhencement
SDE-based model
Consistency model
Primary area: diffusion_based_models

Abstract: This paper proposes a fast and general-purpose image restoration method. The key idea is to achieve few-step or even one-step inference by conducting consistency distilling or training on a specific mean-reverting stochastic differential equations. Furthermore, based on this, we propose a novel linear-nonlinear decoupling training strategy, significantly enhancing training effectiveness and surpassing consistency distillation on inference performance. This allows our method to be independent of any pre-trained checkpoint, enabling it to serve as an effective standalone image-to-image transformation model. Finally, to avoid trivial solutions and stabilize model training, we introduce a simple origin-guided loss. To validate the effectiveness of our proposed method, we conducted experiments on tasks including image deraining, denoising, deblurring, and low-light image enhancement. The experiments show that our method achieves highly competitive results with only one-step inference. And with just two-step inference, it can achieve state-of-the-art performance in low-light image enhancement. Furthermore, a number of ablation experiments demonstrate the effectiveness of the proposed training strategy. our code is available at https://github.com/XiaoxuanGong/IR-CM.

IR-CM: The Fast and General-purpose Image Restoration Method Based on Consistency Model
Xiaoxuan Gong
School of Artificial Intelligence and Automation, HUST Huazhong University Science and Technology
Wuhan, China
gongxiaaxuan22860gaa com
Jie Ma*
School of Artificial Intelligence and Automation, HUST
Huazhong University Science and Technology
Wuhan, China majie majieehuut.edd.crr edu. cn
Abstract
This paper proposes a fast and general-puree seneral-purpose image restoration method. The key idea is to achieve few-step even one-step inference by conducting consistency distilling training on specific mean-reverting stochastic differential equations. Furthermore, based on this, propose novel linear-nonlinear decoupling training strategy, significantly enhancing training effectiveness and surpassing consistency distillation on inference performance. This allows our method to be independent of any pre-trained checkpoint, enabling to serve as an effective standalone image- to-image transformation model. Finally, to avoid trivial solutions and stabilize model training, we introduce a simple trigin-guide loss. To validate the effec- tiveness of proposed method, conducted experiments tasks including image deraining, denoising, deblurring, and low-light image enhancement. The experiments show that method achieves highly competitive results with only one-step inference. And with just two-step inference, it can achieve state-of-the-arr performance in low-light image enhancement. Furthermore, number of ablation experiments demonstrate the effectiveness of the proposed training strategy. our code is available https:://// //:::://itiie git..b........ com/ Xiw.xuanGoogg///////a/aa// IM-CM.
Introduction
Image restoration is classic problem in the field of computer vision. It aims to transform low-quality or noisy images into their corresponding high-quality or noise-free counterparts. In many industrial applications (such autonomous driving), there are various complex types of image degradation, including rain and fog interference, glare interference, low-light conditions, and motion blur etc. This necessitates high generality in image restoration algorithms. Moreover, there is often high demand for real-time performance in practical application scenarios, posing even greater challenges to the design of algorithms and models.
Common image restoration tasks include image deraining[ 2, 3, 4, 5, 6], denoising[7, 8, 9, 10, 11], and 1eblurring[12, 13, 14, 15] etc. Due to the demands of industrial applications, research low-light image enhancement 16, 17, 18, 19, 20, 21] is also gradually increasing. However, these methods are often heuristic and difficult apply to general tasks because they typically require domain-specific prior knowledge for training. A more general approach is the recently proposed IR-SDE[22), which
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

does not rely on any prior knowledge. It only requires pairs of Hwwpiiooooooooouuuaiotiuaa images for training and achieves good results across multiple image restoration tasks. However, like most diffusion-basee methods, it requires multi-step sampling for inference, making it difficult to meet the real-time requirements of practical applications.
In recent years, diffusion models have achieved remarkable results in both unconditional image generation[23, 24, 25, 26] and conditional image generation tasks[27, 28, 20, 29, 22, 30, 21, 31. 32]. In more general description, diffusion models can be described using stochastic differential equations (SDEs)[33, 34]. achieves this by converting the original data distribution into a fixed Gaussian distribution, then learning the corresponding distribution gradients (score) through a network, and finally gradually reconstructing the original image using SDE solvers or ODE solvers. Most SDE- based methods can generate high-quality samples, however they often require many steps of sampling to accomplish this. Despite the existence of many methods for accelerating sampling[25, 35, 36), it remains challenging to meet the real-time requirements in industrial applications. Recently, the consistency m3del[37] has been proposed. aims to map any point on the ODE trajectory of the SDE-based model to its origin. Once trained, the model can achieve one-step inference with some decrease model performance. The training of consistency models is divided into two approaches: consistency distillation(CD) and consistency training(CTT), The former involves distilling training based on the ODE trajectories of a pre-trained SDE model. The latter, while not requiring a pre- trained SDE model and can be considered as an independent generative model. typically exhibits lower performance compared to the former[37]. This means that training consistency models often rely on pre- pre-trained models, which inevitably leads to performance degradation.
The purpose of this paper to design a universal image restoration model with fast inference. The proposed model, named IR-CM (Image Restoration Consistency Model), achieves one-step or few- step fast inference through consistency model training. Due to its versatility and flexibility in different image transformation tasks, the IR-SDE serves as the foundation for method. IR-CM can be trained using consistency distillation (CD) on a pre-trained model, but at the cost of a slight decrease in performance. Therefore, we focus more on consistency training (CT), enabling IR-CM to become an independent image restoration model without relying on any pre-trained checkpoint. Furthermore, to improve the effectiveness of CT, we propose novel Linearnoonlinet decoupling training strategy and a novel origin-estimated consistency function, these allows the model's performance to reach or even surpass that of the original model. Finally, we propose simple origin-guided loss stabilize the training process. In summary, the main contributions of our method are as follows:
We propose universal and fast image restoration method that can obtain high-quality images with one step or few-step sampling. For different tasks, training only requires dataset replacement, without the need for any additional prior knowledge.
We have introduced a novel linear-nonlinear decoupling training strategy, enabling our method to achieve even surpass the performance of the original model without relying on any pre-trained checkpoint.
We propose novel ergtineestimated consistency function, which enables our model have a more stable initial state and a smaller solution space, and simple origin-guided loss to stabilize the training process. This makes our method more robust. The ablative experiments demonstrated its effectiveness.
Our method achieves highly competitive performance in multiple tasks (including image deraining, denoising, deblurring. and low-light image enhancement) with one-step inference. With two-step inference, our method achieves state-of-the-art performance in low-light image enhancement task.
2 Preliminaries
2.1 Mean-reverting stochastic differential equation
Our method requires a SDE-based model as the base model for consistency training. Specifically, we choose IR-SDE[22] as the base SDE model due to its excellent generality and applicability. Its forward process involves gradually transforming high-quality images into corresponding low-quality versions with fixed-varaance Gaussian noise. can be represented follows:
dx 0r(p x)dx o,dw,
(I)

---

where E R typically represents the low quality image, and r(0) E R represents its corresponding high-quality version, dr, o, are timedeependent positive parameters, and they satisfy of /0, = 2X² for all times i with positive constant X, and dw represents Brownian motion. In [22], it has been proven that at each time t, the marginal probability distribution of can be represented as follows:
(I) = -x((t)) my, vt),
(2)
mc =- 1 + (a<(0) 11)e
1/- = 121 - e -6.)..)
where 0, fo As increases gradually, me approaches and approaches x² Thus, z²0) (the high-quality image) will gradually transform into its corresponding low-quality version accompanied by Gaussian noise with variance X².
With the above conclusion, we can naturally sample xx()) by x=(t) = m, + /rn.. Then we can train network to estimate the noise 6.. In the inference phase, the reverse-time process of the IR-SDE can be represented as fo3]o
1 oo. V. log p( dt. dr 0:(1 2) -)) 20
(3)
xr(t) -me
V loggogooooo Pc(x)
Ut E(x, p4, 1)
02
where V (x)) is called score function and é(,, nt is the noise estimated by network. Then, similar to other SDE-based models, we can adopt a SDE-solver (or ODE-solver) to reverse the process to restore the low-quality image x(t) back to the high-quality version =00) progressively.
2.2 Consistency model
For a solution trajectory (*t (e of any PF-ODE such as (3), a consistency function can be defined as f(z,, t) = Ty where is a small positive number. This means that when sampling any pair (x,, t) on the trajectory of the PF-ODE, the output of the consistency function is always the initial point Jn of the trajectory. This property referred to as self-connisteney(37)] The example of the consistency function in [37] is as follows:
fol t) Cskip (f)ar + Cout Coue Fo
44
where Cakip (t), Cour are differentiable functions, and they satisfy Cskip( 1, Cout (are,) t) is a trainable network initialized by a pre-trained noise estimation model. Once training is complete, we only need to imput zy and apply (Fr, T) = F, to obtain high-quality sample in one step. There are two training methods for consistency models:
Consistency Distillation (CD) For discrete time sequence I = < 12 ... <IN T, given an arbitrary point (Ie (n41) on PF-ODE trajectory, we can estimate the xt., by following formula:
$ I1, + (fn tn+ 1) (I in+lip)) ip),
(5)
where (-)) represents the update function of one step ODE solver applied to the PF-ODE, and is the weights of pre-trained score matching network. Then the CD loss can represent as follow:
LCD(0, o ; :p = [X(t,)d(fa (I tn+1): fo- (#f, tn))
(6)
where ot represents the exponential moving average (EMA) version of training weights d. it is frozen during backward, and A(-) is a positive weighting function, d(-) denotes a distance function, such as the LI L2 distance. This approach essentially involves distillation training on the pre-trained score matching network, hence referred to as consistency distillation.
Consistency Training (CT) Unlike CD, the CT does not rely pre-trained trained score matching network and can independently train any SDE-based model. The CT loss is represented as follow:
Lor(Ø, x7) == EE (d(o fn41), fo (It., tx))]
(7)
Here It,, and Za,t1 both sampled from the forward process of SDE model.
3

---

3 Method
The key idea of our method lies in employing our proposed two-stage training strategy and the origin-estimated consistency function to conduct consistency training the IR-SDE model. We thus refer to it as an Image Restoration Consistency Model (IR-CM (RR-M)).W. We begin by describing the novel origin-estimated consisteney function, followed by an explanation of the proposed two-stage training strategy and the origin-guided loss. The overall architecture is depicted in Fig 1.
3.1 Origin-estimated consistency function
For any SDE-based model, each step of the reverse process is essentially a prediction of the PF-ODE solution[34] In other words, any point on the PF-ODE trajectory actually contains information about the origin. Based on this, according to equations (2) and (3), the origin prediction function at each point on the PF-ODE trajectory of the IR-SDE is follows:
a()) (11 e c (ar(t) (x,
(8)
During the consistency training phase(both for CD and CT), differ from the formula (4) proposed in [37], we train using the following origin-estimated consistency function (OECF):
So(# 1) = Cokiip Cokip( (1)=((() + Cout ( L; (),),
where the 2o (2,, L; Φ) is obtained by (8), Suppose the pre-trained score model matches the ground truth, i.e., Vi [m, T (2, μ4, 1) = (t)) + o(Di) and (T) 0, Cour Cour(T) = For formulas (4) and (9), Calép(!), xr),, and (l) are all same and constant at each time t. Thus, for simplicity in analysis, we specifically consider the moment = T. Then substituting equation (2) into equations (4) and (9) respectively, obtain:
fo(ET, 7) = =000++ x (T) mr (o) + o(DI);
(10)
Vor
f0OECr T) (0) eT uro Att).
(11)
From equation (10), it can be observed that if equation (4) is chosen the consistency function, the initial solution will have a significant fixed error. In contrast, using the OECF, the initial solution will only have time- deeedeeenddnt higher-order infinitesimal error. Therefore, for a pre-trained score matching model, the OECF offers more stable initial state and hence smaller solution space for all time t.
Intuitively, OFCF can effectively enhance the training performance of CD. In practice, we have found that for CT, OECF also exhibits significant performance improvements compared to equation (4). The relevant ablative experiment results will be presented in Chapter 4. Note that, unlike CD, the pre-trained model in CT is only used to initialize the training weights and is not involved any training process. In practice, training with randomly initialized weights is also feasible, albeit usually resulting in slightly longer convergence times.
3.2 Origin-guided loss
In general training, we randomly sample (x,, t) and (ze41, + 1) using equation (2), and then simply apply (6) for CD or (7) for CT, as illustrated in the lower half of Figure 1. However, we have empirically found that its performance is not ideal, and occasionally, pattern collapse occurs. Upon further analysis, we discovered that this often occurs when t is not sufficiently small during the random sampling in the early stages of training. This also leads to the emergence of mediocre solutions. Below we provide simple theoretical proof.
Theorem 1. Let Cout (t) be monotonic differentiable and satisfy Cout (1) 0, Cout (T) consider (9) as consistency function. When t > 17, for any E gd there always exists a E μ,, t) such that fo( t) y-
Proof. Recall (9), there is
y (t)x(() + Coul (t)
(12)

---

Linear Fitting Stage: =00)-~
NQ image
x(0)
Image noising process
.(F))
7a 0)
image
140%, Can (914 C..
Nonlinear Fitting stage::(0); image
d(o1) oi xidt a die
x(O)
Image degradation process
x(F)
_(0044))
Tels
Teloy.
Image
nolie
Ta(k, Gaye (x(0)) jor (R9d) Rot
Figure I: Two-stage training strategy, where is set the low-quality image to be restored. In Stage One, by setting z(0) = μu, the forward SDE (1) actually degrades into simple noise addition process, upon which consistency training is conducted x(0) is set as the high-quality image and gradually transformed into the corresponding low-quality image with fixed level of noise by (1). Consistency training is then performed each intermediate state (t).
2=(t) with different levels of noise. In Stage Two,
then substitute (8) into (12), we obtain:
Eo (Ti, 14,t) e - (skip(t)x(()
1 (e 01
xxt) Ux
(13)
(1)o
Note that when & 7). both V; and 02 are greater than O, the right-hand side of equation (13) is non- singular. This implies that under loss functions (6) and (7), the consistency function admits arbitrary non-singular solutions, leading pattern collapse. In other hand, (6) and (7) only emphasize the self-consistency between any two points on the PF-ODE trajectory. leading to a lack of determinism in the training process. This uncertainty indirectly contributes to slower convergence of the model.
To address this issue, we additionally introduce following ingiinggiide(OGG) loss to stabilize training process.
Log = Elllfo + x00)
Apere [f + 1))
(x(()))
(14)
where Apere positive constant, and (-)) represents VGG16[38] feature extractor from 2nd and 5th pooling layers. This is equivalent to performing a consistency computation with the origin loss. This effectively avoids the
after each random sampling. hence referred as the emergence of mediocre solutions and adds some certain determinism to the training process, resulting faster convergence. Then the final loss function represented follows:
Lfult LCD/CT \oGLoG,
(15)
where ÀoG is positive constant. The selection of ÀoG will be discussed in the ablation experiment section of Chapter 4.
3.3 Linear-nonlinear decoupling training strategy
With the foundation laid in Sections 3.1 and 3.2, we can naturally train IR-CM by regular CD or CT process, as illustrated in the lower half of Figure 1. Despite this method offers convenience in training IR-CM, we have empirically found that its performance is not optimal in practice. Let us recall the SDE (2) and OECF (9), we obtain:
fo(re,
Caksi (t)p + (t) [=((0) u)e [
Cakiip(t Coue (t) (16)
linoar state f1
noulinear intermediate state
noise fa
5

---

Algorithm 1: Linear-fitting stage
Algorithm 2: NonLinear-fitting stage
Input dataset D, model parameter e OFCF
Input dataset 'D, model parameter Φ, OFCF
fo(s)), learning rate ç. OG Weight AoG; t o:
fo(:.), learning rate &. OG Weight Xog:
+ Φ:
while not convergence do Sample (ILQ-)HQ) ~ DD and
while not convergence do
Sample (ILQ.HHQ) D and ~ [[, 1];
n ~~ U[1, - -];
1 + ILQ. =00) + ILQi
1 -- ILQ. x(0) 6 yuQi
Sample 2t, ~ ~N(men Vtn) and
Sample 2t, ~ ~N(m,, Vt,) and
Itnts ~ (M 1na41
Finti ~ ~N(mon+1> vl.41)5
Ciuu(0,
Ciuu(o.0
Cfr(folIa, tn), fn) folotners bnt1)) + DoG Lo
Ccr Ccr(fo (z,, tn) fo (xt1)) +
@ + Φ EVCiul(
+ stopgrad(0))
Φ + stopgrad (6)
end
end
In general training, the model attempts to simultaneously fit the variations of both linear part fi and nonlinear part f2. Obviously, this is more challenging than fitting fi alone. In practice, the model's s performance at larger values of ís more crucial during training, because any intermediate state for v(), E [n. ?) are unknown during inference, thus we can only set / = T for inference. And note that there is lim,-r f2 0, therefore, the influence of f2 is negligible when / = T. Based on the analysis above, we can then eliminate f2 for all by simply setting xr (0) as μ. By doing so, the original image degradation process is transformed into purely image noising process (as shown in the upper half of Figure 1). Since only the linear part fi and noise fa are being fitted, the model will achieve better performance over the entire PF-ODE trajectory, leading to improved inference performance as well. We refer to this training as the linear- linear-fitting stage and the corresponding pseudocode is shown in Algorithm 1.
Despite the Linear-fitiing stage achieves good performance for one-step inference after training, we can still further improve the performance by employing appropriate multi-step sampling. However, the model trained in the finear-fitting stage cannot perform multi-step sampling inference because the outputed HQ image is not the origin =0) f of PF-ODE trajectory, making it unable to estimate any intermediate states Te after one inference. Therefore, after the Linear-fitting stage, we set (0) to be the HQ image instead of µe and fine-tune the model (Algorithm 2). Since in the previous stage, the model has already fitted/ f1 and fs. this stage mainly focuses on fitting the non-linear part f2- Therefore, this stage is naturally referred to as the com-Linerr-fitinng stage. After training in the non-lineat stage, the model's 's performance at t = T is almost unchanged, meaning there is no change in the performance of one-step inference. But the model will be able to perform multi-step sampling inference (Algorithm 3 in appendix C). In practice, even just two-step sampling inference brings noticeable performance improvement. Relevant comparative experiments and ablative experiments will be presented in Chapter 4.
4 Experiments
4.1 Comparative Experiment
In this section, we will validate the effectiveness of our proposed method on five tasks: image de-raining, image denoising, image deblurring, low-light image enhancement, and nighttime glare removal. The implementation details are described in Appendix A. Specifically, we will compare our method with some of the state-of-thed ate methods on PSNR, SSIM[39], LPIPS[40] metrics and NFE. The NFE (Number of Function Evaluations) refers to the number of function evaluations required to generate an image or data. In other words, it is the number of evaluations needed at each step of the diffusion process. Notably, like other SDE-based models, we prioritize perceptual scores LPIPS over distortion scores PSNR and SSIM. And our metric settings are same as other mentioned baseline methods. For PSNR metric, we perform the calculation in the luminance space (Y channel). For SSIM metric, it refer to [39], and for LPIPS metric, it refer to [40]. All comparison experiments were

---

input
JORDER
MAXIM
PReNet
Restormer
IR-SDE
Ours
Figure 2: Qualitative comparison results on R100L dataset (upper row) and R100H dataset (bottom row). More visual results are available in appendix D.
input
Transweather
MAXIM
Ours
GT
Figure 3: Qualitative comparison results on Raindrop dataset.
Methods
Metrics
Methods
Metrics
Methods Metrics
PSNKT
SSIM
LPIPS.
NFE]
Methods Metrics
PSNRT
SSIM
LPIPS]
NFE.
JORDER[3]
26.25
0.85
0.197
]ORDER[3]
36.61
0.97
0.028
MAXIM[4]
30.81
0.90
0.133
MAXIM[4]
38.06
0.98
0.048
PReNe([1]
29.46
0.90
0.128
PReNet[ I]
37.48
0.98
0.020
Restormer[5]
31.46
0.90
0.086
Restormer[5]
38,99
0.013
IR DRF2D2]22]
31.65
0.90
0.047
100
IR-SDE122
38.30
0.98
0.014
100
CNN-baseline
29.12
0.88
0.153
CNN-baseline
33.17
0.96
0.068
IR-CM-CD (ours)
29.75
0.38
0,064
IR-CM-CD (ours)
34.21
IR-CM-CT (ours)
30.47
0.92
0.016
IR-CM-CT (ours)
36.18
0.98
0.009
IR-CM-CT (ours)
30.71
0.92
0.015
IR-CM-CT (ours)
37.06
0.98
0.005
2
Quantitative comparison results on 1100H dataset. The optimal results are indicated by bold underlining.
Table I:
Table 2: Quantitative comparison results 1100L dataset. The optimal results indicated by bold underlining.
conducted using the original resolution of each dataset (for ease of presentation, the image sizes were adjusted in Figures 2-5).
4.1.1 Image deraining
We validate the effectiveness of the proposed IR-CM based on two datasets: R100L and R R100H[41]. total of 2000 images are used for training, while 200 images are reserved for testing. Our method is qualitatively and quantitatively compared with some of the Mate-of-the-ard image deraining methods including IR-SDE[22]. JORDER[3), Restormer(5), PReNet[1], and MAXIM[4). The comparison results are shown in Table 1, Table 2 and Figure 2. More visual results are available in appendix D. To further validate the effeetiveness of our method real-world rainy scenarios, we conducted comparative experiments the Raindropp [42] dataset containing 1119 pairs of real-world rainy/non- rainy images. The results are shown in Table and Figure 3.
The IR-CM model we proposed surpasses the baseline IR-SDE model and achieves optimal per- formance on both SSIM and L.PIS metries in scenarios of either one-step or two-step reasoning. Furthermore, the comparison results with the CNN-baseline demonstrate that our approach improves network performance while ensuring real-time capabilities. Note that our IR-CM based on IR-SDE for CD or CT, thus IR-CM-CD represents consistency distillation based on pre-trained IR- IR-SDE model as teacher model. And TM-CM-CT only initializes the model using pre-trained IR-SDE checkpoint of faster convergence and does not any teacher model during the CT process. Of course, random
7

---

methods metrics
PSNRT
SSIMI
LPIPS
NFE,
(222
31.87
0.935
0.079
1
2220)))
34.55
0.950
0.051
1
Refusion(2023)
32.61
0.938
0.048
100
IR-CM(I-stee (ours)
32.06
0.934
0.043
IR-CM((-step))) (ours)
32.89
0,936
0.041
2
Table 3: Quantitative comparison with some of image deraining methods on Raindrop dataset. The optimal results indicated by bold underlining.
Methods
Metrics
Mothods
Metries
Methods Meties
PSNRT
SSIMI
LPIPS.
NFEL
Methods / Metrics
PSNR
SSIM
LPIPS!
NFE,
DeepDeblur[43]
29,08
0.91
0.135
DnCNN[7]
31.32
0.87
0.101
DRGAN[15]
31.18
11.2
0.112
HFDNe(8]
32.36
0.89
0.103
1
DeblurGAN-v([44]
29.55
0.93
0.117
[R-SDE
29.48
0.81
0.071
100
MAXIM[4]
32.86
0.94
0.089
Denoising SDH
28.98
0.75
0.088
22
IR-SDE[22]
30.70
0.90
0.064
100
Denoising-OUe (DE((22]
32.39
0.88
0.055
22
DiffIR[4]]
33.20
0.963
CNN-baseline
25.02
0.78
0.102
IR-CM-CD (ours)
28.96
0.089
IR-CM-CD (ours)
24.98
0.78
0.085
IR-CM-CT CMM-TTEL (ours)
29.72
0.95
0.013
IR-CM-CT CM-CT (ours)
25.61
0.82
0.048
IR-CM-CT (ours)
29.87
0.95
0.012
IR-CM-CT (ours)
27.56
0.88
0.027
Table 4: Quantitative comparison results on Go-
Table 5: Quantitative comparison results on Mc-
Pro dataset. The optimal results are indicated by
Master dataset with noise level a = 25. The opti-
bold underlining.
mal results indicated by bold underlining.
weight initialization also an option. In practice, we found that when given enough training time, the performance of both approaches is quite similar.
4.1.2 Image deblurring
We validated the effectiveness of the proposed IR-CM model for image deblurring task based on the GoPro[43] dataset. A total of 2103 images are used for training, while 1111 images are reserved for testing. Our method is qualitatively and quantitatively compared with some of the milestone image deblurring methods including DeepDeblur(43), DeblurGAN-v2[14 DBGAN[15], MAXIM[4), [44] and of course IR-SDE((22.. The comparison results shown in Table 4 and Figure 4. More visual results are available in appendix D.
Our proposed method achieves optimal performance SSIM and LPIPS metrics, surpassing the IR- SDE baseline model as well. This strongly demonstrates the effectiveness of our proposed consistency training approach. Notably, like other SDE-based models, prioritize perceptual scores LPIPS over distortion scores PSNR and SSIM.
4.1.3 Image denoising
Note that the last term in (1) is Gaussian process. Hence, we can consider special case where μ = a()). In this case, the IR-SDE degenerates into a pure additive noise process. This implies that any point along the PF-ODE trajectory can serve noisy image to be processed. In this scenario, we can only apply the conventional consistency training approach and cannot utilize the proposed linear-nonlinear decoupled training strategy because z0) must be set the HQ image and r(f) represents the corresponding low-quality (LQ) image.
Similar to [22], we collected approximately 5000 HQ images from the DIV2K[45), Flickr2K[45], and BSD500[46] datasets for training. and subsequently tested on the McMaster[47] dataset. To demonstrate the competitiveness of our method against state-of-the-an approaches, we compare it with DrCNN[7], FFDNet(8), as well as the special denoising methods proposed [22], namely Denoising-SDE: and Denoising-ODt Our method achieves the optimal result in perceptual scores. Comparative experimental results are shown in Table 5, and visual results are available appendix
D.

---

input
DeepDeblur
DBGAN
DeblurGANv2
MAXIM
IR-SDE
Ours
Figure 4: Qualitative comparison results on GoPro dataset. More visual results are available in
appendix D.
input
DiffLL
GlobalDiff
Retinexformer
LLFlow
IR-SDE
Ours
GT
Figure 5: Qualitative comparison results on LOLv2 dataset. More visual results are available in
appendix D.
Methods
Metries
Components
Metrics
Methods / Metries
PSNRT
SSIMT
LPIPST
NEEL
Group
OECF
OGL
LLDT
PSNR
SSIM
LPIPS
DiILL
28.36
0.88
0,207
10
x
X
23.06
0.82
0.144
GlobalDiff 20]
28.82
0.90
0.095
x
23.62
Retinexformer 16]
27.71
0.86
0.097
3
V
23.99
0.86
0.120
LLFlow[17]
26.02
0.93
0.099
4
4
x
25.04
0.91
0.059
[R-SDE[(2]
27.05
0.90
0.087
100
X
23.96
0.121
CNN-baseline
23.16
0.81
0.156
6
25.01
0.89
0.072
IR-CM-CD (ours)
26.33
0.89
0.098
7
24.45
0.86
0.101
(ours)
27.61
0.93
0.027
IR-CM-CT (ours)
30.20
0.95
0.021
Table 7: Ablation experiment results on LOLv2
Table 6: Quantitative comparison results
dataset. OECF: rrggin-sttimated consistency func- tion, OGL: origin-guided loss, LLDT: linear- nonlinear decoupling training strategy.
LOLv2 dataset. The optimal results are indicated by bold underlining.
4.1.4 Low-light image enhancement
We compare our method with existing researches for low-light image enhancement tasks on the LOLv2[48] dataset consisted of 689 pairs training images and 100 pairs testing images. We compare our proposed method with some recent approaches for low-light image enhancement task, including: DiffLL[21], GlobalDiff[20), Retinexformer( [6]] LLFlow[17]. Note that IR- IR-SDE is trained following the experimental settings described in [22], while IR-CM-CD is based on consistency distillation using the trained weights of IR-SDE. Our method outperforms the state-of-the-- art diffusion- -based methods, DiffLL and GlobalDiff, and requires only two sampling steps. The corresponding quantitative and qualitative comparison results are shown in Table and Figure 5 respectively. More visual results are available in appendix D
4.1.5 Runtime Comparation
To validate the superiority of our method in inference real-time performance, we compared with several other state-of-thh-art (SOTA) methods across three common resolution sizes. The results are shown Table8. Compared the baseline method IR-SDE, our method significantly reduces inference time by introducing a consistency training process. In some real-time demanding applications, one-step sampling inference allows for fast predictions while maintaining competitive performance. On the other hand, two-step sampling inference can substantially improve model

---

performance with only a slight trade- off in inference speed. Users can choose the approach based on their specific requirements. Additionally, a discussion on the model complexity will be provided in Appendix E.
image size methods
MAXIM
Restormer
IR-SDE
DiffLL
GlobalDiff
DiffIR
(R-CM(1-step)
IR-CM(2-step)
256x256
0.092
0.117
7.325
0.087
0.134
0.574
0.073
0,145
600x400
0.297
0.395
27.418
0.309
0.637
2.743
0.273
0,547
1280x720
1.223
1,512
98.731
1.045
1.881
7.155
0,992
1.984
Table 8: Inference time comparison with some of SOTA methods on three typical image sizes, All tests were conducted using an NVIDIA 2080Ti GPU. The optimal results are indicated by bold underlining.
4.2
Ablation Experiments
4.2.1
Components ablation experiments
The main innovations of this paper lie in the OECF, Origin-guided loss, and the linear-nonlinear decoupling training strategy proposed in the previous section. To verify the effectiveness of each component, we conducted series of ablation experiments using low-light image enhancement as an example. To ensure fairness, each control group used the same checkpoint for model initialization. and the total number of epochs was kept consistent during training. And only the performance of one-step sampling inference was considered. The corresponding results are shown in Table 7. The effectiveness of the proposed OECF can be demonstrated by group pairs (1, 3), (2, 4), (5, 7), (6, loss is evident from
8). The significant improvement in perceptual scores due to the origin- the comparisons between group pairs (1, 2), (3, 4), (5, 6), (7, 8). Finally, the notable performance decoupling training strategy can seen from the comparisons between groups (1, 5), (2, 6), (3, 7), (4, 8). Note that for groups 5 and 7, we only use the origin- guided loss during the Linear- fittar--ingiggg stage, and not during the nonlinear- nonlincar-fitting stage.
enhancement the model due to the linear-nonlined
4.2.2 Selection of origin-guided loss weight
The setting of ÀoG also affects the performance of IR-CM. If set too low, the model may become unstable and perform poorly, as discussed in Section 3.2. If set too high, it will affect self-consistency property, leading to a degradation into a CNN-baseline method, We experimented with multiple values and empirically found that the best performance achieved around 0.8. The related experimental results are discussed in Appendix B.
4.2.3 Linear-fitting stage only & Multiple sampling
In fact, after completing the linear-fitting stage, the model can already perform one-step sampling inference, however, it cannot perform multi-step sampling as discussed in Section 3.3. To further evaluate the model's performance, with l-step, 2-step, 4-step, and 6-step sampling. Empirically, we consider 2-step sampling to be the most cost-effective choice. The related experimental results are discussed in Appendix C.
tested it under the conditions of fitting stage only, and
Conclusion
This paper proposes multi-task image restoration and enhancement method, IR-CM, based on consistency training approach, enabling few-step or even one-step sampling inference. Specifically, we proposed the Origin-estimated Consistency Function (OECF), which provides more stable initial state and a smaller solution space for the consistency training process. Furthermore, to make the training process more robust and prevent trivial solutions, we introduced the Origin-guided Loss (OE Loss). Based on these, we developed a Linear-Nonlinear Decoupling Training Strategy, which not only accelerates the training process but, more importantly, enables the model to perform multi-step sampling inference, further enhancing its performance. Finally, a series of comparative experiments and ablation studies demonstrated the effectiveness of the proposed method.
10

---

6 Acknowledgment
Our work is supported by the weapons and equipment pre-research fund, Grant No. 5091 1020603.
References
[1] Dongwei Ren, Wangmeng Zuo, Qinghua Hu, Pengfei Zhu, and Deyu Meng. Progressive image deraining networks: A better and simpler baseline. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3937-3946, 2019.
[2] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Multi-stage progressive image restoration. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 14821-14831, 2021.
[3] Wenhan Yang, Robby T Tan, Jiashi Feng, Zongming Guo. Shuicheng Yan, and Jiaying Liu. Joint rain detection and removal from a single image with contextualized deep networks. IEEE transactions pattern analysis and machine intelligence, 42(6)::377-1333, 2019.
[4] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxim: Multi-axis mip for image processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5769-5780, 2022.
[5] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728-5739, 2022.
[6] Yeachan Park, Myeongho Jeon, Junho Lee, and Myungjoo Kang. Mew-net: Single image deraining with multi-level connections and wide regional non-local blocks. Signal Processing: Image Communication, 105: 16701, 2022.
[7] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang, Beyond gaussian denoiser: Residual learning deep enn for image denoising. IEEE transactions on image processing, 26(7):3 (100)3342231555 2017.
[8] Kai Zhang, Wangmeng Zuo, and Lei Zhang, Fidnet: Toward fast and flexible solution for cnn- cnn-based image denoising. IEEE Transactions on Image Processing, 2799::6088-460 2018.
[9] Rongkai Zhang, Jiang Zhu, Zhiyuan Zha, Justin Dauwels, and Bihan Wen. R31: Connecting deep reinforcement learning to recurrent neural networks for image denoising via residual recovery. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1624-1628. IEEE,
2021.
[10] Shen Cheng, Yuzhi Wang, Haibin Huang, Donghao Liu, Haoqiang Fan, and Shuaicheng Liu. Nbnet: Noise basis learning for image denoising with subspace projection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4896-4906, 2021.
[11] Meng Chang, Qi Li, Huajun Feng, and Zhihai Xu. -patial-adaptive network for single image denoising. In Computer Vision ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 17 171-187. Springer, 2020.
[12] Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3883-3891, 2017.
[13] Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jifí Matas. Deblurgan: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8183-8192, 2018.
[14] Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. Deblurgan-v2: Deblur- ring canders-of-magninne faster and better. In Proceedings of the IEEF/CVF international conference computer vision, pages 8878-8887 2019.

---

[15] Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Bjom Stenger, Wei Liu, and Hongdong Li. Deblurring by realistic blurring. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 2737-2746, 2020.
[16] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang, Retinex- former: One-stage retinex-basee transformer for low-light image enhancement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12504-12513, October 2023.
[17] Yufei Wang, Renjie Wan, Wenhan Yang. Haoliang Li, Lap-Pui Chau, and Alex C Kot. Low-light image enhancement with normalizing flow. arXiv preprint arXiv:2109.05923 2021.
[18] Alexandru Brateanu, Raul Balmez, Adrian Avram, and Ciprian Orhei. Lyt-net: Lightweight yuv transformer-basee network for low-light image enhancement. arXiv preprint (rxxi:2401.152000 2024.
[19] Wu Yuhui, Pan Chen, Wang Guoqing, Yang Yang, Wei Jiwei, Li Chongyi, and Heng Tao Shen. Learning semantic-aware knowledge guidance for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[20] Hou Jinhui, Zhu Zhiyu, Liu Hui, Zeng Huanqiang, and Yuan Hui. Global structure-aware diffusion process for low-light image enhancement. Advances in Neural Information P Processing Systems, 2023.
[21] Hai Jiang, Ao Luo, Haoqiang Fan, Songchen Han, and Shuaicheng Liu. Low-light image enhancement with wavelet-based diffusion models. ACM Transactions on Graphics (TOG), 42(6))1-14, 2023.
[22] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Image restoration with mean-reverting stochastic differential equations. International Conference on Machine Learning, 2023.
[23] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 1166440-8511 2020.
[25] Jiaming Song, Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. 1rxiv22010.02502, October 2020.
[26] Alexia Colicoeur-Martinees Ke Li, Rémi Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105. 14080, 2021.
Tal Kachman, and loannis Mitliagkas.
[27] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713 4726, 2022.
[28] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1161-11477, 2022.
[29] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
[30] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas Schön. Refusion: Enabling large-size realistic image restoration with latent-space diffusion models. In Proceed- ings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition Workshops, pages 1680-1691,,00
12

---

[31] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.01018, 2023.
[32] Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao, Jens Sjölund, and Thomas B Schön. Photo- realistic image restoration in the wild with controlled vision-language models. arXiv preprini arXiv:2404.09732, 2024.
[33] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[34] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 2020.
[35] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-557 -7777,, 2022.
[36] Kaiwen Zheng. Cheng Lu, Jianfei Chen, and Jun Zhu. Dpm-solver-v3: Improved diffusion ode solver with empirical model statistics, Advances in Neural Information Processing Systems, 36, 2024.
[37] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models, arXiv preprint arXiv:2303.0m 0369, 2023.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 2014.
[39] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[40] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[41] Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1357-1366, 2017.
[42] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2482-2491, 2018
[43] Tae Hyun Kim, Seungjun Nah, and Kyoung Mu Lee. Deep multi-scale convolutional neural net- work for dynamic scene deblurring. In Conference on Computer Vision and Pattern Recognition, pages 1-21. IEEE, 2017.
[44] Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang. and Lue Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13095-13105, 2023.
[45] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 126-135, 2017.
[46] Pablo Arbelacz, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898-916 2010.
[47] Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging, 20(2):023016- 023016, 1011.
13

---

[48] Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072-2086, 2021.
[49] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arx1v:/711.05001,2017 2017.
14

---

A Implementation details
We chose the same U-Net backhone network used in IR-SDE[[22. Note that the CNN-baseline uses the same U-Net backbone, with its time embedding set to constant, and directly outputs HQ images from the input LQ images. For all tasks, set the training patch-size to be 128 128 and use batch size of 4, We used the Adam W[49] optimizer with weight decay set to 0.01 and set learning rate 10 - Our models are trained on two 2080Ti GPUs for 200 epochs for each task, with the linear-fitting stage and the nonlinear-fitiin fittinn stage each accounting for 100 epochs. The training hyperparameters are set as follows:
10, ÀOG 0.8, Apere 0.125.
The 0 schedule is defined same as [22]]
f(()
1/T + 2)
0. where 0.008. And the consistency weights Cokip and Cout are set as follow: Cakip(t) = Coue(t), Come (t) = t/T,
f(0)' S(1) cos(
(17)
(18)
where T' is set 100.
B
Selection of origin-guided loss weight
We take the low-light image enhancement task as an example to consider the impact of different ÀoG values the final performance of the model considering only the one-step sampling scenario. The quantitative experimental results are shown in Table 9.
Metries/XoG
0.2
0,4
0.6
0.8
1.0
1.2
PSNR
22.36
22.11
24.42
26.94
27.61
27.67
27.53
SSIM
0.85
0.87
0.89
0.92
0.93
0.92
0.90
LPIPS
0.136
0.129
0.086
0.044
0.027
0.042
0.075
Table 9: Quantitative comparison results of different Aoa.
The optimal results are indicated by
bold underlining.
The results show that the best performance is achieved when ÀoG is 0.8. Although higher PSNR score was obtained with Åoc = 1.0, we prioritize perceptual scores. Therefore, we chose ÅoG 0.8 as the experimental setting in Section 4.1.
C linear-fitting stage only & Multiple sampling
As shown in Algorithm 3, our method, like most SDE- based methods, can improve model performance through multi-step sampling. Specifically, if set M 1, one-step sampling can be achieved. To further evaluate the model's performance, we tested it under I-step, 2-step, 3-step, and 4-step sampling. Additionally, we evaluated the model's performance after completing only the linear-fitting stage (LFS). Similarly, using low-light enhancement as an example, the test results are shown in Table 10 and Figure 6.
Metrics/NFE
LFS only (one-step)
PSNR
27.6139
30,2070
31.4882
31.2916
27.2421
SSIM
0.9301
0.9493
0.9496
0,9490
0.9216
LPIPS
0.02693
0.02080
0.02146
0.02247
0.02803
Table 10: Quantitative comparison results of different sampling step. The optimal results are indicated by bold underlining.
It can be observed that the optimal perceptual score is achieved when the sampling step is 2. Further increasing the sampling steps leads to only slight improvements model performance but implies longer inference times. In the case of one-step sampling, the performance of LFS only is not significantly different from that of the two-stage trained model. Considering all factors, we believe that 2-step sampling inference is the most cost-cffective.
15

---

Algorithm 3: Multi-step sampling inference
Input LQ image æ1Q. OFCF fol:; 1), sequence of time points < l2... < IN.
inference step M;
m + 0, Sample It,, ~ (N, vt.); while m < M do
N. 1u &- FLQ:
:
n - -n/2;
M4. +- + (Io - 1 Sample It, ~ N (me,, Vea );
m7 1--11 +1
end
Output: 20
input
ststpp
2-step
3-step
s-step
GT
Figure 6: Visual results of different sampling step.
D More visual results
input
JORDER
MAXIM
PReNet
Restormer
IR-SDE
Ours
GT
Figure 7: Image deraining visual results R100H dataset.
16

---

input
JORDER
MAXIM
PReNet
Restormer
IR-SDE
Ours
GT
Figure 8: Image deraining visual results on R100L dataset.
input
DeepDeblur
DBGAN
DeblurGANv2
MAXIM
IR-SDE
Ours
GT
Figure Image deblurring visual results on GoPro dataset.
17

---

input
IR-SDE
ours
GT
input
IR-SDE
Ours
GT
Figure 10: Image denoising visual results on McMaster dataset with sigma 25.
O
input
DiffLL
GlobalDiff
Retinexformer
LLFlow
IR-SDE
Ours
GT
Figure 11: Low-light image enhancement visual results on LOLv2 dataset.
18

---

E
Model complexity
Our backbone network conditional Unet, and its computational complexity is proportional to the number of convolutional layers and the size of the image. The complexity can be calculated as follows:
L-1
Σ H 2i
W
O(S *
*
Cin.i * Cout, * K2)),
(19)
2i
where L is number of convolutional layers, H, W' are the height and width the image, Cin.is Cout, represent the imput and output sizes of the convolutional layer respectively, K is the size of the convolutional kernel, and denotes the sample steps. From the above formula, it can seen that model has linear relationship with the height and width of the input image. And it is also linearly related to the number of sampling steps. This advantageous for applying our model to larger-sized images. Additionally, by adjusting the number of sampling steps, trade-off between real-time performance and model performance can be achieved.
Potential societal impact
Our work on conditional image generation tasks, such image restoration and enhancement, has several potential societal impacts. These improvements can benefit fields like medical imaging, where enhanced image quality can aid in better diagnosis and treatment. Additionally, improved image restoration techniques can be valuable in preserving and restoring historical photographs and artworks.
However, there are also potential negative impacts to consider. Enhanced image generation techniques could be misused for creating deceptive content, such as deepfakes, which can have serious ethical and social implications. Therefore, it is crucial to implement safeguards and ethical guidelines to prevent misuse and ensure that the technology is used for beneficial purposes.
We are committed to promoting the positive applications our research while being aware and mitigating potential risks.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately reflect the contributions and scope of the paper in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We emphasized in the introduction and abstract that our work is applicable to conditional image generation tasks, such as image restoration and image enhancement. not suitable for unconditional generation tasks.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
20

---

Answer: [Yes]
Justification: In the Chapter 3, we introduced and demonstrated the effectiveness the three main contributions of this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Appendix A, we elaborate on the experimental details. Additionally, our code will be open-sourced after the review process.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We are pleased to open-source our code, which will be made available on com.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccppccc.oec.cc...ccc.ccttt htapli//guiieeee gwwges/iee/////e Cooeeuumiis wwtamnssssooni ssliciolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Chapter 4 and Appendix A, we provide detailed explanations of our experi- mental procedures, including dataset splits and experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments primarily compared the performance of various existing methods on specific public datasets. Once the model is trained, these metrics (PSNR, SSIM, LPIPS) are typically deterministic and do not change due to other random factors. Therefore, it is unnecessary report error bars in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Appendix A, provide detailed information about the GPU models and memory sizes used in our experiments, as well as the number of training iterations required.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https / http///eeuurp.eeeeeeeeee wwteriis.ccppppuiie cc/public// htcp/ubbic//thieeeeoc Bete/tthiissuuee csGuidelines:
Answer: [Yes]
Justification: We have carefully read the NeurIPS Code of Ethics and ensure strict adherence to
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential negative socictal impacts of our method in Appendix
E.
Guidelines:
23

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our proposed method poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have provided clear citations for the public datasets used in our comparative experiments, as well as for foundational works in the field.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi whwoee com/ con/dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Due submission size constraints, only provide code in the supple- mentary material. The corresponding datasets, test results, and pretrained models will be open-sourced in the future.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourced experiments or research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourced experiments or research involving human subjects.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "b25wNeEuIO", "reviewer": "Reviewer_ksNq", "summary": "This paper proposes a SDE-based two-stage network to tackle the image restoration tasks, including deraining, deblurring denoising, and low-light image enhancement. Based on the existing stochastic differential equation works, the proposed method focuses on the training efficiency and inference speed. It proposes a linear and non-linear decoupling training strategy to enhance the training effectiveness and to surpass consistency distillation on inference. Extensive experiments on multiple different image degradation data show that \nthe proposed method is able to outperforms baseline methods.", "questions": "The authors are suggested to address the issues raised in the weakness section during rebuttal period.", "limitations": "There is no limitation mentioned in the paper or the supplementary material.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strength\n1. the proposed method is a fast universal method that is able to solve multiple image degradation problems including deraining, denoising, low-light enhancement etc.  \n2. Extensive experiments demonstrate the proposed method outperforms the baseline dedicated methods on multiple benchmarking datasets\n3. The proposed training schema may bring impact and benefits to the development of SDE study.", "weaknesses": "Weakness\n1. What does NFE in the quantitative comparison tables (Table 1,2,3,4) stand for? \n2. As the paper is claiming the proposed method is a fast and universal method at inference time, it is suggested to add the inferencing time comparison with respect to baseline methods. \n3. There is no conclusion in the submitted paper.\n4. All the experiments on deraining topic use synthetic rain images. It is more useful and applicable to compare methods on real rain images. \n\nMinors\n1. All the functions are lack of punctuations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work, and we will carefully consider the improvement suggestions you have proposed.\n\nFor Weakness 1.\n\nanswer: \nThank you for your careful review. We indeed overlooked the introduction of NFE. Number of Function Evaluations(NFE) refers to the number of function evaluations required to generate an image or data. In other words, it is the number of evaluations needed at each step of the diffusion process. A lower NFE usually indicates a more efficient model, as it requires fewer computational resources. In diffusion model related work, it often refers to the number of sampling steps during model inference. This introduction will be added to subsequent revision.\n\nFor Weakness 2.\n\nanswer: \nThank you for your useful suggestion. We will add the comparation results of inference time between methods in subsequent revision. The comparation results are shown in Table 1 in the rebuttal PDF material. You can check them out. The results demonstrate the superiority of our method in terms of computational efficiency.\n\nFor Weakness 3.\n\nanswer: \nThank you for pointing out our issue. Due to page limitations, we omitted the conclusion section in the previous submission version. In the subsequent revised version, we will add the following text to the conclusion section:\n\nIn this paper, we propose a fast, general-purpose image restoration method based on a Consistency Model and Stochastic Differential Equations (SDE), capable of inferring high-quality images in one or a few sampling steps. Specifically, we introduce a novel Consistency Function OECF, and theoretical and experimental results demonstrate its performance enhancement for Consistency Models. To stabilize the consistency training process and improve training efficiency, we propose the OG loss, which effectively avoids trivial solutions and accelerates model convergence. Furthermore, to further improve the model's performance within a limited number of training iterations, we introduce a novel linear-nonlinear decoupling training strategy for our proposed model, with theoretical analysis and experimental results proving its effectiveness. In comparative experiments on tasks such as image de-raining, denoising, and deblurring, our model shows highly competitive performance, even achieving state-of-the-art result in low-light image enhancement tasks with two-step sampling inference. Finally, extensive ablation experiments demonstrate the effectiveness of each component we propose.\n\nFor Weaknesses 4.\n\nanswer:\nThank you for your useful suggestion. We have supplemented the comparison experiment results on the Raindrop[1] dataset containing 1,119 pairs of real-world rainy/non-rainy images, which can be viewed in Table 2 and Figure 1 of the rebuttal PDF materials. And it will be added to the subsequent revised version. Experimental results indicate that our method is also highly competitive for rain removal tasks in real-world scenarios.\n\nReference:\n\n[1] \"Attentive Generative Adversarial Network for Raindrop Removal from a Single Image\", 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "esEihVWcTG", "reviewer": "Reviewer_yxtc", "summary": "The paper introduces a method called IR-CM (Image Restoration Consistency Model) for fast and general-purpose image restoration.\nThe key idea is to achieve few-step or one-step inference by employing consistency training on specific mean-reverting stochastic differential equations (SDEs).\nA novel linear-nonlinear decoupling training strategy is proposed to enhance training effectiveness and inference performance without relying on pre-trained models.\nAn origin-guided loss is introduced to avoid trivial solutions and stabilize model training.\nExperiments conducted on tasks such as image deraining, denoising, deblurring, and low-light image enhancement demonstrate highly competitive results with one-step inference and state-of-the-art performance in low-light image enhancement with two-step inference.", "questions": "Besides the above Weaknesses, I still have some questions:\n\nThe one-step inference model cannot perform multi-step sampling, which might limit its application in certain scenarios where more refinement is needed.\n\nHow does the method handle different levels of image degradation beyond the tested conditions?\n\nWhat is the computational complexity of the method, and how does it scale with the size of the image or video?\n\nCan the method be extended to handle video restoration tasks, and if so, what would be the challenges?\n\nHow does the method compare to other state-of-the-art methods in terms of computational resources and efficiency?\n\nAre there any specific use cases or applications where the method excels or falls short?", "limitations": "The societal impacts are discussed in Appendix.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-structured and easy to follow.\n- The method is designed for fast inference, which is crucial for real-time applications.\n- It is a general-purpose model that can be applied to various image restoration tasks without domain-specific prior knowledge.\n- The proposed linear-nonlinear decoupling strategy and origin-estimated consistency function enhance training and inference performance.\n- IR-CM does not depend on any pre-trained checkpoints, making it an independent model.\n- The method achieves competitive or even state-of-the-art results in various image restoration tasks with minimal inference steps.", "weaknesses": "1. While the method is designed to be universal, its performance across different types of image degradations and real-world scenarios might vary.\n2. The introduction of a novel training strategy and loss function may complicate the training process compared to more straightforward models.\n3. The method's performance may be sensitive to the choice of hyperparameters, such as the weight of the origin-guided loss.\n4. Without sufficient data or proper regularization, the model might overfit to the training data, which could reduce its effectiveness on unseen data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our work. We will seriously consider your suggestions and address your concerns.\n\nFor Weakness 1.\n\nanswer:\nOur method aims to construct a general-purpose architecture that does not require any prior information. By simply replacing the dataset, it can accomplish different image restoration and enhancement tasks. Our method is not intended to solve multiple image restoration problems with a single checkpoint. We apologize for the ambiguity caused by the term \"universal\",  a more appropriate term should be \"general-purpose.\" We will replace it in subsequent revisions to eliminate any confusion.\n\nFor Weakness 2.\n\nanswer:\nIn fact, within the same number of training iterations, our proposed two-stage training strategy and OG loss make the training process more stable and efficient , without introducing obvious additional complexity. The proposed linear-nonlinear decoupling training strategy only introduces a simple assignment operation, i.e., replacing the variable $x_0$ from the low-quality image $\\mu$ to the high-quality image. The OG loss merely adds a similarity comparison term to the original loss function. We believe that the introduction of these minimal operations is entirely worthwhile, as they significantly enhance the model's performance (see ablation study Table 6).\n\nFor Weakness 3.\n\nanswer:\nOur hyperparameter choices are derived from extensive experimental testing to achieve optimal results that can adapt to various image restoration tasks. Specifically, we discuss the impact of the OG loss weight $\\lambda_{OG}$ on model performance in Appendix B. Empirically, we found that the model performs best when the $\\lambda_{OG}$ is set to 0.8.\n\nFor Weakness 4.\n\nanswer:\nTo prevent overfitting, we employed regularization techniques with a decay rate set to 0.01, as described in the experimental setup section of Appendix A. \n\nFor Question 1. \n\nanswer:\nOur method, like other SDE-based methods, is capable of multi-step sampling inference. Most of our optimal results are achieved with 2-step inference (see Tables 1-5 in Section 4.1). We also discuss the impact of sampling steps on model performance in Appendix C, where we conclude that 2-step sampling provides the best trade-off between performance and efficiency. The pseudocode for multi-step sampling inference is also included in Appendix C for your reference.\n\nFor Question 2. \n\nanswer:\nWe believe that our method has better generalization ability for different levels of degradation compared to other universal image restoration methods (e.g. MAXIM, Restormer etc). Because our training process not only considers the end-to-end mapping from LQ images to HQ images, but also the mapping from various intermediate states between LQ and HQ images to HQ images. This makes our method robust to different levels of degradation.\n\nFor Question 3. \n\nanswer:\nOur backbone network is a conditional Unet, and its computational complexity is proportional to the number of convolutional layers and the size of the image. The complexity can be calculated as follows:\n$$\n O(S*\\sum_{i=0}^{L-1}(\\frac{H}{2i}*\\frac{W}{2i}*C_{in,i}*C_{out,i}*K^2))\n$$\nwhere L is number of convolutional layers, $H, W$ are the height and width of the image, $C_{in,i}, C_{out,i}$ represent the input and output sizes of the convolutional layer respectively, $K$ is the size of the convolutional kernel, and $S$ denotes the sample steps. From the above formula, it can be seen that our model has a linear relationship with the height and width of the input image. And it is also linearly related to the number of sampling steps. This is advantageous for applying our model to larger-sized images. Additionally, by adjusting the number of sampling steps, a trade-off between real-time performance and model performance can be achieved. We reported the inference speed of our method at different image sizes, which can be found in Table 1 of the rebuttal PDF materials. Our method is highly competitive not only in terms of performance but also in inference speed. We will include these contect in the revised version of the paper.\n\nFor Question 4. \n\nanswer: \nYes, unlike other diffusion-based methods, our method is well-suited for real-time video processing tasks. This is because it does not require a large number of sampling steps and can achieve good inference results with just one or two sampling steps. However, there are still some challenges. For video frames with larger sizes and applications requiring high real-time performance, our method currently struggles to handle these scenarios effectively.\n\nFor Question 5. \n\nanswer: \nWe reported the inference speed of our method at different image sizes, which can be found in Table 1 of the rebuttal PDF materials. ou can refer to the provided supplementary materials. The results demonstrate the superiority of our method in terms of computational efficiency.\n\nFor Question 6. \n\nanswer: \nOur method is suitable for most image-to-image translation tasks with paired training data, as it leverages the data-fitting characteristics of diffusion models and overcomes the issue of image generation randomness present in most diffusion-based methods. However, it still struggles with tasks where obtaining paired training data is challenging (e.g., glare removal). Our future research direction will focus on extending IR-CM to train with unpaired data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A9TcEqBDb2", "reviewer": "Reviewer_14RS", "summary": "This paper modifies the consistency model from image generation to image restoration with three modifications. Each module contains robust theoretical proof and shows great performance improvement.  Linear-nonlinear decoupling training strategy will motivate future works in using consistency model in image restoration. The additional results show the power of the paper both in inference speed and performance.", "questions": "See the weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The modification from Consistency Model in image generation to image restoration is terrific. Each module shows robust theoretical proof 2. The extensive experiments show the great power of IR-CM in inference speed and performance.", "weaknesses": "1. The comparison with other methods seems to be unfair. For example, the result of retinexformer in low-light enhancement is tested on full resolution with y-channel calculation. It seems that all of the results of this paper are tested in 512*512 resolution, which is observed by the unexpectedly high SSIM metrics (Table 4 with comparable SSIM but significantly lower PSNR, Table 5 with over 30 PSNR and 0.95 SSIM, etc.). In my view, it is impossible. $\\textbf{Please show the result in full resolution}$ (i.e., results in 512 resolution will change the result largely and it is meaningless at all) with uniform metrics (like y-channel calculation) but not directly report the result of other papers.\n2. The phenomenon of first-stage training is interesting, as to why setting the $x_0$ equal to $\\mu$ will benefit learning image restoration tasks as it is just a denoising task. I understand that this will ease training, but that's still not enough to explain the phenomenon. \n3. Mistaking the meaning of the setting. Universal image restoration means using one model to train different degradation types at once, not meaning a framework that can be used to train different tasks (the author can freely search in the arxiv).\n4. Typos. Line 143 should change to \"the training process\" as in the inference process neither strategy uses pre-trained model. Line 152 the $c_{out}(T)=1$, not $\\eta$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our work. We will carefully address your questions and resolve your concerns.\n\nFor Weakness 1.\n\nAnswer: \nWe apologize for any misunderstandings. While we adjusted the images in the paper to square shapes to showcase as many comparative experimental results as possible, our method and the baseline methods were tested at the original resolution of each dataset(e.g., 600x400 for the LOLv2 dataset). The experimental result images shown in the paper were adjusted to square shapes purely for ease of presentation. We will make this clear in the revised version. \n\nAnd our metric settings are same as all the mentioned baseline methods (Retinexformer, LLFlow, DiffLL, GlobalDiff and IR-SDE etc). \nFor PSNR metric: We perform the calculation in the luminance space (Y channel) for low-light enhancement using the following formula\n$$\nPSNR=10*log_{10}(\\frac{MAX^2}{MSE}).\n$$\nFor SSIM metric: it refer to the paper \"Image Quality Assessment: From Error Visibility to Structural Similarity, 2004\".\nFor LPIPS metric: it refer to the paper “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, 2018”.\nWe will clarify these information in the subsequent revisions.\n\nWe understand your skepticism regarding the low-light enhancement experimental results, as they significantly surpass previous SOTA methods. While this result is genuine, we were samely surprised by it. And note that with single-step sampling inference, the model’s performance is not significantly different from the baseline methods. However, with two-step sampling inference, the model shows a considerable improvement, even the performance can be further enhanced with three-step sampling (see Appendix C Table 8) in terms of PSNR and SSIM, although this may lead to a decrease in the LPIPS metric.\nThen, upon objective analysis, we believe this remarkable performance is primarily due to the nonlinear fitting stage. At this stage, the model learns to map from various intermediate degraded images to high-quality images. This is more meaningful for low-light enhancement compared to other image restoration tasks because each intermediate degraded image corresponds to a specific luminance scenario. This allows the model to be more robust to a range of different luminance conditions, leading to better performance. \n\nFor Weakness 2.\n\nAnswer: \nRecall the proposed OECF formula (16) here:\n$$\nf_\\phi(x_t,t) = c_{skip}(t)\\mu + c_{skip}(t)[(x(0)-\\mu)e^{-\\bar{\\theta}_t}]\n$$\n\n$$\n+c_{skip}(t)\\sqrt{v_t}+c_{out}(t) \\hat{x}_0(x,t;\\phi)\n$$\n\nAs described in Section 3.3, during the linear fitting stage, by setting $x_0$ as the LQ image $\\mu$, the forward degradation process becomes adding different levels of noise to the LQ images. And the model learns to fit the first term (linear part) and the third term (noise part) of the formula above. Intuitively, at this stage, the model is not only learning to denoise but also to directly map the LQ images to HQ images (linear fitting). Therefore, actually, after the linear fitting stage is completed, the model can effectively perform single-step sampling inference (as discuss in Appendix C). However, at this point, as described in Section 3.3, the model cannot perform multi-step sampling inference, therefore, we still need to perform nonlinear fitting stage to enable the model to fit the second term of the formula above to perform multi-step sampling inference. \n\nIf we train the model in the same training iterations directly using nonlinear fitting from the beginning, it will achieve suboptimal performance (as shown in ablation study Table 6). We claim that the presence of the linear fitting phase allows the model to focus more on the mapping distribution when $t$ is large throughout the training process. In fact, for image restoration tasks, these mappings at larger $t$ values are more important because they signify more difficult restorations. If only nonlinear fitting is performed, the model will evenly focus on all mappings within $[0,t]$. We believe that with a very large number of training iterations, whether or not there is a linear fitting phase will not affect the model's final performance. However, this is impractical because our training time is often limited. Therefore, within limited training iterations, our proposed two-stage training strategy can achieve better performance compared to directly training with nonlinear fitting.\n\nFor Weakness 3.\n\nanswer: Thank you for pointing out our inappropriate wording; \"universal\" indeed has ambiguity. In our subsequent revisions, we will use \"general-purpose\" for clarification.\n\nFor Weakness 4.\n\nanswer: We greatly appreciate your careful reading, and we will thoroughly correct these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "524aAB9Csg", "reviewer": "Reviewer_huxY", "summary": "This paper presents IR-CM, a fast and universal image restoration method leveraging consistency models. The key innovations include a novel linear-nonlinear decoupling training strategy and an origin-estimated consistency function to enhance training effectiveness and inference performance. The proposed method is evaluated across several image restoration tasks, including draining, deblurring, denoising, and low-light image enhancement, showing competitive results with minimal inference steps.", "questions": "See weakness.", "limitations": "It would be better to analyze the limitations or some failed cases of the proposed methods.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed original-estimated consistency function (OECF) provides a more stable initial state and reduces the solution space.\n\n2. The origin-guided loss stabilizes training and prevents pattern collapse.\n\n3. Extensive experiments on four restoration tasks are conducted to validate the effectiveness of the proposed method.", "weaknesses": "1. There is an error in Eq. (8), $v_t$ should be $\\sqrt{v_t}$. Please have a check.\n\n2. There is an inconsistency in Theorem 1, ``$c_{out}(\\eta)=0, c_{out}(\\eta)=1$''. Please have a check. I suggest the author carefully check the writing of this paper and avoid such errors or typos.\n\n3. Even though this work contains rich experiments on multiple tasks, more recent diffusion-based methods should be considered as a comparison, such as DiffIR.\n\n4. All the experiments for draining and denoising are based on the synthetic training and testing data. To better validate the practical value in real applications, more real-world datasets should be considered.\n\n5. It would be more comprehensive to provide a comparison on model complexity and runtime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and support of our work. We will carefully address the issues you raised and make thoughtful revisions.\n\nFor Weakness 1 & 2.\n\nanswer:\nThank you for your thorough reading and careful review. We will diligently address the issues you identified and meticulously re-examine our submission.\n\nFor Weakness 3.\n\nanswer:\nThank you for your useful suggestions. We will include and analyze the comparison results with DiffIR[1] in the revised version.\n\nreference:\n\n[1] \"DiffIR: Efficient Diffusion Model for Image Restoration\", 2023.\n\nFor Weakness 4.\n\nanswer:\nThank you for your meaningful suggestions. We will add comparison results on real datasets. Currently, you can refer to Table 2 and Figure 1 in the provided rebuttal PDF materials to see the additional comparison results on the Raindrop[2] dataset. The comparison results show that our method is also competitive for image restoration tasks in real-world scenarios. This experimental results will be included in the main text in the subsequent revised version.\n\nreference:\n\n[2] \"Attentive Generative Adversarial Network for Raindrop Removal from a Single Image\", 2018.\n\nFor Weakness 5.\n\nanswer:\nThank you again for your meaningful suggestions. We have added a comparison of inference speeds, which you can view in Table 1 of the provided rebuttal PDF materials. The experimental results show that our method is competitive not only in performance but also in inference speed, making it suitable for applications with high real-time requirements. Similarly, this comparison result will be included in the subsequent revised version.\n\nFor Limitation:\n\nanswer:\nThank you again for your meaningful suggestions. We will analyze the limitations of our method in the experimental and conclusion sections of the subsequent revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the PCs, SACs, ACs, and all anonymous reviewers for their patient responses and valuable suggestions. Your contributions have been invaluable in improving the quality of our work. 

We have carefully responded to each reviewer's questions and suggestions. Additionally, we have submitted a rebuttal PDF containing supplementary experimental results, which reviewers can refer to at their convenience.

Meta-review: This paper received a Weak Accept, a Borderline Accept and 2 Borderline Reject. The main concerns raised by the reviewers are: that the writing needs improvements and is incomplete (eg, missing conclusions), that experiments on deraining are carried out only on synthetic data, issues with overfitting or sensitivity to the parameters tuning. One reviewer with the borderline reject rating states that all concerns have been addressed and that he/she will raise the rating. However, it seems that the rating is still not above the borderline. It seems that this is a mistake since all doubts were addressed. The AC has read all the reviews, the rebuttal and the discussion and has concluded that the main concerns
have been addressed and recommends that the paper be accepted.
Decision: Accept (poster)