Hybrid Regularization Improves Diffusion-based Inverse Problem Solving
Authors: Hongkun Dou
Zeyu Li
Jinyang Du
Lijun Yang
Wen Yao
Yue Deng
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion models
inverse problems
Primary area: generative models
TL;DR: We demonstrate a method for solving general inverse problems using hybrid regularization in diffusion models.

Abstract: Diffusion models, recognized for their effectiveness as generative priors, have become essential tools for addressing a wide range of visual challenges. Recently, there has been a surge of interest in leveraging Denoising processes for Regularization (DR) to solve inverse problems. However, existing methods often face issues such as mode collapse, which results in excessive smoothing and diminished diversity. In this study, we perform a comprehensive analysis to pinpoint the root causes of gradient inaccuracies inherent in DR. Drawing on insights from diffusion model distillation, we propose a novel approach called Consistency Regularization (CR), which provides stabilized gradients without the need for ODE simulations. Building on this, we introduce Hybrid Regularization (HR), a unified framework that combines the strengths of both DR and CR, harnessing their synergistic potential. Our approach proves to be effective across a broad spectrum of inverse problems, encompassing both linear and nonlinear scenarios, as well as various measurement noise statistics. Experimental evaluations on benchmark datasets, including FFHQ and ImageNet, demonstrate that our proposed framework not only achieves highly competitive results compared to state-of-the-art methods but also offers significant reductions in wall-clock time and memory consumption.

Published as a conference paper at ICLR 2025
HYBRID REGULARIZATION IMPROVES DIFFUSION- BASED INVERSE PROBLEM SOLVING
Hongkun Dou', Zeyu Li', Jinyang Du!.3 Lijun Yang', Wen Yau?. & Yue Deng!:3,
1 Beihang University 2 Chinese Academy of Military Science
3 Beijing Zhongguancun Academy (douhk, 11 1izeyu123478 dujinyang, yangl i jun, ydeng) ) @buaa edu. cn, wendy07828126.. com
ABSTRACT
Diffusion models, recognized for their effectiveness as generative priors, have become essential tools for addressing a wide range of visual challenges. Recently, there has been surge of interest in leveraging Denoising processes for Regular- ization (DR) to solve inverse problems. However, existing methods often face issues such as mode collapse, which results in excessive smoothing and diminished diversity. In this study, we perform a comprchensive analysis to pinpoint the root causes gradient inaccuracies inherent in DR. Drawing on insights from diffusion model distillation, we propose a novel approach called Consistency Regularization (CR), which provides stabilized gradients without the need for ODE simulations. Building on this, introduce Hybrid Regularization (HR), unified framework that combines the strengths of both DR and CR, harnessing their synergistic po- tential. Our approach proves to be effective across broad spectrum of inverse problems, encompassing both linear and nonlinear scenarios, as well as various measurement noise statistics. Experimental evaluations on benchmark datasets, including FFHQ and ImageNet, demonstrate that our proposed framework not only achieves highly competitive results compared to mate-offthhe-ar methods but also offers significant reductions in wall-clock time and memory consumption. Code is available https://gituu..com/eee aaaadddnng---llaHDaSSS
INTRODUCTION
Diffusion models (Ho et al., 2020; Song et al., 2020b), celebrated for their ability to model complex high-dimensional distributions, are increasingly applied across a broad spectrum of fields. These applications span from image generation (Dhariwal & Nichol, 2021; Saharia et al., 2022b; Rombach et al., 2022), and video synthesis (Cho et al., 2024; Ho et al., 2022; Rühling Cachay et al., 2024) to molecular design (Guan et al., 2023; Hoogehoom et al., 2022; Xu et al., 2022), and protein generation (Wu et al., 2024a; Trippe et al., 2022; Watson et al., 2023). The wealth of knowledge accumulated within pre-trained diffusion models, derived from vast datasets, empowers their utilization as powerful prior models. This capability is particularly valuable for reconstructing accurate outputs from incomplete measurement signals.
Recent research has highlighted the potential of diffusion models as plers in solving inverse problems. For instance, Kawar et al. (2022) pioneered DDRM, innovatively integrating the observation into the inverse denoising process within the spectral domain through singular value decomposition. In similar vein, Wang et al. (2022) introduced DDNM, which method- ically adjusts the inverse diffusion process to align with the observation's 's null space. Despite their novelty, these approaches are limited addressing linear degenerations and often result in low-fidelity images in practical applications. To improve upon these limitations, subsequent innovations like DPS (Chung et al., 2022) and IIGDM (Song et al., 2022) have been developed, focusing approximating the score of posterior distributions to achieve image restoration. However, their dependence on an unimodal estimation for clean samples has led to issues with precision. Additionally, the requirement
posterior sam-
*Correspondence W. Yao and Y. Deng

---

Published as a conference paper at ICLR 2025
Masked
RED diff
HRDIS (ours)
Low res
RED dEf-diff
HRDIS (ours)
Blurry
RED-diff
HRDIS (ours)
Figure 1: Comparison between RED-diff (Mardani et al., 2024) and our proposed HRDIS for image inpainting (left), super-resolution (middle) and nonlinear deblurring (right) tasks. RED-diff adopts the denoising process for regularization, which frequently Ieads to blurry and detail-lacking reconstructions, particularly when addressing intricate image inversion challenges. In contrast, HRDIS consistently outperforms RED-diff, showcasing notably improved results.
to compute the Jacobian matrix of the score network with DPS and TTGDM introduces significant computational demands and potential instability, posing challenges for efficient implementation.
RED-diff (Mardani et al., 2024) provides a novel perspective to addressing diffusion- based inverse problems, conceptualizing them within variational objective framework. This framework employs denoising process regularization technique to alleviate the ill ill-posedness of the inverse problem. Such an optimization- based approach circumvents the necessity for backpropagation through the score network, facilitating compatibility with existing efficient stochastic optimization methods. Despite these advantages, RED-diff encounters practical challenges, notably mode-seek tendency that often leads inversion results to converge on feature-averagen estimation (see Figure 1).
In this study, we investigate the underlying issues of RED-diff, particularly optimization challenges caused by gradient inaccuracy, which we attribute to its reliance single-step denoising estimation as a bootstrap target. Drawing upon principles for the distillation of diffusion models (Song et al., 2023b; Song& & Dhariwal, 2023), develop the Consistency Regularization (CR) technique. This approach enables more stable gradient provision, facilitating the regularization of the target signal for recovery, and obviates the need for computationally intensive simulation of ordinary differential equations (ODEs). Building on this foundation, we further introduce a comprehensive Hybrid Regularization (HR) framework. It provides unification of RED-diff and CR, amalgamating their benefits to enhance the performance of inverse problem solving markedly.
We introduce our framework, Hybrid Regularization for Diffusion- Inverse problem Solving (HRDIS), which serves as a versatile solution for addressing wide range of inverse problems. we evaluate HRDIS across various tasks, encompassing both linear (inpainting, super-resolution. and compressed sensing) and nonlinear (phase retrieval, high dynamic range, and nonlinear deblurring) scenarios, using two widely recognized benchmark datasets: FFHQ (Karras al., 2019) and ImageNet (Deng et al., 2009). Additionally, we demonstrate the framework's 's adaptability to various noise types. Our results show that HRDIS achieves performance that matches or exceeds state-of-the-art methods, while significantly reducing wall-clock time and memory usage.
2
BACKGROUND
2.1
SCORE-BASED DIFFUSION MODELS
The diffusion model (Ho et al., 2020; Song et al., 2020b) introduces a forward stochastic differential equation (SDE), denoted as (x+ (x).). which gradually perturbs the initial data Xo noise:
Paata into
dx f(x, ()dt + g())dw
(1)
where f(-,t) and gt) represent the drift and diffusion coefficients, respectively, and w is the standard Wiener process. We denote the marginal probability densities w.nt x, as (-). By selecting appropri- coefficients, can reparameterize x, 04xo + o,e, where NO T) and obtain a standard
2

---

Published as a conference paper at ICLR 2025
Gaussian density at 1. The forward SDE above is coupled with a probability flow (PF) ODE, which can be expressed as:
dx = f(x, f) - ( log px dt,
(2)
The above equation introduces a time- dependent score finctiin Vx log Pt (x),, which can be approxi- mated minimizing denoising score matching (Hyvärinen & Dayan, 2005) objectives using a neural network Co.
min E, - (leo + o,e,t) t) cll2.
After training, we can approximate Vx logg:(x) Ex(( f) and simulate such ODE in reverse time to sample from the underlying distribution Pdata-
Given that simulating the ODE can be computationally intensive, often requiring hundreds of iterations, consistency distillation (CD) (Song et al., 2023b) has recently been introduced to learn a direct mapping from noise to data. This mapping is parameterized as fo(::-) and trained to maintain self-consistency:
min f + + sg((f((x. t))
where O, and sg(() denotes the hop-gradient operator. Here, Xi+e and x₁ are two adjacent points on the same PF ODE trajectory. As the model easily learns the cases for small t. this loss function propagates the endpoint of the trajectory toward = 1, promoting one-step approximation of the ODE solution.
2.2 DIFFUSION MODELS FOR INVERSE PROBLEMS
The inverse problem arises in various applications across diverse domains. Formally, the general model for the inverse problem can be expressed as:
Axx)) n. y. E R" X. Rm
(5)
R" is the measurement operator and is the noise in the measurement process. When it is Gaussian noise, n (O I2), then we have p(y xp) (A(x)))) I²)). In practical applications, the collection of measurements often results in degradation relative to the original signal. Notably, when n m, the problem becomes ill-posed, necessitating the incorporation of regularizer or prior for deriving meaningful solution. Recent research has demonstrated that diffusion models can serve as aang-and-payy generative priors for sampling from posterior distributions, thereby obviating the need for extensive fine-tuning for specific tasks. To address this challenge, several classes of methods have been proposed. One widely used approach is the projection- (Song al., 2020b; Kawar al., 2022; Wang et al., 2022), which constrains the reverse-time denoising process to the subspace of measurements. However, these heuristics often fail to harmonize the generated samples with the known region (Lugmayr al., 2022).
where A(-):
methods
Alternatively, guidance- based methods have been proposed (Chung et al., 2022; Song et al., 2022) The gradient of the log-likelihood, i.e. Vx, log p(y xx). is approximated and integrated Specifically, in order to obtain p(y xx) = f p(y|))))((e)))) xx, dxx,, DPS (Chung al., 2022) approximate p(xo x²) with an Dirac delta distribution, i.e. px)) = 6(x, &oj)), where Xnit و E [xo|x|] = (x1 - o, (x,, t))/aa represents the single-step denoising estimation by Tweedie's 's formula (Robbins, 1992). Thus, have the following approximation,
into Eq. 2 for posterior sampling from the measurement
Vx, loggly>x,) = Vx. ly A A(x0)))
While these methods prove effective and typically produce clear reconstruction results, they not exempt from limitations. In particular, the gradient backpropagation process of the network ep is computationally expensive and prone instability.
Recently, Mardani et al. (2024) introduced an optimization- based method, RED-diff, with promising results. They tackle the problem by treating posterior sampling variational optimization task,
min D (xxogxx)
q
3

---

Published as a conference paper at ICLR 2025
Denoising Regularization (DR)
Consistency Regularization (CR)
struchune early stages
structure early stages
-- 0.9
0.7
0.5
11.3
0.1
2- 0.9
0.7
0.5
0.3
0.1
Pojt
"Incoraisten" prediction
"Conslatint" prediction
E+
E++
2e
6+
Figure 2: Comparison of intermediate denoised images Aoj between DR and CR (See. 3.2). The top row displays equally spaced visualization as progresses from to O. The bottom row presents Pooe across 5 consecutive timesteps. CR demonstrates more consistent generation and captures intricate features earlier stages than DR. The task is shown 16x super-resolution.
where q _ NN, p² 1) the variational distribution. The aforementioned variational objective can be further simplified as:
min ly - A(011++ Ee.ellellllssu t) c|12]
(8)
reconstruction
regularization
where 14 = 04z + OrE and Pt is the weighting function that depends on the timestep. The first term serves as a reconstruction term, ensuring that the reconstruction results align with the observed data. Meanwhile, the second term resembles the score matching objective, serving as form of regularization to alleviate the ill-posed nature of We designate this technique as Denoising Regularization (DR). Mardani et al. (2024) also demonstrate that the gradient of DR can bypass the need for the Jacobian matrix of Eg. Consequently, the resulting gradient can be expressed as:
V LDR V A(1)) (e,e(ue(eo((,,) -e),
where Mardani et al. (2024) chooses = o. /r, to balance the two terms. At this juncture, we can integrate the above gradient into an off-the-shelf stochastic optimizer and utilize the optimized as the reconstructed signal. RED-diff boasts memory efficiency, circumventing the need for backpropagation within the neural network. Also, offers the advantage of utilizing the degraded image an initial point for optimization, eliminating the necessity of starting with Gaussian noise. However, it is notable that RED-diff encounters challenges related mode collapse, resulting in blurry outcomes and limited diversity. Furthermore, the convergence of RED-diff typically requires thousands optimization steps.
METHODS
In this section, we first address the practical challenges in Denoising Regularization (DR) (Sec. 3.1). Building these findings, introduce Consistency Regularization (CR) alternative approach to overcome the limitations of DR (Sec. 3.2). We then present a unified framework, termed Hybrid Regularization (HR), which seamlessly integrates DR and CR as complementary techniques. Through a comprehensive analysis, we demonstrate how HR effectively leverages the strengths of both DR and CR methodologies to achieve superior performance (Sec. 3.3).
3.1 ANALYSIS OF GRADIENTS IN DR
Mardani et al. (2024) point out that the gradient of the regularization term in Eq. 9 can also be interpreted as the difference between the predicted clean image and variable p. By performing algebraic manipulation, derive it as follows:
wr(eo(me,) t) - = (P: o,E)/a: - (pr o,eo(pr, t))/a
p -ojo v, 1| sg(Aop)))2.
(10)
where Pooe A E[u|μ]] represents the single-step denoising estimate at timestep t. This derivation clarifies the mechanism of Denoising Regularization (DR), with Age serving bootstrapped
4

---

Published as a conference paper at ICLR 2025
ground truth. Regularization is achieved by constraining i to align with Pojz during optimization, thus ensuring it remains within the distribution of clean data.
Despite the effectiveness of this approach, DR has notable limitations. Firstly, the bootstrapped ground truth Poje is highly sensitive to μt, and the stochasticity of noise perturbations introduces significant uncertainty, illustrated in Figure 2. This often results in optimization with inconsistent ground truths, leading to feature- averrgee reconstructions. Secondly, it is well established that single-step denoising outputs from diffusion models are often suboptimal, typically lacking in detail and teggerrreunncy components, This limitation further diminishes the performance of RED-diff, resulting in less precise outcomes.
3.2
CONSISONSI FOR INVERSE PROBLEM SOLVING
Given the uncertainty and imprecision of DR, the deterministic process can be employed to achieve more consistent and accurate regularization. A straightforward approach is applying reverse ODE inversion (Su et al., 2022) to encode μ into noisy data, eliminating the need for random perturbations. Altematively, we can employ a multi-step solver to obtain a more accurate estimate of Aoj (Tang et al., 2023). While these methods improve estimation accuracy, they come with a significant trade-off in terms of computational cost. Fach iteration of gradient descent requires resource-intension inversions and multi-step solver executions, substantially increasing the overall computation time. The added computational burden can be prohibitive, particularly in applications where efficiency is crucial for solving inverse problems.
We observe that the above process can draw inspiration from recent advances in consistency distillation (CD) (Song al., 2023b), which enable efficient approximation of diffusion ODE without the need for intensive simulations. CD enhances self self-consistency by minimizing the difference of outputs between two adjacent points (e.g., and + e) on the ODE trajectory (Eq. 4). Given that the output at t is more accurate than at t + the model can iteratively propagate the trajectory endpoint back to = Motivated by this concept, we present Consistency Regularization (CR) as the difference between two adjacent outputs. The gradient of this regularization term is expressed as follows:
V VpCck= Vully - A(p) + E., (ooitte - Aouk]
(11)
where Poje+== B[ps)ette] The points Mt+e and μ4 are adjacent, and μr can be computed using the Euler solver as j = aeAo + ( t++ The key idea behind Eq. 11 is as follows: the clean image predicted at t has a higher fidelity compared to that at t - E. By progressively incorporating more complex features through the difference between these two estimates, the optimized variable incrementally converges toward the real image at /. O. We can further simplify the regularization term to operate within the noise domain.
Pojo+e
Mane Pojete (μr oreo(ue,
(Bring in pe)
= (e(co((ttt) co co t +)).
(12)
Note that the gradient computation outlined above necessitates two network inferences. To reduce computational costs, can adopt the descending sampling strategy for timesteps as proposed by Mardani et al. (2024), using the noise prediction from the previous step Eg (Hutee t + z). As shown in Figure 2, we observe that CR indeed produces more consistent Poje compared to DR, and produces intricate structural features at an early stage of optimization.
Distorted
CR (Eq. 11)
HR (Eq. 14)
3.3 HYBRID REGULARIZATION
The CR method provides sharper and more diverse restora- tion results; however, we have observed that it occasion- ally produces oversaturation and artifacts, as illustrated in Figure 3. We hypothesize that this issue arises from the lack of randomness in the noise sampling process within CR. Unlike DR, which introduces fresh noise during the optimization process, CR relies on the noise estimation from previous steps. The importance of random noise in enhancing the robustness and quality of generation has been extensively emphasized in the existing literature
Figure 3: Demonstration of the artifacts and oversaturation present in CR. We effectively mitigate these artifacts by in- corporating hybrid noise.
5

---

Published as a conference paper at ICLR 2025
diffusion models (Karras et al., 2022; Xu et al., 2023; Nie al., 2023). These findings motivate us to explore the possibility of devising unified framework that remains compatible with these regu- larization methods, thereby harnessing the strengths of both approaches. In the following, inspired by the study of solvers for diffusion models, we propose a general framework that we dub Hybrid Regularization (HR).
The widely used DDIM solver (Song et al., 2020a) can generate sample x, from x using the following expression:
+ Vo - +
Xx = at Xoate
3,E
(13)
predicted Xo
deterministic noise
random noise
The DDIM procedure begins with estimating 2g using the network, followed by its immediate projection onto the manifold where the noisy data resides at t, according to the forward noising process (Chung et al., 2023b). Note that the noise
be categorized into two parts: deterministic
and random noise. Their total variance (vog 8 2 + B8 = =o, where 3r governs the stochasticity of the sampling process. Specifically, when 3t = 0, the sampling process becomes deterministic. Returning to Eqs. 10 and 12, we observe that the disparity between the two gradients stems from the selection of noise: the former utilizes entirely random noise, while the latter employs deterministic noise predicted by the network. To leverage the benefits of both approaches, we introduce a coefficient 33 to generate hybrid noise éhybrid, maintaining constant total variance. Thus, the gradient can be formulated as follows:
V # CHI
Vully - A(1)1 + E,, (e -nyybia)l, V BEo +e) + VBe
(14)
Ehybrid
We denote the technique described above as Hybrid Regularization (HR), which constitutes a unified formulation. It is evident that both DR and CR represent extreme cases, where / takes values of 1 and 0, respectively. We assume that there exists a sweet spot that can effectively balance these two aspects. the ablation study detailed in Sec. 4.4, we observe that a value 0.2 generally sufficient for our method to perform well across wide range of experiments, obviating the need for task-specific tuning. Unless otherwise specified, we use 3 = 0.2 as the default setting.
Intuitively, hybrid noise exhibits stronger correlation with e (Mr, () compared to entirely random noise. Consequently, HR demonstrates reduced gradient variance, thereby promoting more efficient and stable optimization. Meanwhile, incorporating small degree of stochasticity aids in contract errors accumulated during the inversion process. Another key parameter is the timestep shift E, for which we found that value around the 10-2 order of magnitude works well (Appendix C.8). Our final approach, named Hybrid Regularization for Diffusion-based Inverse problem Solving (HRDIS), presented Algorithm 1, with the Adam optimizer (Kingma & Ba, 2014) employed the default.
Algorithm 1 Sampling procedure for HRDIS.
Input: observation y. measurement operator A(-), number of iterations N, timesteps sampling strategy {s,, )N pretrained model cg(:, )) B,
1: Initialize μ + A-1 (y)
2: for n 1,
do
3:
t +
4: if
then Initialize hybrid noise Chybrid N(0
5:
6:
end
7:
Forward perturb μ4 + arhe + Ehybrid
8:
Calculate gradient d, + Vply A(H)) + wi wr(ee((ee)) - Caybria)
9:
Optimize mean μ <- AdamUpdate dd,
10: Sample fresh noise e ~ A00 I)
11:
Calculate byddid noise Ehybrid f VI (He, + vBe
12:
end for
Output: f

---

Published as a conference paper at ICLR 2025
DDS
y
Conjugate gradient
(eees. f)
Gaussian noise
Xx+1
X
X
Reconstruction
HRDIS
Adam update
(1)
(1) Eg (Ug ..t)
Flexible initialization
H 10
NN
Reconstruction
Figure Comparison of DDS (Chung et al., 2023b) (top) and HRDIS (bottom) graph models. HRDIS maintains additional variable that is optimized during the process, enabling flexible initialization. The superscript (nz) denotes the optimization step.
3.4 DISCUSSION
In this subsection, we discuss the differences between the proposed HRDIS framework and existing approximate posterior sampling methods, focusing DDS (Chung al., 2023b) representative example. We compare the graph model of HRDIS with that of DDS Figure 4. DDS can be understood as approximate posterior sampling. achieved by gradually guiding the unconditional DDIM sampling process. DDS consists of three main steps: 1) Predict Koje from xs. 2) Solve the optimization problem: ming, llly A& |1 11 Roje using the conjugate gradient (CG),
3) Calculate x;-1 using DDIM.Througe these steps, DDS iteratively refines the sampling trajectory Xt, xt.1........ Xy by solving the subproblem in Step 2. In contrast, HRDIS builds on the RED-dit framework and formulates sampling as a stochastic optimization. A key difference is that HRDIS introduces and maintains an additional variable μ. initialized as (y), throughout the optimization. This variable is updated using gradients V,LCHR with an off-the-shelf Adam optimizer.
These fundamental differences result in notably distinct sampling behaviors. DDS gradually transi- tions from noise to reconstruction. In contrast, HRDIS exhibits an evolution that bridges the degraded image and the reconstruction, reflecting its unique generative dynamics (see Figure 17).
EXPERIMENTS
Does our proposed HRDIS effectively mitigate the issue of generating blurry images, observed in RED-diff (Mardani et al., 2024), while simultancously producing sharp and diverse reconstruction outcomes? Q2: How does the performance of HRDIS compare that of state-of-the-as diffusion model-based inverse problem solvers, such as LIGDM (Song et al., 2022), FPS (Dou & Song, 2024), among others? Furthermore, what is the computational efficiency of our approach compared to these alternatives? Q3: What the optimal choice for the hyperparameter 3 in regulating stochasticity within the framework of hybrid regularization?
We structure our experiments to address the following inquiries. Q1:
To address Q1, we examine the inversion results of challenging images within the dataset, assessing whether HRDIS can effectively restore high-frequency details compared to RED-diff. For Q2, we conduct quantitative comparison of quality and efficiency against state-of-the-ar algorithms. Lastly, to explore Q3, we conduct experiments to analyze the hyperparameter 3 selection.
We assess the effectiveness of the proposed HRDIS method across range of image restoration tasks, encompassing both linear and nonlinear inverse problems such as image inpainting, super-resolution, compressed sensing (CS), phase retrieval, high dynamic range (HDR) tasks, and nonlinear deblurring. Our experiments are conducted on the ImageNet 256 x 256 (Deng al., 2009) and FFHQ 256 x 256 (Karras et al., 2019) datasets, with results derived from 1k validation
Experimental Setup:
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative evaluation (FID, LPIPS, CA (unit:%) of solving linear inverse problems on
ImageNet 256x256-16 1k validation dataset. Bold: best, underline: second best.
Inpaint (10-20%)
Inpaint (20-30'3)
SR (x4)
CS (25%)
Method
LPIPS
LPIPS 4
FID
CA
LPIPS
FID 4
LPIPS |
FID
CA
DDRM (Kawar et al., 2022)
0.071
13.88
73.3
0.123
25.09
70.2
0.265
47.43
65.3
0.250
59.40
51.7
DPS (Chung et 2022)
0.146
27.65
69.0
0.182
36.64
64.6
0.197
37.35
67.7
0.168
34.20
68.2
IGDM (Song et 2022)
12.85
0.118
72.8
0.075
22.98
73,5
DDNM (Wang at, 2022)
0.075
14.06
72.0
0.106
26.61
67.9
0.251
47.15
61.9
0.247
33.16
58.4
GDP (Fei et al. 2023)
0.070
13.91
72.5
0.135
27.83
67.3
0,234
44,08
62.4
0.283
60.01
52.3
(Chung
20236)
0.067
0.130
67.9
0.270
55.1
FPS (Dou & Song, 20244)
0.065
13.07
73.9
0.121
24.90
70.5
0.189
34 88
67.8
0.124
32.76
67.0
RED-diff (Murdani 2024)
0.067
13.20
73.6
0.117
24.67
69.5
0.249
44.16
65.8
0.108
29.90
69.3
HRDIS (Ours)
0.054
10.94
74.7
0.096
20.10
71.3
0.137
33.01
68.8
0.059
22.04
72.3
images, consistent with previous research standards (Chung et al., 2022; Mardani et al., 2024). We
utilize pre-trained diffusion models from (Dhariwal & Nichol, 2021) and (Choi et al., 2021). Our comparative analysis includes benchmark techniques, namely DDRM (Kawar et al., 2022), DPS (Chung et al., 2022), IIGDM (Song et al., 2022), FPS (Dou & Song, 2024) and RED-diff (Mardani et al., 2024). Additional details on the experimental setup can be found in the Appendix B.2.
4.1 LINEAR INVERSE PROBLEMS
Distorted
Sample
Sample
Sample
RED-diff
We begin our experiments with image inpaint- ing, using the freeform masks provided by (Sa- haria et al., 2022a). Specifically, we apply 10%- 20% and 20%-30% masks on the ImageNet dataset and employ the more challenging 30%- 40% mask on FFHQ. For the super-resolution experiments, we utilize average pooling to per- form 4x downsampling on ImageNet and 16x downsampling on FFHQ In the compressed sensing (CS) task, we adopt an orthogonal sam- pling matrix applied the image blocks, with sampling rate of 25% for ImageNet and 10% for FFHQ We evaluate our results using two widely adopted metrics: Learned Perceptual Im- age Patch Similarity (LPIPS) (Zhang et al., 2018) and Fréchet Inception Distance (FID) (Heusel et al., 2017), computed on the test images. Additionally, for the ImageNet dataset, we report classifier accuracy (CA) using a pre-trained ResNet50 (He et al., 2016).
Figure 5: Diversity of reconstructions generated by HRDIS (Columns 2-4).
Tables 1 and 2 present the quantitative outcomes of linear inverse problem solving on ImageNet and FFHQ, respectively. It evident that HRDIS consistently ranks within the top two across all metrics and significantly outperforms other methods in numerous instances. Particularly noteworthy is HRDIS's exceptional performance in the inpainting and CS task, where it outshines all other techniques. While ITGDM demonstrates superiority in the super-resolution task, HRDIS closely follows as the runner-up. Following the quantitative analysis, we present a visual comparison with RED-diff in Figure 5. It evident that RED-diff frequently converges to the blurry images, which may not always align with the desired inversion result. In contrast, HRDIS exhibits the ability to generate diverse and plausible restored images across various random seed settings, offering more flexible and robust solution to the image restoration task. Figures 6 and 7 provides a qualitative comparison with other state-of-the-an inversion methods. Our observations indicate that DDRM tends to generate less realistic results and struggles to guide the reverse diffusion process toward achieving globally harmonious outcomes. Additionally, ITGDM occasionally exhibits instability, particularly evident in challenging samples, attributed to the necessity of backpropagating through the score network, resulting in failures in certain instances. In contrast, HRDIS can effectively recognize context and produce better restorations. See Appendix C.9 for more visualizations.
The wall-clock time and GPU memory of the different algorithms for the inpainting are reported in Table 3. Our observations reveal that the optimization step of our method can be order of magnitude fewer than that of RED-diff and FPS. In comparison to state-of-the tf- guidance- based methods (DPS, TIGDM), our approach demonstrates exceptional lightweightness and memory efficiency. Although
8

---

Published as a conference paper at ICLR 2025
Distorted
(Ours)
Distorted
(Ours)
Imagee inpaming
liperocl.too
Figure 6: Comparison of the proposed HRDIS with alternatives for inpainting (left) and super- resolution (right) on ImageNet 256 x 256 and FFHQ 256 x 256.
Table 2:
Quantitative evaluation (FID, LPIPS)
Table 3: Comparison wall-clock time and memory consumption, measured on single RTX 3090 GPU.
of solving linear inverse problems FFHQ 256x256-16 Ik validation dataset.
Inpaint (30-10%)
SR (x16)
CS (10%)
ImageNet
FFHQ
Method
LPIPS
FID |
L.PISS
FID
1.PIPS
FDD
Method
Time (sfing)
Memory
Time (sfing)
Memory
DDRM
0.099
23.91
0.324
91.11
0.535
130.2
DDRM
10
4.4G
2.5G
DPS
0.138
44.75
274
0.0844
33.25
HIGM
33
3.8G
6.1G
DONM
0.103
19:72
0.210
41.50
0.154
56.15
DDNM
13
4.6G
2.70
GDP
0.112
24.38
0.237
43.51
0.267
91.84
GDP
84
4.3G
2,4G
DDS
0.109
22.12
0,213
41.81
0.256
85.96
FPS
95
5.2G
3.3G
RED-dRT
0.086
17.55
0.286
73.39
0. 1.184
62.19
RED diff
82
4.3G
32
2.4G
HRDE (Ours)
0
15.42
0.213
40.97
0.088
35.27
HRDIS (Our))
11
1.3G
2/10
DDRM is also efficient, its applicability is limited to linear operator A(.), and its efficiency is not
guaranteed in scenarios where fast singular value decomposition is not feasible.
4.2 NONLINEAR INVERSE PROBLEMS
We evaluate the nonlinear inverse problem on the FFHQ dataset, begin- ning with phase retrieval. Given the inherent instability of phase recovery, we follow the strategy employed by Chung et al. (2022), utilizing an over- sampling rate of 2.0 and reporting the best results under four random seeds. Next, we address the High Dynamic Range (HDR) task, which incorpo- rates a truncation function to crop pixel values, represented as A(x) Clip (2x,
Table 4: Quantitative evaluation (FID, LPIPS) of solving nonlinear inverse problems FFHQ 256x256-1K validation dataset.
Phase retrieval
HDR
Nonlinear dehlurring
Method
LPIPS 1 FID
LPIPS 1.
FID |
LPIPS
FID
0.387
$4.64
0.407
84.64
0.279
dif
0.462
62.47
0.063
18.20
0.329
80.13
HRDIS (Ouns)
0.089
38.34
0.044
13.78
52.36
1,, we adopt
pre-trained network to simulate the blurring degradation operator as described Tran al. (2021).
Since methods such as DDRM, IIGDM, and FPS are not scalable to these nonlinear challenges, we compare our approach exclusively with DPS and RED-diff.
Table 4 presents the quantitative metrics for various solvers applied to the nonlinear inverse problem. Notably, the L.IPS scores for phase retrieval are significantly higher than those of the baseline, demonstrating a substantial improvement. Figures 7 and 8 provide qualitative results. We can see that the phase retrieval task, HRDIS generates reconstructions that closely match the reference image, whereas DPS and RED-diff yield unrealistic outputs. Additionally, DPS encountered challenges with the HDR task, failing to reconstruct the original image accurately. For the nonlinear deblurring task, the RED-diff method exhibited significant gradient variance, leading pronounced loss of detail and reduced fidelity. In contrast, our observations indicate that HRDIS consistently generates highly realistic samples, in these more challenging nonlinear scenarios.

---

Published as a conference paper at ICLR 2025
(Ours)
Distorted
DPS
RED-diff
HADIS (Ours)
Reference
Compeesee seniigg
Pase ealiin
Figure 7: Comparison of the proposed HRDIS with alternatives for compressed sensing (left) and phase retrieval (right) on ImageNet 256 256 and FFHQ 256 256.
(Ours)
Distorted
(Curs)
No-linear detluring
Nerinner HOR
Figure 8: Comparison of the proposed HRDIS with alternatives for HDR task (left) and nonlinear
deblurring (right) on FFHQ 256 256.
4.3 HANDLING VARIOUS NOISE STATISTICS
As an optimization-bu based framework, HRDIS is adept at managing various types of noise. In this subsection, we empirically evaluate HRDIS's performance under noisy conditions. To assess its robustness, we apply freeform mask for inpainting and introduce three types of noise: Gaussian, Poisson, and speckle, into the observations. The results, detailed in Appendix C.2, show that HRDIS consistently outperforms other methods in most scenarios. This demonstrates the framework's robustness and adaptability handling various measurement statistics.
4.4 ABLATIONS: INFLUENCE OF
We conduct an ablation analysis to investigate the impact of the param- eter 3 within the HRDIS method, focusing on inpainting and super- resolution using the ImageNet dataset, as well as nonlinear deblurring on the FFHQ dataset. The quantitative results in Figure 9 indicate that value of 0.2 consistently achieves lower LPIPS and FID scores across different tasks. As shown in Figure 11 (Appendix C. excessively high values of 3 result in over-smoothing, while very low values introduce noticeable artifacts. Notably, = 0.2 strikes an optimal balance, effectively reconstructing plausible figh-frequency details.
trpaintnggtmageeet
Superrssiueeetmmppeecc
20.5
15.0
72
84
08
2.0
1.8
1.0
02
0.335
1.0
0.2
0.
83
0.8
2.0
1.0
0.2
0.8
1.0
00
022
04
08
10
Figure 9: Quantitative results for varying values of B {0, 0.5,
CONCLUSION
This paper addresses the challenges associated with solving diffusion-baase inverse problems using Denoising Regularization (DR). tackle these challenges, we introduce the Consistency Regu- larization (CR) method, which effectively mitigates the issue of inaccurate gradient estimations. Additionally, we explore the integration of hybrid noise, resulting in the development of a unified HRDIS framework that fosters synergy between the two regularization techniques. The proposed framework is versatile, making it applicable to both linear and nonlinear inverse problems, as well accommodating diverse measurement statistics. Comprchensive experimental evaluations demonstrate that HRDIS not only surpasses current state-of-thed -te--at art methods but also maintains high computational efficiency.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62325101, 6203 1001, and also supported by the National Key Laboratory of Unmanned Acrial Vehicle Tech- nology in NPU under Grant WR202403, and supported by the Fundamental Research Funds for the Central Universities.
REFERENCES
Gabriel Cardoso, Yazid Janati El Idrissi, Sylvain Le Corff, and Eric Moulines, Monte carlo guided diffusion for bayesian linear inverse problems. arXiv preprint arXiv:2308. 079833 2023.
Joseph Cho, Fachrina Dewi Puspitasari, Sheng Zheng, Jingyao Zheng, Lik-Hang Lee, Tae-Ho Kim, Choong Seon Hong, and Chaoning Zhang. Sora as an agi world model? a complete survey on text-to-video generation. arXiv preprint arXiv:2403.0555 240000533. 2024.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Condi- tioning method for denoising diffusion probabilistic models. arXiv preprint 2it:::2008.023388 2021.
Hyungjin Chung, Jeongsol Kim, Michael T Mccann, Mare L Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.1400 2022.
Hyungjin Chung, Jeongsol Kim, Sehui Kim, and Jong Chul Ye. Parallel diffusion models of operator and image for blind inverse problems. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6059-6063 2023a.
Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Fast diffusion sampler for inverse problems by geometric decomposition. arXiv preprint arXiv:2303. 057554, 2023b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 8894,, 2021.
Zehao Dou and Yang Song. Diffusion posterior sampling for linear inverse problem solving: A filtering perspective. In The Twelfth International Conference on Learning Representations, 2024.
Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9935-9946, 2023.
Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. arXiv preprint WrXiv:2303.03543 2023.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update nule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8644 2022.
11

---

Published as a conference paper at ICLR 2025
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pp. 8867-8887. PMLR. 2022.
Aapo Hyvärinen and Peter Dayan, Estimation of non- normaallle statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in Neural Information Processing Systems, 35:26565-265777, 2022.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593- 23606, 2022.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXi:::412.69966 2014.
Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Luciddreamer: Towards high-fidelity text-to-3d generation via interval score matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6517-6526, 2024.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool.
Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11461-11471, 2022.
Morteza Mardani, Jiaming Song, Jan Kautz, and Arash Vahdat. A variational perspective solving inverse problems with diffusion models. The Twelfth International Conference on Learning Representations, 2024.
Naoki Murata, Koichi Saito, Chich-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Yuki Mitsufuji. and Stefano Ermon. Gibbsddrm: A partially collapsed gibbs sampler for solving blind inverse problems with denoising diffusion restoration. In International Conference on Machine Learning. pp. 25501-25522. PMLR, 2023.
Shen Nie, Hanzhong Allan Guo, Cheng Lu, Yuhao Zhou, Chenyu Zheng, and Chongxuan Li. The blessing of randomness: Sde beats ode in general diffusion-based image editing, arXiv preprint arXiv:2311.01410. 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Herbert E Robbins. An empirical bayes approach to statistics. In Breakthroughs in Statistics: Foundations and basic theory, pp. 388-394. Springer, 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems. 36, 2024.
Salva Rühling Cachay, Bo Zhao, Hailey Joren, and Rose Yu. Dyffusion: A dynamics-informed diffusion model for spatiotemporal forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pp. 1-10, 2022a.
12

---

Published as a conference paper at ICLR 2025
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36443 2022b.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308. 16512, 2023.
Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08123, 2023a.
Jaaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse guided diffusion models for inverse problems. In International Conference Learning Representations, 2022.
Yang Song and Prafulla Dhariwal. Improved techniques for training consistency models. arXiv preprint 1r.iv:2300.4411 2023.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020b.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with score- based generative models. arXiv preprint arXiv:21/1.08005 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ily Sutskever. Consistency models. arXiv preprint arXiv:2303.01469, 2023b.
Xuan Su. Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. arXiv preprint arXiv:2203.08382, 2022.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 2023.
Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1956-11965. 2021.
Brian L Trippe. Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206.00 041 19, 2022.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.004900 2022.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, -209 2023.
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv: 04560, 2018
Kevin E Wu, Kevin K Yang, Rianne van den Berg. Sarah Alamdari, James Y Zou, Alex X Lu, and Ava P Amini. Protein structure generation via folding diffusion. Nature Communications, 15(1): 1059, 2024a.
13

---

Published as a conference paper at ICLR 2025
Luhuan Wu, Brian Trippe, Christian Naesseth, David Blei, and John P Cunningham. Practical and asymptotically exact conditional sampling in diffusion models. Advances in Neural Information Processing Systems, 36, 2024b.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923) 2022.
Tongda Xu, Ziran Zhu, Dailan He, Yuanyuan Wang, Ming Sun, Ning Li, Hongwei Qin, Yan Wang, Jingjing Liu, and Ya- Qin Zhang. Consistency models improve diffusion inverse solvers. arXiv preprint arXiv:2403. 12063, 2024.
Yilun Xu, Mingyang Deng, Xiang Cheng, Yonglong Tian, Ziming Liu, and Tommi Jaakkola. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 76836806-76838, 2023.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large- scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03360 2015.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018.
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiezhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising diffusion models for plug-and-play image restoration. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1219-1229, 2023.
14

---

Published as a conference paper at ICLR 2025
A DISCUSSION
A.1 IMPACT STATEMENT
Our proposed framework presents novel approach to addressing inverse problems by applying diffusion models. Given its versatility and computational efficiency, we anticipate that our framework will benefit the relevant research community. However, it is crucial acknowledge the potential misuse of our approach in generating deceptive or malicious content. Additionally, reliance on generative techniques like diffusion models raises concerns regarding inherent biases present in the vast datasets utilized for training. These biases have the potential to manifest in the generated outputs. It may be necessary establish ethical guidelines and regulations to ensure responsible development and deployment of relevant technologies.
A.2 LIMITATIONS AND FUTURE WORKS.
While our proposed framework demonstrates efficacy in addressing both linear and nonlinear sce- narios, its primary limitation arises from the requirement to ascertain the form of the measurement operator or the computational procedure. Consequently, an enticing avenue for future exploration is the extension of our approach to tackling blind inverse problems (Chung et al., 2023a; Murata et al., 2023), where such prior knowledge is unavailable. We provide preliminary experiment in Appendix C.7. Furthermore, considering the prevalence of diffusion models operating within latent spaces (Rombach et al., 2022), a promising direction for advancement involves extending our methodology to encompass latent diffusion models. Finally, since HRDIS involves stochastic optimization for non-convex problems, its convergence behavior remains open theoretical question and warrants further investigation.
A.3 RELATED WORKS
Recent research has delved into the utilization of diffusion generation for solving inverse problems preemptively, Earlier studies (Song & Ermon, 2019; Song et al., 2020b; Lugmayr et al., 2022) focused on substituting observational components within the generation process to facilitate tasks like image inpainting. (Song et al., 2021) applied this technique to medical image reconstruction, while DDRM (Kawar et al., 2022) extended it to address more general degeneracies through singular value decomposition. Meanwhile, ILVR (Choi et al., 2021) achieved reference- generation using low-pass filter. DDNM (Wang et al., 2022) tackled inverse problem-solving by refining only the zero space during generation. Despite these advancements, these methods often remain confined to linear degeneracies and struggle with producing highly realistic generation results.
Subsequent research, such DPS (Chung et al., 2022) and IIGDM (Song et al., 2022), has aligned the generation process with observations by incorporating a guidance term of likelihood, resulting in clearer predictions. However, drawback of these approaches is the computational expense and instability associated with backpropagating through the score network. Additionally. Zhu et al. (2023) combined diffusion modeling with Half--uadratic-SSlittie
for plug-and-play image restoration. There has also been a series of works (Trippe et al., 2022; Wu et al., 2024b; Dou & Song, 2024; Cardoso et al., 2023) has also linked a posteriori sampling of diffusion models to sequential Monte Carlo. These methods typically maintain multiple particles during the generation process and therefore require more memory. PSLD (Rout et al., 2024) and ReSample Song al. (2023a) explored the use of latent diffusion to solve the inverse problem. More recently. RED-diff (Mardani et al., 2024) offers new perspective on diffusion- based inverse problem solving by modeling it as a variational problem. Despite its efficacy, RED-diff suffers from mode-collapse. In response to this limitation, we introduce the HRDIS framework this paper, building upon RED-diff to significantly enhance the accuracy and efficiency of image inversion. Xu et al. (2024) discusses the of consistency models to improve DPS, although this approach may be limited by the fact that consistency model checkpoints not widely available.
Some text-to-3D techniques (Poole et al., 2022; Shi et al., 2023; Liang et al., 2024: Wang et al., 2024) also pertinent to our work, they commonly optimize the score distillation loss to generate 3D assets from a 2D prior. This loss bears resemblance to RED-diff. However, our approach diverges in its objective. While text-to-3D efforts focus lifting 2D prior to 3D, our focus lies in leveraging
15

---

Published as a conference paper at ICLR 2025
diffusion prior to solving inverse problems. Thus, methodology differs fundamentally in purpose. despite similarities in the optimization paradigm.
B ADDITIONAL METHOD DETAILS
B. ADDITIONAL JUSTIFICATION FOR CR
We aim to leverage deterministic probability flow to enhance Eq. 10, where the objective is closely linked the distillation of diffusion models, as both tasks involve fitting the endpoint of the PF ODE provided by the pre-trained diffusion model. However, nuanced distinction exists between them: Eq- 10 optimizes the image μt, whereas the distillation process involves optimizing the parameters of the distillation network (t)).. Specifically, the typical procedure for distilling pre-trained diffusion model involves simulating an inverse PF ODE collect X Pdata which yields training data for distillation. Subsequently, the following loss function is employed for distillation:
min (x,,, xol12.
(15)
To circumvent the need for extensive simulation, Song et al. (2023b) introduces an innovative alternative approach termed Consistency Distillation (CD). Their study reveals the self-consistency property within the ODE trajectories of the diffusion model, wherein points along the same trajectory correspond to identical initial points at t 0 (Figure 10 (a)). Therefore, they propose an indirect distillation method by minimizing predictions at adjacent times (e.g., and + e) for single-step generation,
min (Xi+er + - ( f))
(16)
where for (xe+te, +) and fx 1) can be viewed student and teacher predictions, respectively. Since the boundary condition (xn, 0) = Xp is satisfied, the prediction fx,(,, 1) is closer to the ground truth than that + s. Through iterative propagation, the model aligns the endpoint of the trajectory such that, for any t, f (xe, t) & Xo.
In the context of diffusion based inverse problem solving, we aim to penalize || - x 112 for regu- larization (Figure 10 (b)). However, obtaining the ideal 2n is often challenging, it may require inversion and solving the ODE (Su et al., 2022; Liang et al., 2024). Notably, adjacent points of l4 easier to obtain. Thus, we incorporate insights from CD, which rely the difference between outputs at two points along the same trajectory to iteratively refine μ.. This process enables us to indirectly align p with the endpoint of the ODE.
Noise
Forward Process
Consistency Distillation (CD)
Consistency Regularization (CR)
Reverse Process
Original objective (X,, t) xolli
Original objective
|| x l
X
Simulation free objective (X (x..)
Simulation free objective
sg(Pop Ame
Optimize
Optimize:
Self-consistency property
(a) Diagram for Self-consistenny property CD
(b) Comparison between CD) and CR (Sec. 3.2)
Figure 10: Schematic illustrating the motivation for Consistency Regularization (CR), (a) Consistency Distillation (CD) (Song et al., 2023b) is trained to map points on any ODE trajectory (gray dashed line) of diffusion models to the trajectory's origin in single step, ensuring the self-consistency property is maintained. (b) CR and CD share the same concept transforming the original objective into a simulation-free objective based on self-consistency. However, they differ in that the former optimizes μ while the latter optimizes parameter ip
16

---

Published as a conference paper at ICLR 2025
Table 5: Hyperparameter choice for the proposed method.
Problem
Inpainting
SR
CS
Phase retrieval
HDR
Nonlinear deblurring
N
150
100
300
500
100
100
1
1
0.5
1
0.02
A2
0.25
0.20
0.20
0.25
0.25
0.25
learning rate
0.1
0.5
0.5
0.7
0.1
0.1
B.2 EXPERIMENTAL SETUP
All experiments, including time and memory calculations, were conducted on single NVIDIA RTX 3090 GPU. We employed the Adam optimizer (Kingma & Ba, 2014) with momentum parameters set to (0.9, 0.99). The parameter 3. utilized in synthesizing the hybrid noise, remained fixed at 0.2 throughout our experiments. We also chose descending timestep from 1 to 0 as in (Mardani al., 2024). The denoiser weight is determined the inverse signal-to-nowe ratio (SNR), 0x oz/on.
In practice, we introduce two coefficients to balance the reconstruction and regularization terms,
VuCHR = A V lly A(r)) + AeBr,s,elwe,ce t) Chybria)]
(17)
For most tasks, optimizing the number steps in the range of = 100 ~ 150 produces satisfactory results. For more challenging degenerations, such compressed sensing and phase retrieval, use more steps to improve performance further. In addition, for the phase retrieval task, which is particularly sensitive to initial noise, found that starting with random noise for the first 250 steps, followed by hybrid noise for the remaining 250 steps, significantly improves performance. Table details the selected hyperparameters of proposed method.
Below, we give the Pytorch-style pseudocode for our HRDIS implementation.
Add noise to perturbation ' muu
noise_xt xt torch. randa_aann__ (mu)
if step 0: hybrid_ncise noise_xt
else:
# Synthesize hybrid noise
hybrid_neise
peta) ssrrt deter_noise + beta. ()
noise_xt
xt alpha_t. () * mu (1 a alpha_t) sqrt () + hybrid_noise
call the dencising model to get the noise eet
with torch ()
et model (xt, t) de
Compute reconstruction and regularization terms e_obs y_0 A (mu)
clossoobs (c_obs**2) mean /
Moss noss_noise ss torch. mul ( (et hybrid_noisse _oise))_ .eeachh (),, mu) mean ()
Compute the weights of two items alpha_t).sqrf () / alpha_t.
snr iny
vt lambda
t lambda 2 snr inv
loss = w_t loss_noise + v_t Icss_obs
Adam step for 'mu'
loss boss.backwardd
optimizer.stepl
Store the noise 'et' for the next iteration deter_noise = et clonee
step +=
Listing I: Pseudocode of HRDIS for performing one optimization step.
17

---

Published as a conference paper at ICLR 2025
For the implementation of baseline methods, including DDRM (Kawar al., 2022), DPS (Chung et al., 2022), DDNM (Wang et al., 2022), GDP (Fei et al., 2023), DDS (Chung et al., 2023b), FPS (Dou & Song. 2024), and RED-diff (Mardani et al., 2024), provided by the respective authors. However, since no official implementation of ITGDM (Song et al., 2022) was available, faithfully reproduced it using the pseudo-code provided by the authors. For the hyperparameters, we primarily adhered to the original configurations, with slight fine-tuning achieve optimal performance. DDRM and IIGDM were configured with 100 steps, while DPS, FPS, and RED-diff required 1000 steps for effective performance. Additionally, we observed that increasing the number of particles in FPS yielded only marginal improvements, so we set 1 for our experiments.
utilized the official repositories
We use the default code and settings of each competitor from their official homepages as below.
DDRM (Kawar et al., 2022): https//gittbb.comm/ajjaaaaaaaaaa
DPS (Chung et al., 2022): http://ghhbbboomm////////////hhn
DDNM (Wang et al., 2022): hupp://gittubbcooo///yyyuu////Dh
GDP (Fei al., 2023): http://thhcccommmoyyooo///rrrttt
DDS (Chung et al., 2023b); https//giihub.comm///hhaaay///
FPS (Dou & Song, 2024): http://gghhabb.oo//cccoooooooofs SM2-2203
RED-diff (Mardani et al., 2024): https//giihbb.com////aWb//DDDDDd
C ADDITIONAL RESULTS
C.I IMPACT OF FOR HYBRID NOISE
We present the effect of varying 3 on the outputs of different inverse problems in Figure 11. When / set to 0, corresponding the CR described in Sec. 3.2, observe the generation of high-frequeney information, albeit accompanied by severe artifacts. When is set to 1, the result aligns with the DR used in RED-diff (Mardani al., 2024), producing blurry solution. Notably, when is set to 0.2, we identify a sweet spot that effectively balances the two aforementioned aspects, thus unlocking their potential for synergy.
Distorted
0.0
0.2
0.5
1.0
Figure 11: Restoration results for various inverse problems under different B.
18

---

Published as a conference paper at ICLR 2025
C.2 NOISY INVERSE PROBLEMS
In this section, we empirically verify the performance of HRDIS under noisy observation conditions. Specifically, simulate four types of noise: Gaussian noise with standard deviations of 0.05 and 0.1, Poisson noise with the noise level set to 1.0, and speckle noise with a standard deviation of 0.1.
As an optimization- framework, HRDIS accommodates noise without requiring modifications to the algorithm. For Gaussian noise with standard deviation of 0.05, we set the weight Y of the reconstruction term to 0.1, and for other noise types, we reduce A 0.05, The ablation study in RED-diff (Mardani et al., 2024) indicated that the method performs optimally when the time- dependent parameter wy is set to SNR, = 01/at. While this configuration effective for HRDIS under noiseless conditions, presents challenges in noisy settings, as the decreasing wt tends toward O, leading to insufficient regularization and potential overfitting to noisy data. To address this issue, we implemented a clipping mechanism for Wt (i.e., torch.clip
1 min = 2.0) ) ensuring that the SNR.
regularization term remains effective throughout the optimization process.
The quantitative results presented Tables 6 and 7 demonstrate that HRDIS consistently outperforms other methods across most scenarios. Furthermore, qualitative comparisons in Figures 7 15 reveal that JIGDM is particularly susceptible to noise, whereas DDRM, though effective at noise removal, often results in blurred reconstructions. Additionally, DPS exhibits instability and is prone to producing artifacts in the presence of Poisson and speckle noise. Overall, our method exhibits strong robustness various types of noise, confirming its effectiveness across different measurement statistics.
Table 6: Quantitative evaluation (FID, LPIPS, CA (unit:%) of solving inpainting problems under Gaussian noise on ImageNet 256x256 and FFHQ 256x256 validation dataset. Bold: best, underline: second best.
ImageNet 256x256
FFHQ 256x256
Method
LPIPS +
FID
CA f
LPIPS 1
FID
DDRM (Kawar et al., 2022)
0.109
30.1
70.1
0.138
45.1
IIGDM (Song al., 2022)
0.203
35.9
66.3
0.236
44.7
0.05
RED-diff (Mardani et al., 2024)
0.143
30.8
66.8
0.187
55,9
HRDIS (Ours)
0.129
29.4
70.4
0.153
31.1
DDRM (Kawar et al., 2022)
0.173
50.2
63.4
0.179
59.3
11GDM (Song et al., 2022)
0.412
62.8
52.0
0.440
69.6
0.1
RED-diff (Mardani et al., 2024)
0.278
47.9
60.3
0.340
91.5
HRDIS (Ours)
0.192
38.3
65.4
0.159
31.5
Table Quantitative evaluation under Poisson and speckle noise on FFHQ 256x256-1k validation dataset. Bold: best, underline: second best.
Poisson
Speckle
Method
LPIPS J
FID
LPIPS
FID +
DPS Chung al. (2022)
0.226
73.09
0.240
85.41
RED-diff Mardani et al. (2024)
0.192
58.96
0.232
78.72
HRDIS (Ours)
0.148
29.13
0.152
30.75
19

---

Published as a conference paper at ICLR 2025
Distorted
DPS
RED-diff
HRDIS (Ours)
Reference
Figure 12: Comparison of the proposed HRDIS with alternatives under Poisson noise, including DPS (Chung et al., 2022), RED-diff (Mardani al., 2024) and HRDIS (Ours).
Distorted
DPS
RED-diff
HRDIS (Ours)
Reference
Figure 13: Comparison of the proposed HRDIS with alternatives under speckle noise, including DPS (Chung et al., 2022), RED-diff (Mardani al., 2024) and HRDIS (Ours).
20

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
NGDM
HRDIS (Ours)
Reference
Y 0.05
Figure 14: Comparing methods for inpainting problem with Gaussian noise (7 = 0.05), including DDRM (Kawar al., 2022), IIGDM (Song al., 2022) and HRDIS (Ours).
Distorted
DDRM
NGDM
HRDIS (Ours)
Reference
y 0.1
Figure 15: Comparing methods for inpainting problem with Gaussian noise (7 = 0.1), including DDRM (Kawar al., 2022), JIGDM (Song al., 2022) and HRDIS (Ours).
21

---

Published as a conference paper at ICLR 2025
C.3 ABLATION FOR COMBINATION OF DR AND DDIM SOLVER
straightforward approach is to incorporate existing DDIM solvers (Song et al., 2020a) using hybrid noise to enhance DR. In this subsection, we evaluate this approach. The quantitative results, presented Table 8, indicate that incorporating DDIM into DR improves the performance of RED-diff, aligning with the conclusions from Sec. 3.1. While the random perturbations in RED-diff introduce uncertainty, the DDIM solver mitigates this issue to some extent. However, DR+DDIM still falls short compared to HRDIS. By providing smooth interpolation between DR and CR, HRDIS forms more flexible and synergistic framework. Figure 16 illustrates qualitative comparisons that further highlight the advantages of HRDIS.
Table 8: Ablation for the combination of DR and DDIM across these tasks: ImageNet256x256- Inpainting, ImageNee56x256-Supe Resolution, and FFHHQ256x26-Nonninea Deblurring.
Inpainting
Super Resolution
Nonlinear Deblurring
Method
LPIPS |
FID 4
CA 1
LPIPS L
FID
CA f
LPIPS 4
FID +
DR
0.117
24.6
69.5
0.249
44.2
65.8
0.329
80.1
DR+DDIM
0.113
23.5
70.4
0.141
35.6
65.7
0.295
73.7
HRDIS (Ours)
0.096
20.1
71.3
0.138
33.0
68.8
0.236
52.4
Distorted
DR+DDIM
HRDIS (Ouns)
Reference
Distorted
Figure 16: Ablation for combination of DR and DDIM solver.
C.4 COMPARISON WITH DDS
Here, we show some qualitative comparisons with DDS to demonstrate the advantages of the proposed HRDIS. Among them, Figure 17 shows the evolution of the two methods during the inversion process. Figure 18 shows the results of three linear tasks.
C.5 ABLATION FOR REGULARIZATION WITH CONSISTENCY MODELS
In this subsection, we present an ablation study, exploring the use of the output from the Consistency Model (CM) (Song et al., 2023b) as regular bootstrap target for regularization. Due to the lack of available checkpoints for benchmark datasets such ImageNet FFHQ at resolution of 256, conducted experiments using a checkpoint trained on the LSUN bedroom dataset (Yu et al., 2015). Figure 19 provides qualitative results from these experiments. Our findings indicate that incorporating the CM output as a bootstrap target slightly enhances the restoration gigh-frequency details, as offers more reliable target for regularization.
22

---

Published as a conference paper at ICLR 2025
DDS x
Distorted
DDS-
Reference
HRDIS
DDS-X
Distorted
DDS-Rait
Reference
HRDIS-H
Figure 17: Visualization of evolution in DDS (Chung et al., 2023b) and HRDIS (Ours) for image inversion.
Distorted
(Durs)
Distorted
DDS
HRDIS (Ours)
Reference
Figure 18: Qualitative comparison with DDS (Chung et al., 2023b).
23

---

Published as a conference paper at ICLR 2025
Osterted
RED def
RED dif-MM
HIZS (Cus)
Reference
Ditarted
RED-Fff
RED-dIf-CM
HIUS Dusii
Reforence
Figure 19: Qualitative results on inpainting using the output of the CM (Song et al., 2023b) regularization term.
C.6 FURTHER EXPERIMENTAL. RESULTS
We provide quantitative evaluations based the standard PSNR and SSIM metrics in Table 9,
Table 10 and Table 11.
Table 9; Quantitative evaluation (PSNR, SSIM) of solving linear inverse problems on ImageNet 256x256- validation dataset. Bold: best, underline: second best.
Inpaint (10-20%)
Inpaint (20-30%)
SR (x4)
CS (25%)
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR 1
SSIM
DDRM (Kawar al., 2022)
26.79
0.919
23.74
0.860
26.00
0.743
24.34
0.656
DPS (Chung al., 2022)
24.35
0.812
21.86
0.765
24.81
0.710
24.53
0.673
TIGDM (Song et al., 2022)
22.98
0.896
20.18
0.825
25.23
0.731
28.58
0.740
DDNM (Dou & Song. 2024)
22.58
0.894
22.54
0.865
21.39
0.529
23.39
0.588
DDS (Dou Song, 2024)
22.02
0.807
19.26
0.815
22.25
0,580
23.02
0.569
FPS (Dou Song, 2024)
24.81
20.76
0.793
24.90
0.703
25.77
0.690
RED-diff (Mardani
2024)
26.60
23.45
0.869
25.89
0.746
0.669
HRRDI (Ours)
25.65
0.913
22.12
0.835
25.54
0.708
28.79
0.807
Table 10: Quantitative evaluation (PSNR, SSIM) of solving linear inverse problems on FFHQ
256x256-16 1k validation dataset.
Inpaint (30-40%)
SR (x16)
CS (10%)
Method
PSNR
SSIM
PSNR f
SSIM
PSNR
SSIM 1
DDRM
23.58
0.862
22.94
0.676
24.34
0.656
DPS
22.97
0.804
20.76
0.529
25.60
0.707
HIGMM
21.85
0.833
21.63
0.614
27.49
0.779
DDNM
20.43
0.816
20.83
0.574
25.13
0.708
DDS
20.07
0.810
20.77
0.570
22.39
0.609
FPS
22.17
0.828
20.95
0.553
25.86
0.715
RED-diff
23.78
0.867
22.64
0.654
26.58
0.761
HRDIS (Ours)
23.06
0.837
21.75
0.622
27.85
0,802
Table 11: Quantitative evaluation (PSNR, SSIM) of solving nonlinear inverse problems on FFHQ 256x256-16 1k validation dataset.
Phase retrieval
HDR
Nonlinear deblurring
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM 1
DPS
19.64
0.507
21.19
0.780
21.65
0.563
RED-diff
17.97
0.488
25.97
0.869
19.59
0.484
HRDIS (Ours)
30.08
0.814
27.51
0.891
23.06
0.597
C.7 BLIND INVERSE PROBLEM ON FEAL-WORLD SAMPLES
In this subsection, we initially explore the utilization of the proposed HRDIS for blind inverse problem-solving real-world images. To address the challenge of degradation, conducted
24

---

Published as a conference paper at ICLR 2025
additional experiments by incorporating learnable degradation model. Inspired by the approach presented in GDP (Fei et al., 2023), we assume a simple degradation model defined as:
y fx M =: (x),
where f is scalar and M a mask, both of which are initially unknown. These parameters, along with the image are optimized alternately using HRDIS. The optimization is performed as follows: Update f and M: These parameters are updated using the gradient:
Vi.llyy - A
2. Update μc: The image is updated using the gradient:
Vully (A)M(u))l ( Enybria]-
We performed preliminary experiments on real-world low-light images from the LOL dataset (Wei al., 2018). The results in Table 12 demonstrate that HRDIS achieves superior performance compared to GDP-rz/ (Fei et al., 2023), with the added advantage of reduced computational cost. Specifically, HRDIS requires only about 300 NFE, compared to 1000 NFE for GDP. Figure 20 provides qualitative results that illustrate the effectiveness of HRDIS in reconstructing high-qu sigh-qualtty images from degraded inputs. These findings highlight HRDIS's potential in addressing blind inverse problems effectively and efficiently.
Distorted
HRDIS (Ours)
GDP-x,
Reference
Figure 20:
Qualitative comparison of blind reversal problems.
Table 12: Quantitative evaluation blind inverse problem on the LOL dataset.
Method
PSNR
SSIM
FID
GDP-2t
7.32
0.57
238.92
GDP-zp
13,93
0.63
75.16
HRDIS (Ours)
14.25
0.65
74.89
C.8 ABLATION OF
To investigate the influence of E in the CR term, conducted an ablation study focusing inpainting and super-resolution tasks using the FFHQ dataset. The diffusion model operates over normalized time interval [0, 1), and we evaluated the performance of across several magnitudes.
25

---

Published as a conference paper at ICLR 2025
The quantitative results of the ablation study presented in Figure 21. We observed that performs optimally when it is around the order of 10 Key observations include:
When e is too small (10-9), the CR effect diminishes, resulting in blurred outputs.
When & is too large (10-)), the resulting image quality degrades due to the discretization emror
These findings highlight the importance of carefully selecting to achieve optimal performance. Figure 22 provides qualitative visualizations of the ablation study. These examples clearly demonstrate the impact of different values on the final output.
Inpainting (FPHQ)
Super-resolution (FFHQ)
0.27
0.100
0.26
0.25
0.095
0.24
22
20
19
0
17
16
10:3
10-7
10-1
10
10
Figure 21: Quantitative results for varying values of {0.001, 0.005, 0.007, 0.0.0.02.........000
0.02, 0.1).
0.001
0.01
0.1
Distorted
Figure 22: Visualization of the output of different E.
26

---

Published as a conference paper at ICLR 2025
C.9
ADDITIONAL FIGURES
In this subsection, we show additional qualitative results for HRDIS.
Distorted
DDRM
ΠGDM
HRDIS (Ours)
Reference
Figure 23: Comparing methods for the image inpainting problem on ImageNet 256 x 256, including DDRM (Kawar al., 2022), TIGDM (Song al., 2022) and HRDIS (Ours).
27

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
NGDM
HRDIS (Ours)
Reference
Figure 24: Comparing methods for the image inpainting problem on FFHQ 256 x 256, including
DDRM (Kawar al., 2022), JIGDM (Song al., 2022) and HRDIS (Ours).
28

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
MGDM
HRDIS (Ours)
Reference
2222
Figure 25: Comparing methods for the image super-resolution problem on ImageNet 256 x 256, including DDRM (Kawar al., 2022), ITGDM (Song al., 2022) and HRDIS (Ours).
29

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
ΠIDM
HRDIS (Ours)
Reference
Figure 26: Comparing methods for the image super-resolution problem on FFHQ 256 256, including DDRM (Kawar et al., 2022), IIGDM (Song et al., 2022) and HRDIS (Ours).
30

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
RGDM
HRDIS (Ours)
Reference
Figure 27: Comparing methods for the compressed sensing problem on ImageNet 256 x 256, including DDRM (Kawar et al., 2022), IIGDM (Song al., 2022) and HRDIS (Ours).
31

---

Published as a conference paper at ICLR 2025
Distorted
DDRM
ΠGDM
HRDIS (Ours)
Reference
Figure 28: Comparing methods for the compressed sensing problem on FFHQ 256 x 256, including
DDRM (Kawar al., 2022), ITGDM (Song al., 2022) and HRDIS (Ours).
32

---

Published as a conference paper at ICLR 2025
Distorted
DPS
RED-diff
HRDIS (Ours)
Reference
Figure 29: Comparing methods for the phase retrieval problem on FFHQ 256 x 256, including DPS
(Chung et al., 2022), RED-diff (Mardani al., 2024) and HRDIS (Ours).
33

---

Published as a conference paper at ICLR 2025
Distorted
DPS
RED-diff
HRDIS (Ours)
Reference
Figure 30: Comparing methods for the nonlinear HDR problem on FFHQ 256 256, including DPS
(Chung et al., 2022), RED-diff (Mardani et al., 2024) and HRDIS (Ours).
34

---

Published as a conference paper at ICLR 2025
Distorted
DPS
RED-diff
HRDIS (Ours)
Reference
EITE
STREA
Figure 31: Comparing methods for the nonlinear deblurring problem on FFHQ 256 x 256, including
DPS (Chung et al., 2022), RED-diff (Mardani al., 2024) and HRDIS (Ours).
35


Reviews:
[{"review_id": "0culrSYWpK", "reviewer": "Reviewer_Yt8A", "summary": "This paper builds on the concept of denoising regularization (DR) for solving inverse problems. Inspired by recent advancements like Red-Diff, which utilizes denoising score matching as a regularization term, the authors introduce a new approach termed consistency regularization (CR). CR adopts the central idea of consistency distillation (CD), which improves self-consistency by minimizing the difference between outputs at two consecutive reverse sampling points. However, the authors note that a direct application of CD can lead to oversaturation and artifacts due to insufficient randomness in the noise sampling process within CD. To address this, they propose a hybrid regularization (HR) framework, which incorporates hybrid noise modeling from the DDIM formulation. Experimental results suggest that HR achieves state-of-the-art performance in various inverse problem settings.", "questions": "- Ablation study: conduct ablation study where the DR formulation combined with hybrid noise model\n- Difference: please state the difference between the one-step DDS formulation and the proposed hybrid noise model based HR approaches. Aside from the use of the adjacent sanpling time, what is the difference?\n- Comparative study with DDS formulation", "limitations": "", "rating": 6, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The use of a hybrid noise model for score distillation is a notable advance compared to DR and CR.", "weaknesses": "Despite its contributions, the paper has several notable limitations. First, to substantiate HR's advantages over DR, a comparative ablation study incorporating the hybrid noise model into the DR framework should have been conducted. The reviewer suspects that HR's benefits stem from mitigating DR's mode-collapse behavior due to the hybrid noise model rather than the use of CR, a point the current manuscript and experiments have not sufficiently addressed.\n\nSecond, although the hybrid noise model may appear novel within the Red-Diff framework, the final formulation lacks originality. Specifically, the hybrid noise model and gradient step in equation (11) represent a simplified, one-step version of the decomposed diffusion sampling (DDS) framework [1]. In DDS, the denoised image, obtained through Tweedie’s formula, is updated using a multi-step conjugate gradient (CG) approach, after which hybrid noise is introduced within the DDIM framework. The authors appear unaware of DDS, as there are no comparisons against it.\n\n[1] Chung, H., Lee, S., & Ye, J. C. Decomposed Diffusion Sampler for Accelerating Large-Scale Inverse Problems. In The Twelfth International Conference on Learning Representations. 2024\n\nThird, the reviewer finds the authors' assertion about the growing interest in DR for inverse problems unconvincing. Posterior sampling remains predominant in achieving state-of-the-art results in inverse problems. Moreover, contrary to the authors' claims, recent approaches such as DDS bypass Jacobian computation by leveraging piecewise linear manifold structures. Therefore, this assertion appears unsupported.\n\nFinally, the literature review and experimental comparisons with the latest approaches such as DDS,  are insufficient, lacking several significant recent works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U41X6Ma8Q7", "reviewer": "Reviewer_tzUN", "summary": "This paper proposes a simple and effective hybrid regularization scheme for solving inverse problems using pretrained diffusion models. The authors first analyze the potential issues of existing optimization-based inverse problem solving methods, such as Denoising Regularization (DR). Based on the analysis, they propose Consistency Regularization (CR) to address the inaccurate gradient issues inspired by Consistency Distillation (CD). Then, a Hybrid Regularization (HR) method is further proposed to benefit from both the DR and CR. The proposed method is evaluated on FFHQ and ImageNet datasets, with comprehensive experiments on different distortion and inverse problems. The results show that the proposed method is efficient and can reconstruct better details than the state-of-the-art methods.", "questions": "- Would the idea of Consistency Regularization (CR) lead to less diversity in the inversed image? As shown in the original consistency distillation paper [Song et al. 2023], the amount of variations in the generated images  (measured by recall [Kynka¨anniemi et al. 2019]) can be reduced due to consistency distillation.\n- As discussed in the \"Weakness\" section above, could the authors provide more insight on why the proposed method can lead to over-sharpened or noisy images?\n- The equation of consistency distillation has a moving average (EMA) term in the original paper [Song et al. 2023], denoted by $\\theta^{-}$. Is it also the case in Equation 12 or the same network is applied for two different time steps?\n- Equation 14 shows that the regularization requires 2x network inference denoted by epsilon_theta, but this does not slow down the optimization process. Is it because the effect of reducing the number of iterations is more significant than the extra computation cost?\n- I am a bit confused about what should I expect in Figure 6, since I am not familiar with compressed sensing and phase retrievel. Given such highly distorted images, is it possible to recover the results that can match the reference so well? The same question applies to Figure 7 on the nonlinear deblurring results.\n\nSome minor comments:\n- In line 421-422, \"FFHD\" should be \"FFHQ\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed Consistency Regularization (CR) and Hybrid Regularization (HR) methods are simple and effective, and can be applied to both linear and nonlinear inverse problems.\n- The results look much sharper than RED-diff (e.g., Figure 1) thanks to the proposed CR and HR methods. Also, the results are overall outperforming the state-of-the-art methods.\n- The reference seems comprehensive. The authors even mention score distillation sampling (SDS) introduced in text-to-3D literature, which indeed shares some similarity with RED-diff and the proposed method.\n- The paper includes comprehensive results, comparisons and ablation studies that are important to understand the effect of the proposed method.\n- It is nice that the authors provide pseudo code in the appendix and code in the supplemental.", "weaknesses": "- The proposed CR method can lead to over-sharpened or noisy images, but it is not clear to me why this happens. To my understanding, this is not the case for image generation from noise based on the original consistency distillation paper [Song et al. 2023]. As shown in Figure 2 and Figure 3, the Consistency Regularization (CR) results do get better structures but also introduce artifacts and noise. Also in Figure 1 middle column, the images look a bit noisy even under the Hybrid Regularization (HR) method. In Figure 10 the problem seems more visible ($\\beta=0$). I would like to hear more insights on this if this is a potential issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DeDQMYhFpV", "reviewer": "Reviewer_HwgS", "summary": "This paper is an improvement over previous work called RED-diff, which is variational method for diffusion posterior estimation.   \n\nBy analyzing the weakness of RED-diff and drawing inspiration from consistency model, this paper proposes consistency regularization to replace the denoising regularization in RED-diff. \n\nBy analyzing the limitation of consistency regularization and drawing inspiration from DDIM, this paper further proposes hybrid regularization which combines denosing and consistency.  \n\nExperiments show the proposed method achieve great improvement regarding image quality and sampling speed.", "questions": "It is interesting that beta=0.2 is a common sweet spot for across different tasks, is there any insight on this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The writing follows a problem analysis and problem soving manner, which is very logical and frendly to readers.   \n\nThe improvement over previous works regarding image quality and speed is significant. The experiments are thorough, especially, different previous works are analyzed in detail regarding their image quality.  \n\nThe Consistency Regularization (CR)  and Hybrid Regularization (HR) are all well motivated by previous works on diffusion sampling, i.e., consistency model and DDIM etc.", "weaknesses": "eq. 10 is already derived in the previous work Mardani et al. (2024) in section 4.3, it might be better to clarify this.  \n\nThere exists a previous work which uses consistency model for diffusion posterior sampling ('Consistency Models Improve Diffusion Inverse Solvers') and should be discussed. What if we follow this paper to use consistency model directly in the original RED-diff framework?  Can this, to some extent, play the same role as Consistency Regularization (CR)?  \n\n[main concern] The consistency to the measurement is not evaluated and compared. This line of works solve inverse problems, shoud the consistency to the measurement also be a important metric in addition to the image reconstruction quality? For example, In DPS and RED-diff, PSNR between reconstruction and ground truth is reported, which can represent the consistency to measurement to some extent. I am not requiring achieving SOTA PSNR and SOTA FID at the same time. But it helps to have some fidelity evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fKgvwmF51P", "reviewer": "Reviewer_86ej", "summary": "This paper proposes a novel approach called Consistency Regularization (CR), which provides stabilized gradients without the need for ODE simulations. Building on this, we introduce Hybrid Regularization (HR), a unified framework that combines the strengths of both DR and CR, harnessing their synergistic potential. The approach proves to be effective across a broad spectrum of inverse problems, encompassing both linear and nonlinear scenarios, as well as various measurement noise statistics. The experiment is clear and convincing.", "questions": "1. It would be interesting to explore whether the method proposed in this paper can be extended to other inversion problems, such as image editing.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method combines the strength of both Consistency Regularization and Denoising processes with Regularization. \n2. The experimental results are promising, including non-linear problems.\n3. The writing quality is strong.", "weaknesses": "1. The method lacks comparisons with recent works such as DDNM [a] and GDP [b].\n2. Although the method addresses non-linear problems, it has not been evaluated on blind problems. The performance on real-world samples should be demonstrated to validate its practical effectiveness.\n\n\n[a] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model, ICLR 2023.\n[b] Generative Diffusion Prior for Unified Image Restoration and Enhancement, CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)