Diffusion Posterior Proximal Sampling for Image Restoration

Authors: hongjie  wu
mingqin  zhang
hong  kong
mengting  luo
ji-zhe  zhou
hu  chen
jiancheng 2024 lv
linchao  he
dongdong  chen
kunming  luo
  diffusion
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:07:05

Abstract: Diffusion models have demonstrated remarkable efficacy in generating high-quality samples.
Existing diffusion-based image restoration algorithms exploit pre-trained diffusion models to leverage data priors, yet they still preserve elements inherited from the unconditional generation paradigm.
These strategies initiate the denoising process with pure white noise and incorporate random noise at each generative step, leading to over-smoothed results.
In this paper, we present a refined paradigm for diffusion-based image restoration. Specifically, we opt for a sample consistent with the measurement identity at each generative step, exploiting the sampling selection as an avenue for output stability and enhancement. The number of candidate samples used for selection is adaptively determined based on the signal-to-noise ratio of the timestep.
Additionally, we start the restoration process with an initialization combined with the measurement signal, providing supplementary information to better align the generative process.
Extensive experimental results and analyses validate that our proposed method significantly enhances image restoration performance while consuming negligible additional computational resources.

Main paper: 
 introduction: 
 Diffusion models [19, 50, 53] have gained broad attention due to the powerful ability to model complex data distribution, and have been applied to a wide range of tasks such as image generation [13, 33, 37] , molecule generation [12] , and natural language processing [30] . Recent research has demonstrated that pre-trained unconditional diffusion models can be effectively employed to address image restoration problems [9, 26, 47, 49, 54] to leverage rich priors in a plug-and-play fashion, and achieve significant advancements.However, for diffusion-based image restoration algorithms, they still retain a process inherited from unconditional generation. In particular, (i) these methods start generating the image that corresponds to the measurement with a white noise as initialization, and (ii) a fully stochastic noise is introduced at each step of the generation process. We argue that this paradigm is inappropriate for solving image restoration tasks. Firstly, randomness enriches the diversity of the generated samples in unconditional generation [31, 56] . However, for image restoration tasks where the identity information of the measurement needs to be preserved [57] , randomness DPS Ours (𝑛=2) Ours (𝑛=100) ReferenceFigure 1 : Examples of the super-resolution (×4) task to illustrate the efficiency of our method, where 𝑛 denotes the number of candidate samples, and DPS also refers to the case where 𝑛 = 1.leads to uncontrollable generation outcomes [4] . Secondly, in the generative process of diffusion-based model, the current state is randomly sampled [8, 47] from the predicted distribution without any correction. Considering the Markovian reverse process, the sampling quality of the next state heavily depends on the current sampling result. If the current result deviates considerably from expectations, the subsequent generative process will encounter a significant exposure bias problem [29, 35] . As a consequence, randomness introduces fluctuations and instability into the generative process, ultimately leading to over-smoothed results.In this paper, we advance toward a more refined paradigm for solving image inverse problems with pre-trained diffusion models. Specifically, to ensure the sampling result is closely consistent with the measurement identity, we opt for the proximal sample at each step from multiple candidate samples, as opposed to making a random choice. By mitigating uncertainties, the sample consistently progresses toward the desired targets, resulting in a stable generative process and improved output quality. Moreover, we start the generation process with an initialization composed of both the measurement signal and white noise, rather than pure white noise. Consequently, the denoising process commences from a closer starting point, facilitating a faster convergence toward the desired result.We theoretically analyze that our approach reduces variance in comparison to existing state-of-the-art methods which introduce random sampling. And we conduct extensive experiments to demonstrate the superior restoration capabilities of our method across diverse image restoration tasks, such as super-resolution (SR), deblurring, and inpainting, with only a marginal additional cost. Furthermore, sufficient experimental analyses are shown to validate the effectiveness of our proposed strategy.In summary, our contributions are as follows:• We pioneer to improve the generation quality by exploiting sampling choice from the predicted distribution of each reverse step.  
background 2.1 denoising diffusion probabilistic models: 
 Denoising Diffusion Probabilistic Models (DDPMs) [19] is a class of models that incorporate a forward (diffusion) process and a corresponding reverse (generative) process. Consider a 𝑇 -step forward process, the noised sample x 𝑡 at time step 𝑡 can be modeled from the previous state x 𝑡 -1 :EQUATIONwhere N denotes the Gaussian distribution, and 𝛽 𝑡 is a pre-defined parameter increasing with 𝑡. Through reparameterization, x 𝑡 satisfies the following conditional distribution 𝑞(x 𝑡 |x 0 ) = N (x 𝑡 ;√ ᾱ𝑡 x 0 , (1 -ᾱ𝑡 )I), where ᾱ𝑡 = 𝑡 𝑖=0 𝛼 𝑖 and 𝛼 𝑡 = 1 -𝛽 𝑡 . On the other hand, the reverse process of DDPM is formulated by the transition kernel parameterized by 𝜃 :EQUATIONwhere the learning objective aims to minimize the KL divergence between the forward and backward processes. The epsilon-matching objective is typically set as:EQUATIONwhere x 0 is sampled from training data and x 𝑡 ∼ 𝑞(x 𝑡 |x 0 ). Once we have access to the well-trained 𝝐 𝜃 (x 𝑡 , 𝑡), a clean sample can be derived by evaluating the generative reverse process Eq. (2) step by step. Furthermore, one can view the DDPM equivalent to the variance preserving (VP) form of the stochastic differential equation (SDE) [50] . Accordingly, the epsilon-matching objective Eq. ( 3) is equivalent to the denoising score-matching [44] objective with different parameterization:EQUATIONThis paper focuses on solving the image restoration, or image inverse problems with unconditional diffusion model [8, 26, 47] . Our goal is to retrieve the unknown x 0 from a degraded measurement y: where A (•) : R 𝑑 ↦ → R 𝑛 is the degradation operator and n ∼ N (0, 𝜎 2 𝑦 I) denotes the measurement noise. The restoration problem can be addressed via conditional diffusion models by substituting the score function ∇ x 𝑡 log(x 𝑡 ) in the reverse-time SDE with the conditional score function ∇ x 𝑡 log(x 𝑡 |y), which can be derived by Bayes' rule:EQUATIONEQUATIONThe first prior term can be approximated by a well-trained score network. However, the analytic solution of the second likelihood term is computationally intractable, since there only exists an explicit connection between y and x 0 . To solve this dilemma, Chung et al. [8] propose diffusion posterior sampling (DPS) to approximate the likelihood using a Laplacian approximation:∇ x 𝑡 log 𝑝 (y|x 𝑡 ) ≃ ∇ x 𝑡 log 𝑝 (y| x0|𝑡), where x0|𝑡 is the denoised estimate via Tweedie's formula [14, 51] . Consequently, the generative distribution 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 , y) can be modeled as:EQUATIONThe mean of Gaussian distribution 𝝁 𝜃 (x 𝑡 , 𝑡, y) is obtained by:EQUATIONwhere x0|𝑡 = 1 √ ᾱ𝑡x 𝑡 -√ 1 -ᾱ𝑡 𝝐 𝜃 (x 𝑡 , 𝑡) , and 𝜆 𝑡 is a tunable step size. 
diffusion posterior proximal sampling 3.1 random sampling induces uncertainties and error accumulation: 
 Existing approaches [9, 26, 47, 49, 54] address image restoration or inverse problems following a process derived from unconditional generation. Specifically, the process (i) initiates the denoising process with pure white noise, and (ii) incorporates random noise at each reverse step. We contend that randomness is unsuitable for restoration problems that demand the preservation of measurement identities, such as super-resolution or deblurring. Furthermore, the cumulative impact of random noise at each step results in the smoothing of output, thereby yielding low-quality generated samples. Recent investigations [16, 32] align with our findings, highlighting that randomness introduces instability and fluctuations, ultimately culminating in suboptimal samples.On the other hand, we observe a discrepancy in the inputs provided to the noise prediction network 𝝐 𝜃 . During the training phase, the network is fed with ground truth training samples. However, in the inference stage, the input x 𝑡 is randomly sampled from the predicted distribution. In cases where the predicted distribution is inaccurate, random sampling can exacerbate the deviation from the expected values, introducing substantial exposure bias [35] to the generative process. While the data consistency update ∇ x 𝑡 ||y -A x0|𝑡 || does offer mitigation by adjusting the sample to align with the measurement, it demands delicate design and specifically-tuned parameters [48] . And the optimal parameter values vary across datasets and tasks. Consequently, the selection of sampling choices holds significance as it directly influences the input to the network, thereby affecting the generation quality.To tackle the challenges posed by random sampling, in this paper, we propose to extract multiple candidate samples from the predicted distribution, and select the most proximal one [36] to our anticipated target. Our motivation comes from the following idea: considering x 0 is an ideal but unknown solution for the image restoration problem, the sample taken from the predicted distribution x 𝑡 -1 ∼ 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 , y) should be close to posterior 𝑞(x 𝑡 -1 |x 𝑡 , x 0 ), as it models the desired reverse process.The mathematical formulation of our selection process is given by:EQUATION)wherex 𝑖 𝑡 -1 ∼ 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 , y), 𝑖 ∈ [0, 𝑛 -1], and x * 𝑡 -1 is our proposed deterministic solution 1 via DDIM [46] with unknown x 0 (see supplementary material for detailed derivation):EQUATION)1 the symbol * means an ideal but unknown solution. Here𝐶 1 = √ 1 -ᾱ𝑡 -1 / √ 1 -ᾱ𝑡 and 𝐶 2 = √ ᾱ𝑡 -1 - √ ᾱ𝑡 √ 1 -ᾱ𝑡 -1 / √1 -ᾱ𝑡 are introduced for simplicity.The underlying motivation of our approach is depicted in Figure 3 . Without changing the diffusion denoiser, our approach directs the sampling results toward a predefined target, mitigating the drawbacks induced by randomness. Moreover, the generative process remains stochastic, benefiting from the injected noise [1, 24] since it corrects prediction errors and imprecise parameter settings from previous steps. Consequently, our method converges more rapidly and attains superior results.However, accessing x 0 during inference is infeasible, rendering x * 𝑡 -1 also unknown. Now, one important contribution of this paper is to project Eq. ( 9) onto the measurement subspace. This is feasible as Ax * 𝑡 -1 can be approximated under the condition y ≈ Ax 0 when the degradation operator is linear and 𝜎 𝑦 is assumed within a moderate range.EQUATIONSpecifically, by projecting onto the measurement subspace, we choose x 𝑡 -1 by the following:x 𝑡 -1 = arg min x 𝑖 𝑡 -1 ∥A (x 𝑖 𝑡 -1 -x * 𝑡 -1 )∥ 2 2 (12)Since x 0 is expected to be more accurate than the denoised result x0|𝑡 , the measurement y ≈ Ax 0 can provide a stable and strong supervisory signal to correct the sampling result. Then by means of sample selection, we effectively control the result x 𝑡 -1 within a more desirable region, even though we have no access to the x 0 . As A is irreversible and complex in numerous scenarios [6, 7] , obtaining its pseudo-inverse poses challenges. Therefore, we propose to randomly sample 𝑛 candidates x 𝑖 𝑡 -1 , 𝑖 ∈ [0, 𝑛 -1] from the predicted distribution 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 , y), and choose the one with the minimal deviation from our anticipated values. Finally, Eq. ( 9) becomes:EQUATION)The process of sample selection is also denoted as noise selection, given by x 𝑖 𝑡 -1 = 𝝁 𝜃 (x 𝑡 , 𝑡, y) + 𝜎 𝑡 z 𝑖 𝑡 . We detail the full algorithm in x 𝑡 -1 ← 𝝁 𝜃 (x 𝑡 , 𝑡, y) + 𝜎 𝑡 z 𝑡 10: end for 11: return x0 Algorithm 1, and name our algorithm Diffusion Posterior Proximal Sampling (DPPS). The chosen x 𝑡 -1 is referred to as the proximal sample due to the similarity to the proximal operator [36] .𝐷 𝑖 = ∥A 𝝁 𝜃 (x 𝑡 , 𝑡, y) + 𝜎 𝑡 z 𝑖 𝑡 -𝐶 1 x 𝑡 -𝐶 2 y∥ 2It is commonly perceived that the generative process of diffusion does not exhibit uniform significance across all timesteps [15, 24, 27] . Based on this insight, we adaptively adjust the number of candidate samples during the generative process according to the signal-to-noise ratio to enhance image quality and reduce computational cost. Through extensive experimentation, we discovered that using a larger number of candidate samples in the final stage of generation yields better results. The number of candidate samples 𝑛 at each step can be expressed as:EQUATION)where 𝑡 ∈ [0, 999] is the timestep, 𝜆 𝑡 = ᾱ𝑡 1-ᾱ𝑡 represents the signalto-noise ratio, and 𝑁 𝑚𝑎𝑥 is a hyper-parameter that adjusts the maximum sampling frequency. The minimum frequency is set to 2, ensuring our proximal sampling strategy.Recent studies have pointed out that initializition have a significant impact on the generated results [4, 32, 43] , and the discrepancy between the two distributions account for a discretization error [2] . In this paper, inspired by [16] , we simply initialize the sample in the same way as during training, making the best use of the available measurementEQUATION)where A 𝑇 means the transpose of operator. We argue this trick provides a modicum of information as a signal to the reverse model, as it realigns the distribution of initial latent with the training distribution [16] .3.5.1 Relevance to Existing Methods. DPS [8] proposes to estimate the intractable likelihood under Laplacian approximation, and adopts random sampling from the predicted distribution. However, the randomly chosen sample may not be well-coordinated Input DDRM DPS DiffPIR Ours Reference with the measurement information. Another study MCG [9] iteratively applies projection onto the measurement subspace after the denoising step to ensure data consistency. However, the imposed projections lead to accumulated error due to measurement noise.Our algorithm can be viewed as a special case of DPS, where we choose the sample that exhibits better measurement consistency. It absorbs the advantages of the robustness to noise from DPS and the faithful data consistency of MCG. Moreover, the better data consistency is achieved by injecting a guided adaption z 𝑡 , which also helps correct the prediction errors in last generation steps [23, 24] , leading to enhanced output quality.Selecting the proximal sample introduces some extra computational overhead. In practice, the diffusion model conducts back-propagation only once, aligning with the mainstream approaches [8, 9, 48, 49] . The evaluations for Eq. ( 13) are considerably cheaper than gradient calculation, making the additional computational costs manageable. We show the computational resources for different settings in Section 5.4.Here, we provide some theoretical supports for our methodology. Despite our proposed proximal sampling being random, we can theoretically approach the desired point within an upper bound with mild assumptions. The proposition, detailed in supplementary material, proves that we can converge to our desired target x * 𝑡 -1 , when the number of candidate samples is large enough. Furthermore, our method doesn't require an exceptionally large number of candidates to achieve promising performance. This is because the variance of our method decreases when the proximal one is selected from any number of candidates. Proposition 3.1. For the random variable z ∼ N (0, I) and its objective function:EQUATION)The variance for 𝑓 (z) is denoted by Var (𝑓 (z)). We have Here, Var DPS (𝑓 (z)) is the variance of DPS [8] , Var MC (𝑓 (z)) is the variance of Monte Carlo sampling, and Var Ours (𝑓 (z)) is the variance of our proximal sampling method.In line with our theoretical analysis, empirical observations suggest that the proximal sample strategy demonstrates improved data consistency and reduced stochasticity, as evidenced by the experimental results. 
related work 4.1 diffusion models for image restoration: 
 To solve image restoration or image inverse problems with diffusion models, plenty of notable works [8, 13, 26, 47] have been introduced. The first category of approaches [13, 41, 42] involves training conditional diffusion models or approximating the likelihood with synthetic image pairs as training data. However, these methods require specific training for each task and lack generalization across a wide range of inverse problems. In contrast, the second category of approaches addresses inverse problems with unconditional diffusion models by guiding the reverse diffusion process [49, 59] . Several studies [8, 47, 48] concentrate on estimating the time-dependent likelihood 𝑝 𝑡 (y|x 𝑡 ) for posterior sampling. Among them, Song et al. [48] enhances the likelihood approximation with a Monte-Carlo estimate. Ma et al. [32] tackles image super-resolution by selecting the best starting point. [20] [21] [22] effectively constrain the results of each generative step, adeptly tackling low-light image enhancement and shadow removal. Recently, latent diffusion models have also seen advancements [37, 52] and have been widely adopted in various image inverse problem scenarios [11, 17, 38, 39, 45] . Furthermore, while several deep learning-based proximal optimization algorithms have been proposed [28, 55] , our method stands out as the first to make the diffusion sampling as a proximal optimization process.The exposure bias [29, 35] in diffusion models is described as the misalignment between the training input x 𝑡 and the reverse process input x 𝑡 , which is essentially a mismatch between the predicted noise 𝝐 𝜃 and the actual noise 𝝐. Ning et al. [35] mitigate the exposure bias problem by perturbing the training input, rendering the network more robust to inaccurate inputs during generation. Ning et al. [34] propose dynamic scaling to correct the magnitude error of 𝝐 𝜃 and improve sample quality. Li et al. [29] present a novel solution by adjusting the corruption level of the current samples. This paper provides a new way to effectively alleviate the exposure bias problem to some extent by reducing the uncertainty of random sampling. 
experiments 5.1 setup: 
 Dataset, Model. To showcase the effectiveness of the proposed methods, we conducted experiments on two standard datasets, namely FFHQ 256×256 [25] and ImageNet 256×256 [40] . The evaluation encompasses the first 1k images in the validation set of each dataset. The diffusion models pre-trained on ImageNet and FFHQ are sourced from Dhariwal and Nichol [13] and Chung et al. [8] , respectively. For comprehensive comparisons, we include state-of-the-art diffusion-based image restoration solvers, including the Plug-and-play alternating direction method of multipliers (PnP-ADMM) [5] , Score-SDE [49] , Denoising Diffusion Restoration Models (DDRM) [26] , Manifold Constrained Gradient (MCG) [9] , DPS [8] , LGD-MC [48] , and DiffPIR [59] . Sampling frequency parameter 𝑁 𝑚𝑎𝑥 is set to 50. To ensure fair comparisons, we utilized the same pre-trained diffusion models for all diffusion-based methods. All experiments were executed with a fixed random seed.Degradation Operator. The degradation operators are defined as follows: (i) For inpainting, 80% of the pixels (all RGB channels) in the image are masked. (ii) For Gaussian blur, a blur kernel of size 61 × 61 with a standard deviation of 3.0 is employed. (iii) For motion blur, the kernel, following the procedure outlined in Chung et al. [9] , has a size of 61 × 61 and an intensity value of 0.5. (iv) For SR × 4, the operator involves 4× bicubic down-sampling. Additive Gaussian noise with a variance of 𝜎 𝑦 = 0.01 is applied for all degradation.Metrics. The metrics employed for the comparison encompass: Peak Signal-to-Noise Ratio (PSNR) as distortion metrics; Learned Perceptual Image Patch Similarity (LPIPS) [58] distance, and Frechet Inception Distance (FID) [18] as perceptual metrics.We present the statistic results of the general image restoration tasks on both FFHQ and ImageNet datasets, as detailed in Table 1 . To the dataset with homogeneous scenarios, i.e., FFHQ, our proposed Reference Input DPS Ours (𝑛=10) Ours (𝑛=20) Ours (𝑛=100) method demonstrates impressive results across all metrics, establishing its superiority over existing state-of-the-art methods. When dealing with more varied images within the ImageNet dataset, our approach exhibits substantial outperformance against all baseline methods in terms of FID and LPIPS, while maintaining comparable levels in PSNR.The visual results for inpainting, SR, and Gaussian deblurring are shown in Figure 4 , showcasing the evident superiority of our proposed method. While DDRM attains commendable distortion results with fewer Neural Function Evaluations (NFEs), it faces limitations in reliably restoration results for image inpainting tasks characterized by a very low rank of the measurement. DiffPIR achieves satisfactory results in various scenarios; however, its performance is tied to the analytical solution for the data consistency term and exhibits sensitivity to measurement noise. DPS differs from our approach in that it introduces random noise at each generation step, leading to over-smoothed and unstable restoration results, as depicted in Figure 4 (column 3). Conversely, our proposed method circumvents such drawbacks. The generative process is stabilized by directing the sample to a predefined target, yielding realistic and detailed restoration outcomes. It is noteworthy that the generated results by our method exhibit minimal variation with different seeds (refer to supplementary material), aligning with the intended design of identity preserving.To investigate the impact of our proximal sampling on the generated results, we conducted further experimental analyses using the first 150 images from the FFHQ validation set. In addition to the naive DPS, we incorporated the results of DPS with DDIM [46] deterministic sampling, denoted as DPS_DDIM, which mitigates the effects of randomness. Results for (a) mean 7 . It is evident that our method facilitates a more stable optimization process, yielding markedly superior results compared to DPS and DPS_DDIM within the same period. This observation also supports the claim that the measurement can provide reliable supervisory signals for the sampling result.Robust to Hyper-parameter. We anticipate that our proposed proximal sampling serves as an adaptive correction for prediction errors resulting from imprecise parameter settings. To substantiate this claim, we conducted experiments with different 𝜆 𝑡 , comparing the results of our proximal sampling with those obtained through random sampling. As illustrated in Figure 6 , our method consistently outperforms random sampling across different parameter settings. Notably, the performance of the random sampling method exhibits fluctuations with changes in parameters. In contrast, our method demonstrates stability over a range of parameter configurations, thereby validating the assertion.To examine the impact of the variable 𝑛 on computational efficiency, we conducted an evaluation that consisted of measuring memory consumption and computational time required for the SR×4 task. This evaluation was performed on a single NVIDIA RTX 3090 GPU, and the results are presented in Table 2 . Notably, when the value of 𝑛 is relatively small (≤20), our method shows a substantial enhancement in the restoration metric as 𝑛 increases while incurring minimal increases in memory consumption and computation time. Additionally, the carefully designed adaptive sampling frequency strategy further enhances restoration efficiency within less computation time. These findings highlight the effectiveness and validity of our approach. 3 and Figure 5 . Figure 5 illustrates a noticeable enhancement in output quality and details with increasing 𝑛. This observation is reinforced by Table 3 , where both the LPIPS and FID metrics exhibit a notable decrease as 𝑛 increases. The augmented number of candidate samples corresponds to a higher likelihood of proximity to the intended target, thereby providing enhanced supervision for the generative process. The enhancement of the proposed adaptive sampling frequency further enhance the efficiency and applicability of our proposed method. It is noteworthy that the improvement becomes relatively marginal when 𝑛 exceeds 20 and may even lead to a decrease in PSNR. There are several reasons for this phenomenon. (i) The approximation is only conducted within the range space, neglecting the null space. This can lead to a certain degree of misalignment with the reference image. (ii) Since we utilized y ≈ Ax 0 in our algorithm, the presence of measurement noise (n in Eq. ( 5)) affects the accuracy of our sample selection. (iii) It can also be attributed to the well-known trade-off between perception and distortion metrics [3] . Consequently, a larger value of 𝑛 does not necessarily lead to better results.Level. The precision of the approximation is impacted by the level of measurement noise n. We explore the effects of different noise levels on the proposed method compared with random sampling. The results presented in Table 4 reveal that (i) both methods experience a decline in performance as the noise level increases, and our approach consistently outperforms random sampling in all settings. (ii) For inpainting, both methods experience considerable degradations as the noise level increases, demonstrating sensitivity to the noise levels. (iii) For SR, notable performance degradation is observed in our method as the noise level increases, while the method adopting random sampling demonstrates greater robustness in this regard. 
conclusion: 
 In this paper, we introduce a novel approach to mitigate problems caused by misaligned random sampling in diffusion-based image restoration methods. Specifically, our approach advocates selecting the proximal sample that is more consistent with the observed measurement in the predicted distribution. An adaptive sampling frequency strategy is followed to optimize the computational efficiency of the proposed method. In addition, we propose a realignment of the inference initialization involving measurement information to better align with expected generations. Experimental results validate a substantial performance improvement compared to SOTA methods. Our method innovatively takes advantage of stochastic sampling in the diffusion generative process, exploiting the sampling selection to enhance the generation quality and offering new insights for future diffusion inference algorithms. 
 
 Appendix:  


Decision: This paper was reviewed by three experts in the field. The reviewers reached a consensus of positive feedback, which are Weak Accept, Weak Accept, and Accept. All reviewers appreciate the novelty of this proposed sampling strategy, which solves a long standing problem in the diffusion sampling process. Experimental results also show an impressive improvement. ACs also share the similar feedback as the reviewers. Based on this, the decision is to recommend the paper for acceptance to ACM Multimedia 2024.

We recommend that the authors carefully read all reviewers' final feedback, and revise the manuscript as needed. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: The paper introduces a novel paradigm for diffusion-based image restoration called Diffusion Posterior Proximal Sampling (DPPS). It addresses the issue of over-smoothing in existing algorithms by selecting the most data-consistent sample from multiple candidates at each generative step, starting the restoration with an initialization that incorporates measurement signals. Extensive experiments demonstrate DPPS's superior performance in image restoration tasks with minimal additional computational cost.
strength_weakness: strength_weakness: 1. **Improved Image Restoration Quality**: The proposed Diffusion Posterior Proximal Sampling (DPPS) method enhances image restoration performance by selecting the most data-consistent sample from multiple candidates, leading to more accurate and detailed results compared to existing methods.

2. **Efficient Use of Computational Resources**: DPPS achieves significant improvements in image restoration with only a marginal increase in computational cost, making it an efficient solution that does not require excessive computational resources.

3. **Robustness to Hyper-parameters**: The method demonstrates stability and robustness across a range of hyper-parameter configurations, meaning it can perform well without the need for fine-tuning to specific parameter settings.
limitations: limitations: The visual comparison images provided by the author are too few; additional visual comparison images for other tasks should be included, such as inpainting and deblurring. Moreover, the number of comparative methods is also too limited. It is suggested that the author compares more methods.
paper_summary: paper_summary: The paper presents a pioneering approach to enhance the quality of image generation by strategically utilizing sampling choices from the predicted distribution at each reverse step in the diffusion process. It introduces an efficient proximal sampling strategy that is closely aligned with the measurement identity, which is particularly effective for solving diffusion-based image restoration problems. The extensive experimental results provided in the paper demonstrate that this proposed method surpasses other state-of-the-art techniques in terms of image restoration performance, with the added benefit of requiring only minimal additional computation.
strength_weakness: strength_weakness: 1. The methods proposed in this paper appear to be highly effective in achieving data consistency in diffusion-based image restoration methods, which I find to be quite challenging while this paper manages to accomplish it.
2. Surprisingly, the solutions presented in this paper only require a single time model inference, significantly enhancing the practicality of the proximal sampling strategy. In fact, I have read a similar work requiring multiple sampling. The specific implementation in this paper exceeds my expectations, and I consider it to be interesting and novel.
3. The image restoration results presented in this paper, especially on data consistency, are impressive.
limitations: limitations: 1. I find the handling of noise in the degradation model in this paper somewhat confusing. Eq. (5) suggests that $y=A(x_0)+n$, yet in L350, it is stated that $y≈Ax^*_0$. I wonder why the noise $n$ can be ignored? The method maybe fails when noise is large? (e.g., $\sigma_y=50/255$)
2. For the degration process $A(x)$, I think it can not always be equivalent to $Ax$, which means multiplicative distribution law can not be casually used in Eq.(11)~(13). (maybe need an additional statement)
3. $x_0$ and $x^*_0$ seem to have the same meaning? If so, I suggest unifying the variable symbols.
4. As the paper mentioned, DPPS can be seen as a developed version of DPS. While I acknowledge that the proximal sampling strategy proposed in this paper is practical and novel, having only single novel method throughout the paper may not be sufficient to meet the novelty requirements expected by ACM MM as a top conference.  

The 4th comment is the main reason why I tend to rate this paper as "Borderline". I feel that even if this work could add just one more contribution, I would be inclined to give it a higher rating.   
For example, the revised work could attempt to make the posterior sampling strategy more intelligent, such as adaptively adopting different sampling frequencies for different timestamps to futher imporve image quality and reduce computational cost.
paper_summary: paper_summary: The paper introduces an approach called Diffusion Posterior Proximal Sampling (DPPS) for image restoration using diffusion models. It enhances the restoration process by selecting samples that are more consistent with the measurement identity at each generation step and by initializing with a combination of the measurement signal, which significantly improves performance with minimal additional computational cost.
strength_weakness: strength_weakness: 1. The DPPS method improves the stability of the generative process by reducing randomness, resulting in higher quality image restoration outcomes.
2. The research provides empirical evidence of the superior performance of the proposed method across various image restoration tasks such as super-resolution, deblurring, and inpainting.
3. The initialization strategy and proximal sampling strategy presented in the paper offer new perspectives and directions for the application of diffusion models in image restoration.
limitations: limitations: 1. The article seems to use the results of each step to constrain the trajectory. There have been many previous application articles [1] [2] [3] that have considered similar research. The author should compare and analyze them to further confirm the innovation.

[1] Global structure-aware diffusion process for low-light image enhancement

[2] Low-light image enhancement with wavelet-based diffusion models

[3] DeS3: Adaptive Attention-Driven Self and Soft Shadow Removal Using ViT Similarity

2. For the experimental part, especially the visual comparison, the authors should provide a more detailed analysis of why more details are lost as the number of n increases.
3. The author should provide more details on the computational complexity and parameter size of the overall architecture compared to previous methods.
4. Can the proposed method be extended to other tasks? If so, it will be very helpful.