Block Coordinate Plug-and-Play Methods for Blind Inverse Problems
Authors: Weijie Gan
Shirin Shoushtari
Yuyang Hu
Jiaming Liu
Hongyu An
Ulugbek Kamilov
Venue: NeurIPS 2023 poster (2023)
Keywords: inverse problems
plug-and-play priors
computational imaging
nonconvex optimization
TL;DR: We present a block-coordinate PnP method with theoretical analysis for blind inverse problems by applying learned denoisers as priors on both the image and measurement operator.

Abstract: Plug-and-play (PnP) prior is a well-known class of methods for solving imaging inverse problems by computing fixed-points of operators combining physical measurement models and learned image denoisers. While PnP methods have been extensively used for image recovery with known measurement operators, there is little work on PnP for solving blind inverse problems. We address this gap by presenting a new block-coordinate PnP (BC-PnP) method that efficiently solves this joint estimation problem by introducing learned denoisers as priors on both the unknown image and the unknown measurement operator. We present a new convergence theory for BC-PnP compatible with blind inverse problems by considering nonconvex data-fidelity terms and expansive denoisers. Our theory analyzes the convergence of BC-PnP to a stationary point of an implicit function associated with an approximate minimum mean-squared error (MMSE) denoiser. We numerically validate our method on two blind inverse problems: automatic coil sensitivity estimation in magnetic resonance imaging (MRI) and blind image deblurring. Our results show that BC-PnP provides an efficient and principled framework for using denoisers as PnP priors for jointly estimating measurement operators and images.

Block Coordinate Plug-and-Play Methods for Blind Inverse Problems
Weijie Gan
Shirin Shoushtari
Washington University in St. Louis weijie. gan@wustl. edu
Washington University in St. Louis shirin@rustl.... edu
Yuyang Hu Washington University in St. Louis h. yuyanguuustll.c.mmmmmm edu
Jiaming Liu Washington University in St. Louis j jiaming liu@wustl.edi
Hongyu An Washington University in St. Louis hongyuanOwustl.edd
Ulugbek S. Kamiloy
Washington University in St. Louis kamilov@vustl.ed
Abstract
Plug-and (PnP) prior is well-known class of methods for solving imaging inverse problems by computing fixed-points of operators combining physical mea- surement models and learned image denoisers. While PnP methods have been extensively used for image recovery with known measurement operators, there is little work on PnP for solving blind inverse problems. We address this gap by presenting a new block-coordinate PnP (BC-PnP) method that efficiently solves this joint estimation problem by introducing learned denoisers priors on both the unknown image and the unknown measurement operator. We present new conver- gence theory for BC-PnP compatible with blind inverse problems by considering nonconvex cata-fidelity terms and expansive denoisers. Our theory analyzes the convergence of BC-PnP to stationary point of implicit function associated with an approximate minimum mean-squared error (MMSE) denoiser. We numerically validate method two blind inverse problems: automatic coil sensitivity estimation in magnetic resonance imaging (MRI) and blind image deblurring. Our results show that BC-PnP provides efficient and principled framework for using denoisers as PnP priors for jointly estimating measurement operators and images.
1 Introduction
Many problems in computational imaging, biomedical imaging, and computer vision can be formu- lated as inverse problems involving the recovery of high-quality images from low-quality observations. Imaging inverse problems generally ill-posed, which means that multiple plausible clean images could lead to the same observation.
thus common to introduce prior models on the desired images. While the literature prior modeling of images is vast, current methods are often based on deep learning (DL), where a deep model is trained to map observations to images [1-3].
Plug- Plug-and-ple and-play (PnP) priors [4.5] is one of the most widely-used DL frameworks for solving imaging inverse problems. PnP methods cireumvent the need to explicitly describe the full probability density of images by specifying image priors using image denoisers. The integration of state-of-the-an deep denoisers with physical measurement models within PnP has been shown be effective in number of inverse problems, including image super-resolution, phase retrieval, microscopy, and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

medical imaging [6-13] (see also recent reviews [14, 15]). Practical success of PnP has also motivated novel extensions, theoretical analyses, statistical interpretations, as well as connections to related approaches such as score matching and diffusion models [16-29].
Despite the rich literature on PnP, the existing work the topic has primarily focused on the problem of image recovery where the measurement operator is known exactly. There is little work on PnP for blind inverse problems, where both the image and the measurement operator are unknown. This form of blind inverse problems are ubiquitous in computational imaging with well-known applications such as blind deblurring [30] and parallel magnetic resonance imaging (MRI) [31]. In this paper, we address this gap by developing a new PnP approach that uses denoisers as priors over both the unknown measurement model and the unknown image. and efficiently solves the joint estimation task as a block-coordinate PnP (BC-PnP) method. While variant of BC-PnP was proposed in the recent paper [21], it was never used for jointly estimating the images and the measurement operators. Additionally, the convergence theory in [21] is inadequate for blind inverse problems since it assumes convex data-fidelity terms and nonexpansive denoisers. We present a new convergence analysis applicable to nonconvex data-fidelity terms and expansive denoisers. Our theoretical analysis provides explicit error bounds on the convergence of BC-PnP for approximate minimum mean squared error (MMSE) denoisers under set of clearly specified assumptions. We show the practical relevance of BC-PnP by solving joint estimation problems in blind deblurring and accelerated parallel MRI. Our numerical results show the potential of denoisers to act as PnP priors over the measurement operators as well as images. Our work thus addresses a gap in the current PnP literature by providing a new efficient and principled framework applicable to a wide variety of blind imaging inverse problems.
All proofs and some details that have been omitted for space appear the supplementary material.
2 Background
Inverse Problems. Many imaging problems can be formulated as inverse problems where the goal is to estimate an unknown image x E R" from its degraded observation y Ax + e, where A E Txxx is a measurement operator and R is the noise. A common approach for solving inverse problems is based on formulating an optimization problem
a E arg min fxx) with f(x) = g(z) + h(x))
where g is the data-fidelity term that quantifies consistency with the observation y and h is the regularizer that infuses a prior on æ. For example, a widely-used data-fidelity term and regularizer in computational imaging are the least-squares g(z) = = A y1l²2 and the total variation (TV) functions h(x=)= = T Da 1. where D is the image gradient, and > Ó regularization parameter.
The traditional inverse problem formulations assume that the measurement operator A known exactly. However, in many applications, it is more practical to model the measurement operator as A(0), where 0 E I2 are unknown parameters to be estimated jointly with æ. This form of inverse problems are often referred to as blind inverse problems and arise in wide-variety of applications, including pratellel MRI [32-37], blind deblurring [30, (30,38,33 (4,38,,3) 39I, and computed tomography [40-43).
DL. There is a growing interest in DL for solving imaging inverse problems [1-3]. Instead of explicitly defining regularizer, DL approaches for solving inverse problems learn mapping from the measurements to the desired image by training a convolutional neural network (CNN) to perform regularized inversion [44-48]. Model-based DI. (MBDL) has emerged as powerful DL framework for inverse problems that combines the knowledge of the measurement operator with an image prior specified by a CNN (see reviews [3. 49]). The literature of MBDL is vast, but some well- known examples include PnP. regularization by denoising (RED), deep unfolding (DU), compressed sensing using generative models (CSGM), and deep equilibrium models (DEQ) [50-54]. All these approaches come with different trade-off's in terms of imaging performance, computational and memory complexity, flexibility, need for supervision, and theoretical understanding.
The literature on DL approaches for blind inverse problems is broad, with many specialized methods developed for different applications. While an in-depth review would be impractical for this paper, we mention several representative approaches adopted in prior work. The direct application of DL. to predict the measurement operator from the observation was explored in [55,56]. Deep image prior (DIP) was used as learning- free prior to regularize the image and the measurement operator
2

---

in [57,58]. Generative models, including both GANs and diffusion models, have been explored as regularizers for blind inverse problems in 60J. Other work considered the use of a dedicated neural network to predict the parameters of the measurement operator, adoption of model adaptation strategies, and development of autocalibration methods based on optimization [34-37,61 61-64).
PnP. PnP [4.5] is one of the most popular MBDL approaches based on using deep denoisers as imaging priors (see also recent reviews [14, 15J). For example, the proximal gradient method variant of PnP (referred to as PnP-ISTA in this paper) can be formulated as a fixed-point iteration [65]
x t Do (z*) with 2² = 1-*x-1_ -ee---VVglae Vg(xx=1)
(2)
where Dar is denoiser with parameter > 0 for controlling its strength and 7 >> 0 is step-size. The theoretical convergence PnP-ISTA has been explored for convex functions 9 using monotone operator theory [20, 22] well for nonconvex functions based on interpreting the denoiser MMSE estimator [23]. The analysis in this paper builds on the convergence theory in [23] that uses an elegant formulation by Gribonval [66] establishing direct link between MMSE estimation and regularized inversion. Many variants of PnP have been developed over the past few years [6-12], which has motivated an extensive research its theoretical properties [16, 18, 22,
Block coordinate regularization by denoising (BC-RED) is recent PnP variant for solving large-scale inverse problems by updating along subset of coordinates at every iteration [21]. BC-RED is based on regularization by denoising (RED), another well-known variant PnP that seeks to formulate an explicit regularizer for given image denoiser [17, 19J. BC-RED was applied to several non-blind inverse problems and was theoretically analyzed for convex data- aata-fidelity terms.
PnP was extended to blind deblurring in [71, 72] by considering an additional prior on blur kernels that promotes sparse and nonegative solutions. PnP was also applied to holography with unknown phase errors by using the Gaussian Markov random field model as the prior for the phase errors [73]. Calibrated RED (Cal-RED) [43] is a recent related extension of RED that calibrates the measurement operator during RED reconstruction by combining the traditional RED updates over an unknown image with a gradient descent over the unknown parameters of the measurement operator. However, this prior work does not leverage any learned priors for the measurement operator and does not provide any theoretical analysis.
Our contributions. (I) Our first contribution is in the use of learned deep denoisers for regularizing the measurement operators within PnP. While the idea of calibration within PnP was introduced in [43], denoisers were not used as priors for measurement operators. (2) Our second contribution is the application of BC-PnP as an efficient method for jointly estimating the unknown image and the measurement operator. While BC-RED was introduced in [21] as a block-coordinate variant of PnP, the method was used for solving non-blind inverse problems by using patch- image denoisers. (3) Our third contribution is new convergence theory for BC-PnP for the sequential and random block-selection strategies under approximate MMSE denoisers. Our analysis does not assume convex data-fidelity terms, which makes it compatible with blind inverse problems. Our analysis can be seen as an extension of [23] to block-coordinate updates and approximate MMSE denoisers. (4) Our fourth contribution is the implementation of BC-PnP using learned deep denoisers priors for two distinct blind inverse problems: blind deblurring and auto-calibrated parallel MRL. Our code-which we share publicly-shov the potential of learning deep denoisers over measurement operators and using them for jointly estimating the uknown image and the uknown measurement operator.
3 Block Coordinate Plug-and-Play Method
We propose efficiently solve blind inverse problems by using a block-coordinate PnP method, where each block represents one group of unknown variables (images, measurement operators, etc). The novelty of our work relative to [21] in solving blind inverse problems by introducing learned priors on both the unknown image and the uknown measurement operator. Additionally, unlike [21], our work proposes a fully nonconvex formulation that is more applicable blind inverse problems.
Consider the decomposition of a vector x E R into 2 1 blocks
= (
.x6 E R X ... X R Rb
with
n2 + 74g.
(3)
3

---

Algorithm Block Coordinate Plug-and-Play Method (BC-PnP)
1: input: initial value x E RR", parameters E RT, and step-size y >
2: for k 2...3.... do
3: Choose an index is E (1.....1)
4: x + -== i G (x--1)
where (z) = U. ((z) with G(x) ( Da VVg(z))).
5: end for
For each i E f1, ... b), we define a matrix U, E R that injects a vector in R" into R" and its transpose U. that extracts the ith block from a vector RR. For any a E IR",, we have
h U,S; 1=1
with æ; U. x E R", i=
U.U. -1 EV.U
(4)
1=1
Note that (4) directly implies the norm preservation | are interested in clork-coordinate algorithm that uses only a subset of operator outputs corresponding coordinates in some block i (1,...; b). block-coordinate operator G IR" + IR: as
+ for any R" We
Hence, for operator G IR= IR", define the
(x = [G(z)], = U. E RR; #E R".
(5)
We are in- in-particular interested in two operators: (a) the gradient Vg(x)= = (V1g(x)... ... Vog(z))
of the data- fidta-iiility term g and (b) the denoiser (x) = (Do, (z1), ... Do, (ax)). where the vector (0),'.. ,a) E R consists of parameters for controling the strength of each block denoiser. Note how the denoiser acts in a separable fashion across different blocks.
When 6 = 1, we have U₁ uf I and BC-PnP reduces to the conventional PnP-ISTA [23,65], When > have at least two blocks with BC-PnP updating only one block time
x Do, a (V
when j + is when j = jE .b.
x
(6)
As with any coordinate descent method (see [74] for a review), BC-PnP can be implemented using different block selection strategies. One common strategy is simply update blocks sequentially as i = 1 + mod(k - 1, 6), where mod(-) denotes the modulo operator. An alternative is to proceed in epochs of consecutive iterations, where at the start of each epoch the set {1.... 6} is reshuffled, and ig is then selected consecutively from this ordered set. Finally, one can adopt a fully randomized strategy where indices is are selected i.i.d. random variables distributed uniformly over f1..... .b).
Throughout this work, we will assume that each denoiser Do, is an approximate MMSE estimator for the following AWGN denoising problem
2; n2 with
Pat:
II)
where i E (1,.... b) and z; E R. We rely only on an approximation of the MMSE estimator of 2;; given z;, since the exact MMSE denoiser corresponds the generally intractable posterior mean
Dr, = E[2,|2,] =
2Pm. =z (x) (z))2;) dæ.
(8)
Approximate MMSE denoisers are a useful model for denoisers due to the use of the MSE loss
C(D.,) E ||| Do, (z:)
for training deep denoisers, well as the optimality of MMSE denoisers with respect to widely used imagc-quality metrics such as signal-to-ne noise ratio (SNR).
As simple illustration of the generality of BC-PnP, consider b = 2 with the least-squares objective
g(z) l1y A(0)v with
(v. 0).
(10)

---

where v E R denotes the unknown image and 6 E R" denotes the unknown parameters of the measurement operator. BC-PnP can then be implemented by first pre-training dedicated AWGN denoiser Do; for each block i and using it as a prior within Algorithm 1. It is also worth noting that the functions g in (10) nonconvex with respect to the variable a: E TR.. In the next section, we present the full convergence analysis of BC-PnP without any convexity assumptions on g and nonexpansiveness assumptions on the denoiser D..
4 Convergence Analysis of BC-PnP
In this section, we present two new theoretical convergence results for BC-PnP. We first discuss its convergence under the sequential updates and then under fully random updates. It worth mentioning that BC-RED with fully random updates was theoretically analyzed in [21]. The novelty of our analysis here lies in that it allows for nonconvex functions and expansive denoisers Da,, The nonconvexity of is essential since most data-fidelity terms used for blind inverse problems are nonconvex. On the other hand, by allowing expansive Dr,, our analysis avoids the need for the spectral normalization techniques that were previously suggested for PnP methods [21,22].
In the following, we will denote as D .D D',)) the exact MMSE denoiser in (8). Our analysis will require five assumptions that will serve as sufficient conditions for our theorems. Assumption 1. The blocks æ; independent with non-degenerate priors Pa; over R
As reminder, a probability distribution Pa, degenerate over I"; if it is supported on a space of lower dimensions than nı, Assumption 1 is required for establishing explicit link between the MMSE denoiser (8) and the following regularizer (see also [23, 66) for additional background)
h(ar) Σ h,(a)), = (æ1, xrn) E IR",
(11)
where each function h2 is defined (see the derivation in Section D.2 of the supplement)
hx(xs) = 전 - - (D;; 1 (xx)) + o of ltr,
for æ; E lm(D))) for æ, & lm(D),)
(12)
+00
where y > 0 is the step size, (D=;)-1 1 lm(D:,)) + R is the inverse mapping, which is well defined and smooth over Im(D;, and ho. (.) log (Px, (-)). where pz, is the probability distribution over the AWGN corrupted observations (7). Note that the function h; is smooth for any ø; E Ilm(",,) which is the consequence of the smoothness of both (D;,)~1 and hog-
Assumption 2. The function g is continuously differentiable and Vg is Lipschitz continuous with constant L > O. Additionally, each block grudient Vey is block Lipschitz continuous with constant I, > 0. We define the largest block Lipschitz constant as L-max == max ". L
Lipschitz continuity of the gradient Vg is standard assumption in the context of imaging inverse problems. Note that we always have the relationship (L/b) Lmax S (see Section 3.2 in [74]). Assumption 3. The explicit data-fidelin term and the implicit regularizer are bounded from below inf g(x)) -0, inf h(()) > -00 ZER
(13)
Assumption 3 implies that there exists f - 0o such that ((x) 2 f for all æ E RD, Since Assumptions 1-3 correspond to the standard assumptions used in the literature, they are broadly satisfied in the context of inverse problems.
Our analysis assumes that at every iteration, BC-PnP uses inexact MMSE denoisers on each block. While there are several ways to specify the nature of "inexactnesss"" we consider the case where at every iteration & of BC-PnP the distance of the output of D,, to D is bounded by constant ey. Assumption 4. Each block denoiser D,, Dø satisfies
[|Do, (z+) D', (z+ lll Ek,
i (l,.....,b b]. k 1,2,3, .......
where D;, is given in (8) and 2े a. 1 ~ig((>
5

---

For convenience, we will define quantities s² == = and = (1/t) +.. =i) that correspond to the largest and the mean gquared-distaneee between the mexacc and exact denoisers. Assumption 4 states that the error of the approximate MMSE denoiser used for inference is bounded relative to the exact MMSE denoiser, which reasonable when the approximate MMSE denoiser is a CNN trained to minimize the MSE.
It has been shown in the prior work [23, 66] that the function h in (11) is infinitely continuously differentiable over Im(D'). Our analysis requires the extension of the region where h smooth to include the range of the approximate MMSE denoiser, which is the goal of our next assumption.
Assumption 5. Each regularizer h; in (12) associated with the MMSE denoiser (8) continously differentiable and has a Lipschitz continuous gradient with constant M, > 0 over the set
IDx, = {a E 1.
|| Dú, (z)1l2 Se, E ER E (1,....
We will define Mmax max MM,,, Ma. } to be the largest Lipschitz constant and Im. (D;) {or E R" æ; E (D:,)) E (l..... b}} to be the set over which h is smooth. Assumption expands the region where the regularizer associated with the exact MMSE denoiser is smooth by including the range of the approximate MMSE denoiser. For example, this assumption is automatically true when the exact and approximate MMSE denoisers have the same range, which is reasonable when the approximate MMSE denoiser is trained to imitate the exact one.
Our first theoretical result considers the sequential updates, where at each iteration, ig is selected as i = =1 mod - 1, 6) with mod(-) being the modulo operator. We can then express any iterate ib produced by BC-PnP for 2 as
cb (a
mg8 = (ar ( 1)6+11
Note that th E lm. (D;) since each block is an output of the denoiser. We prove the following result. Theorem 1. Run BC-PnP under Assumptions 1-5 using the sequential block selection and the step 0 <<> < 1/La Then, we have
1 E
(VF S (F(a9) - f-) C2E
min | (x 113
where C O and C2 > 0 iteration independent constants. additionally the sequence of error terms (2 is square-summable, we have that (x - 0 as 1 - 0.
Our second theoretical result considers fully random updates, where at each iteration, is is selected as i.i.d. random variables distributed over {1.... b). In this setting, we analyze the convergence of BC-PnP in terms of the sequence {G(=*)) Note that it is straightforward to verify that Zer(G)= = ff), which makes this analysis meaningful. We prove the following result.
Theorem Run PC-PnP under Assumptions 1-5 using the random i..dd. block selection and the step 0 <<> < Lm Then, we have
DI ((((()))-))) -
min E [I[[[a M1 SE 15k5t
|[ (G(=a--))3 112
D2s
where D1 > 0 and D2 are iteration independent constants, If additioonally the sequence of error squarr-ssummbbee we have that G(x²) a.s. as + co.
terms {E:]221
The expressions for the constants in Theorems and 2 are given in the proofs. The theorems BC-PnP asymptotically
show that if the sequence of approximation errors is square-summable achieves a stationary point of f. On the other hand, if the sequence of approximation errors is not gquare-summable the convergence only up to an error term that depends on the average of the squared approximation errors. Both theorems can thus be viewed as more flexible alternatives for the convergence analysis in [21]. is also worth mentioning that the theorems are interesting even when the denoiser errors are not square summable,, since they provide explicit error bounds on convergence.
While the analysis [21] assumes convex and nonexpansive D,, the analysis here does not require these two assumptions. It instead views the denoiser D, as an approximation to the MMSE estimator D where the approximation error bounded by Eµ at every iteration of BC-PnP. This view compatible with denoisers trained to minimize the MSE loss (9).

---

1x |000
11
RMSE (at)
SSIM (an)
RMSE (0)
Messenttl
0.50
umt
0333
10
0.700
Figure 1: Illustration of the PC-PnP convergence using the sequential and random i..dd block selection rules on CS-PMRI with the sampling factor R == 8. Lefimost two plots: Evolution of the distance between two consecutive image and CSM iterates. Rightmost three plots: Evolution the RMSE and SSIM metrics relative to the true solutions across BC-Pul iterations. Note how both block selection rules lead nearly identical convergence behaviour of BC-PnP in this experiment.
5 Numerical Validation
We numerically validate BC-PnP on two blind inverse problems: (a) compressed sensing parallel MRI (CS- (CS-PMRI) with automatic coil sensitivity map (CSM) estimation and (b) blind image deblurring. We adopt the traditional fz-norm loss in (10) the data-fidelin term for both problems. We will use x to denote the unknown image and A to denote the unknown parameters of the measurement operator. We use the relative root mean squared error (RMSH) and structural similarity index (SSIM) as quantitative metrics to evaluate the performance.
We experimented with several ablated variants of BC-PnP, including PnP, PnP-GDø, and PnP- oracleg. PnP and PnP-oraclep denote basic variants of PnP that use pre-estimated and ground truth measurement operators, respectively. PnP-GDg is variant of PnP based on [43], where θ is estimated without any DL prior. It is worth noting that PnP-oraclee is provided as an idcalized reference in experiment. As discussed in the following subsections, we also compare BC-PuP against several widely-used baseline methods specific to CS-PMRI and blind image deblurring.
5.1 Compressed Sensing Parallel MRI
The measurement operator of CS-PMRI consists of complex measurement operators A E C that depend on unknown CSMs {0,} } in C". Each sub-measurement operator can be parameterized as (0,) = P Fdiag((.), where F is the Fourier transform, P E R m"x", is the sampling operator, and diag(0.) (0)) forms a matrix by placing O, on its diagonal. We used T2-weighted MR brain acquisitions of 165 subjects obtained from the validation set of the fastMRI dataset [75] as the the fully sampled measurement for simulating measurements. We obtained reference 0, from the fully sampled measurements using ESPIRiT [76]. These 165 subjects were split into 145, 10, and 10 for training, validation, and testing, respectively. BC-PnP and baseline methods were tested on 10 2D slices, randomly selected from the testing subjects. We followed [75] to retrospectively undersample the fully sampled data using 1D Cartesian equispaced sampling masks with 10% auto-calibration signal (ACS) [76] lines. We conducted our experiments for acceleration factors R 6 and 8. We adopted DRUNet [12] as the architectures of D. for training both the image and CSM denoisers. BC-PnP and its ablated variants initialized using CSMs O, pre-estimated using ESPIRiT [76] and images 2o <- A(O,) y, where AH denotes the Hermitian transpose of A.
We considered several baseline methods, including ENLIVE [35], ESPIRIT-TVV [76], Unet [77], and ISTANet+ [51]. ENLIVE an iterative algorithm that jointly estimates images and coil sensitivity profiles. ESPIRiT-TV is an iterative algorithm that applies TV reconstruction method in (1). Unet is trained to map raw measurements to desired ground truth without the knowledge of measurement operator. ISTANet+ denotes widely-used DU architecture. We tested ESPIRiT-TV and ISTANet+ using CSMs pre-estimated using ESPIRIT.
Figure illustrates the convergence behaviour of BC-PnP on the test set for the acceleration factor R = 8. Figure 2 illustrates reconstruction results for the acceleration factor R = 6. Table summarizes the quantitative evaluation of BC-PnP relative to other PnP variants and the baseline methods. These results show that joint estimation can lead to significant improvements and that BC-PuP can perform as well as the idealized PnP-oraclee that knowns the true measurement operator.
7

---

ESPIRIT-TE TV
+
PnP
BC-PnP (Ours)
PnP-orackes
Ground-trin -uund
0.201 0.90
0.113 0.36
0.132 / U:95
0,064 / 0.9h
0:004 0.9V
RMSE / SSIM
CSS
Figure 2: Illustration of results from several well-kno known methods on CS-PMRI with the sampling factor R = 6. The quantities in the top-left corner of each image provide RMSE and SSIM values for each method. The squares at the bottom of each image visualize the error and the corresponding zoomed area in the image. Note how BC-PnP using deep denoiser the unknown CSMs outper- forms uncalibrated PnP and matches PrP-oracleg that knows the true CSMs.
Table 1: RMSE and SSIM performance of several methods CS-PMRI. The table highlights the best and second best results. The Calibration column highlights methods specifically designed to solve the blind inverse problem. Note how the use of DL prior over the measurement operator enables BC-PnP to outperform PnP and PnP-GDg and approach the performance of the oracle algorithm.
R
R=
Method
Calibration (Y/N)
RMSE
SSIM,
+RMSE,
RMSE
,S 1 TRMSE, 4
ENLIVE [35]
/
0.371
0.763
0.419
0.730
ESPIRiT-TV [76]
/
0.218
0.884
0.256
0.361
0.818
0.356
Unet [77]
X
0.218
0,904
0.195
0.907
ISTANet+ [51]
X
0.110
0.946
0.140
0.928
PnP
X
0.111
0,950
0,256
0.171
0.924
0.356
PnP-GDa [43]
0.116
0,950
0.254
0.163
0.926
0.355
BC-PnP (Ours)
0.091
0.961
0.247
0.122
0.946
0.337
PnP-oracles
x
0.069
0.969
0.000
0.082
0.962
0.000
not available in practice for blind inverse problems.
5.2 Blind Image Deblurring
The measurement operator in blind image deblurring can be modeled as A(0)x = 0 where is the unknown blur kernel, is the unknown image, and is the convolution. We randomly selected 10 testing ground truth image from CBSD68 [80] dataset. We generated 25 x 25 Gaussian kernels with 10'. We tested the algorithms on 2 Gaussian kernels. We used a pre-trained image denoiser, as in the experimental setting of [12]. The kernel denoiser was trained on 10,000 generated kernels at several noise levels. We adopted DrCNN with layers as the architectures D, for training kernel denoisers. BC-PnP and its ablated variants are initialized with the blur kernels Og pre-estimated using Pan-DCP [39] and images 2y <- A(0,)
We compared BC-PnP against several baseline methods, including Pan-DCP [39]. SelfDeblur [58], DeblurGAN [78], USRNet [79]. Pan-DCP is an optimization-biin -bsodd method that jointly estimates image and blur kernel. SeliDeblur trains two deep image priors (DIP) [81] to jointly estimate the blur kernel and the image. DeblurGAN is supervised learning based method that lacks the capability for kernel estimation, but can reconstruct images via direct inference. USRNet is a DU baseline that was tested using blur kernel estimated from [39]. The results of DeblurGAN and USRNet are obtained by running the published code with the pre-trained weights.
Figure 3 illustrates the reconstruction results with a Gaussian kernel. Figure demonstrates that BC-PnP can reconstruct the fine details of the snake skin, as highlighted by the white arrows, while both Pan-DCP and PnP produce smoother reconstructions. Additionally, BC-PnP generates more accurate blur kernel compared to the ground truth kernel, whereas Pan-DCP and SelfDeblur yield blur kernels with artifacts, Table presents the quantitative evaluation of the reconstruction results using
We used hta.gggchh...om//haaaggggggg BByeSR for generating the kernels.

---

Pan-DCP
SelfDeblur
PnP
BC-PnP (Ours)
PnP-oracley
Ground-truth
8.29/0.19 $220
0:30 0.45
(0.55
0052 0.00
RMSE SSIM RRMS,,
Figure 3: Illustration of results from several well-known methods on blind image deblurring with the Gaussian kernel. The squares the top of each image show the estimated kernels. The quantities in the top-left corner of each image provide RMSE and SSIM values for each method. The squares at the bottom of each image highlight the error and the corresponding zoomed image region. Note how the BC-PnP using deep denoiser the unknown kernel significantly outperforms the traditional PnP method and matches the performance of the oracle PnP method that knows the true blur kernel. Note also the effectiveness of BC-PnP for estimating the unknown blur kernel.
Table 2: Quantitative evaluation of BC-PnP in blind image deblurring. We highlighted the best and second best results, respectively. The Calibration column highlights methods specifically designed to solve the blind inverse problem. Note how the use of prior over the measurement operator enables BC-PnP to nearly match the performance of the oracle algorithm.
Method
Calibration (Y/N)
RMSE&
SSIM RRSSE 1 RMSE SSIM
RMSE +
Pan-DCP [39]
0.087
0.835
0.283
0.114
0.733
0.246
SelfDeblur [58)
0.219
0.495
0.775
0.176
0.553
0.831
DeblurGAN [78]
0.090
0.823
0.118
0.716
USRNet [79]
0.106
0.855
0.114
0.769
PnP PnP-GDa [43]
0.082
0.857
0.283
0.106
0.763
0.246
0.082
0.857
0.283
0.108
0.767
0.246
BC-PnP (Ours)
0.055
0.921
0.097
0.098
0.794
0.107
PnP-oracleo"
0.051
0.929
0.000
0.088
0.817
0.000
not available in practice for blind inverse problems.
Gaussian kernel, indicating that BC-PnP outperforms the baseline methods and nearly matches the SSIM and RMSE values of PnP-oraclep that is based on the ground truth blur kernel.
6
Conclusion
The work presented this paper proposes a new BC-PnP method for jointly estimating unknown images and unknown measurement operators in blind inverse problems, presents its theoretical analysis in terms of convergence and accuracy, and applies the method to two well-known blind inverse problems. The proposed method and its theoretical analysis extend the recent work PnP by introducing a learned prior on the unknown measurement operator, dropping the convexity assumptions the data-fidelity term, and nonexpansiveness assumptions numerical validation of BC-PnP shows the improvements due to the use of learned priors on the measurement operator and the ability of the method match the performance of the oracle PnP method that knows the true measurement operator. One conclusion of this work is the potential effectiveness of PnP for solving inverse problems where the unknown quantities are not only images.
the denoiser. The
Limitations
The work presented in this paper comes with several limitations. The proposed BC-PnP method is based on PnP. which means that its performance inherently limited by the use of AWGN denoisers priors. While denoisers provide convenient, principled, and flexible mechanism to specify priors,

---

they are inherently self-supervised and their empirical performance can thus be suboptimal compared to priors trained in a supervised fashion for specific inverse problem. PnP running over many iterations can also have higher computational complexity compared to some end-to-end alternatives, such as DU with small number of steps. Our analysis is based on the assumption that the denoiser used for inference computes an approximation of the true MMSE denoiser. While this assumption is reasonable for deep denoisers trained using the MSE loss, it is not directly applicable to denoisers trained using other common loss functions, such as the -norm SSIM. Finally, is common with most theoretical work, our analysis only holds when our assumptions are satisfied, which might limit its applicability in practice. Our future work will investigate ways to improve the results presented here by exploring new PnP strategies for relaxing assumptions for convergence, considering end-to-end trained variants of BC-PnP based on DEQ [53,54], and exploring BC-PnP using explicit regularizers [26-28].
Broader Impact
The expected impact of this work is in the area of imaging inverse problems with potential applications to computational imaging. There is a growing interest in computational imaging to leverage pre- trained deep models for estimating the unknown image well as the unknown parameters of the imaging system. The ability to better address such problems can lead to new imaging tools for biomedical and scientific studies. While novel DL methods, such the proposed BC-PnP approach, have the potential to enable new technological capabilities, they also come with a downside of being more complex and requiring higher-levels of technical sophistication. While our aim is to positively contribute to humanity, one can unfortunately envisage nonethical use of imaging technology.
Acknowledgments and Disclosure of Funding
Research presented in this article was supported in part by the NSF CCF-2043134. This work is also supported by the NIH RO1 E01323327 RF1 AG082003,, RFINS116565, and R2INS127425.
References
[1] M. T. McCann, K. H. Jin, and M. Unser, "Convolutional neural networks for inverse problems in imaging: A review," IEFF Signal Process. Mag., vol. 34, no. 6, pp. 85-95, 2017.
[2] A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos, "Using deep neural networks for inverse problems
in imaging: Beyond analytical methods," IEEE Signal Process, Mag., vol. 35, no. 1, pp. 20-36, Jan. 2018.
[3] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, "Deep learning techniques for inverse problems imaging." " IEFF J. Sel. Areas Inf. Theory, vol. 1, no. 1, pp. 39-56, May 2020.
[4] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, "Plug-and-play priors for model based reconstruc- tion," in Proc. IEEE Global Conf. Signal Process, and Inf. Process., Austin, TX, USA, Dec. 3-5, 2013, pp. 945-948.
[5] S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, "Plug-and- play priors for bright field electron tomography and sparse interpolation," IEEE Trans. Dec. 2016.
Comput. Imaging, vol. 2, no. 4. pp. 408 408-423,
[6] C. Metzler, P. Schniter, A. Veeraraghavan, and R. Baraniuk, "prDeep: Robust phase retrieval with flexible deep network." in Proc. Int. Conf. Mach. Learn., Stockholmsmässan, Stockholm Sweden, Jul. 10-15 2018, pp. 3501-3510.
[7] K. Zhang, W. Zuo,, Gu, and I.. Zhang, "Learning deep CNN denoiser prior for image restoration," in Proc. IEFF Conf. Comput. Vis. Pattern Recognit., Honolulu, USA, July 21-26, 2017, pp. 3929-3938.
[8] T. Mcinhardt, M. Moeller, C. Hazirbas, and D. Cremers, "Learning proximal operators: Using denoising networks for regularizing inverse imaging problems," in Proc. IEEE Int. Conf. Comp. Vis., Venice, Italy, Oct. 22 22-29, 2017, pp. 1799 1808.
[9] W. Dong, P. Wang. W. Yin, G. Shi, F. Wu, and X. Lu, "Denoising prior driven deep neural network for image restoration." IEEE Trans. Pattern Anal. Mach. Intell, vol. 41, no. 10. pp. 2305-2318. Oct 2019.
[10] K. Zhang, W. Zuo, and L. Zhang, "Deep plug-and-play super-resolution for arbitrary blur kernels"" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Long Beach, CA, USA, June 16-20, 2019, pp. 1671-1681.
[11] K. Wei, A. Aviles-Rivero, J. Liang, Y. Fu, C.-B. Schönlieb, and H. Huang. "Tuning-fre free plug-and-pla proximal algorithm for inverse imaging problems," in Proc. Int. Conf. Mach. Learn., 2020.
10

---

[12] K. Zhang, Y. Li. W. Zuo, L. Zhang, L. Van Gool, and R. Timofte, "Plug-and-play image restoration with deep denoiser prior." IEEE Trans. Patt. Anal. and Machine Intell., 2022.
[13] R. Liu, Y. Sun, J. Zhu, L. Tian, and U. S. Kamilov, "Recovery of continuous 3D refractive index maps from discrete intensity-only measurements using neural fields," Nat. Mach. Intell., vol. 4, pp. 781 7911 Sept. 2022.
[14] R. Ahmad, C. A. Bouman, G. T. Buzzard, S. Chan, S. Liu, E. T. Reehorst, and P. Schniter. "Plug-and-play methods for magnetic resonance imaging: Using denoisers for image recovery." IEEE Signal Process. Mag., vol. 37, no. I. pp. 105-116, 2020.
[15] U. S. Kamilov, C. A. Bouman, G. T. Buzzard, and B. Wohlberg, "Plug-and-play methods for integrating physical and learned models in computational imaging." IEEE Signal Process. Mag., vol. 40, no. pp. 85-97, Jan. 2023.
[16] S. H. Chan, X. Wang, and O. A. Elgendy, "Plug-and-play ADMM for image restoration: Fixed-point convergence and applications," IEEE Trans. Comp. Imag. vol. 3, no. 1. pp. 84-98, Mar. 2017.
[17] Y. Romano, M. Elad, and P. Milanfar, "The little engine that could: Regularization by denoising (RED)."
SIAM J. Imaging Sei., vol. 10, no. 4. pp. 1804 1844, 2017.
[18] G. T. Buzzard, S. H. Chan, S. Sreehari, and C. A. Bouman, "Plug-and-play unplugged: Optimization free reconstruction using consensus equilibrium," " SIAM J. Imaging Sci., vol. 11, no. 3, pp. 2001-2020 Sep. 2018.
[19] E. T. Reehorst and Schniter, "Regularization by demoising: Clarifications and new interpretations." IEEE Trans. Comput. Imag., vol. no. I, pp. 52 67, Mar. 2019.
[20] Y. Sun, B. Wohlberg, and U. S. Kamilov, "An online plug-and and-play algorithm for regularized image reconstruction," IEEE Trans. Comput. Imag., vol. 5, no. 3, pp. 395-4088 Sept. 2019.
[21] Y. Sun, J. Liu, and S. Kamilov, "Block coordinate regularization by denoising," in Proc. Ady. Neural Inf. Process. Syst., Vancouver, BC, Canada, Dec. 2019. pp. 382-392.
[22] Ernest K. Ryu, J. Liu. S. Wang, X. Chen, Z. Wang. and W. Yin, "Plug-and-play methods provably converge with properly trained denoisers," in Proc. Int. Conf. Mach. Learn Long Beach, CA, USA, Jun. 09-15 2019, vol. 97, pp. 5546-5557.
[23] X. Xu, Y. Sun, J. Liu, B. Wohlberg, and U. Kamilov, "Provable convergence of plug-and-play priors with mmse denoisers," " IEEE Signal Process. Lett., vol. 27, pp. 1280-1284, 2020.
[24] J. Liu, S. Asif. B. Wohlberg, and U. S. Kamilov, "Recovery analysis for plug-and-pul priors using the restricted eigenvalue condition." in Proc. Adv. Neural Inf. Process. Syst., December 6-14, 2021, pp. 5921-5933.
[25] Z. Kadkhodaic and E. P. Simoncelli, "Stochastic solutions for linear inverse problems using the prior implicit in demoiser," in Proc. Adv. Neural Inf. Process. Syst., December 6-14, 2021, pp. -3242-13244
[26] R. Cohen, Y. Blau, D. Freedman, and E. Rivlin, "It has potential: Gradient-driven denoisers for convergent solutions to inverse problems," in Proc. Adv. Neural Inf. Process. Syst., December 6-14, 2021, pp. 18152 18164.
[27] S. Hurault, A. Leclaire, and N. Papadakis, "Gradient step denoiser for convergent plug-and-plas -lad.ppy." in Proc. Int. Conf. Learn. Represent., 2022.
[28] S. Hurault, A. Leclaire, and N. Papadakis, "Proximal denoiser for convergent plug-and-play optimization with nonconvex regularization." in Proc. Int. Conf. Mach. Learn., 2022, pp. 9483-9505.
[29] R. Laumont, V. De Bortoli, A. Almansa, J. Delon, A. Durmus, and M. Pereyra, "Bayesian imaging using plug & play priors: When Langevin meets Tweedie," SIAM J. Imaging Scí., vol. 15, no. 2, pp. 701-737, 2022.
[30] P. Campisi and K. Egiazarian, Blind image deconvolution: theory and applications, CRC press, 2017.
[31] J. A. Fessler, "Optimization methods for magnetic resonance image reconstruction," IEEE Signal Process. Mag., vol. 1, no. 37, pp. 33 40, Jan. 2020.
[32] L. Ying and J. Sheng, "Joint image reconstruction and sensitivity estimation in SENSE (JSENSE)" Magn. Reson. Med., vol. 57, no. 6, pp. 1202- June 2007.
[33] M. Uecker, T. Hohage, K. T. Block, and Frahm, "Image reconstruction by regularized nonlinear inversion-Joint estimation of coil sensitivities and image content," Magn. Rexon. Med., vol. 60, no. 3, pp. 674-682, Sept. 2008.
[34] Y. Jun, H. Shin, T. Eo, and D. Hwang, "Joint deep model-basee MR image and coil sensitivity reconstruction network (joint-icnet) for fast MRI." in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2021, pp. 5270- 5279.
11

---

[35] H C. M Holme, S. Rosenzweig, F. Ong, R. N Wilke. M. Lustig, and M. Uecker, "ENLIVE: An efficient nonlinear method for calibrationless and robust parallel imaging, Scientific reports, vol. 9, no. I, pp. 1-13, 2019.
[36] M. Arvinte, S. Vishwanath, A, H. Tewfik, and J. L. Tamir, "Deep J-Sense: Accelerated MRI reconstruction
via unrolled alternating optimization." in Proc. Med. Image. Comput. Comput. Assist. Intervent, Apr. 2021.
[37] A. Sriram, Zbontar, T. Murrell, A. Defazio. C L. Zitnick, N. Yakubova, F. Knoll, and P. Johnson. "End-to-end variational networks for accelerated MRI reconstruction," in Proc. Med. Image. Comput.
Comput. Assist. Intervent, 2020, pp. 64-73.
[38] D. Kundur and Hatzinakos, "Blind image deconvolution," IEEE Signal Process. Mag., vol. 13, no. 3.
pp. 43 64, 1996.
[39] J. Pan, D. Sun, H. Pfister, and M. Yang, "Deblurring images via dark channel prior," IEFF Trans. Pottern Anal. Mach. Intell., vol. 40, no. 10. pp. 2315-2328, 2017.
[40] E. Malhotra and A. Rajwade, "Tomographic reconstruction from projections with unknown view angles exploiting moment boaeett-based relationships." in Proc. IEEE Int. Conf. Image Proc., 2016, pp.
[41] D. C Lee, P. Hoffmann, D. I. Kopperdahl, and T. M Keaveny, "Phantomless calibration of ct scans for measurement of hmd and hone intenghh-imerr-pprrator reanalysis precision," " Bone, vol. 103, pp. 325-333, 2017.
[42] C. Cheng, Y. Ching, P. Ko, and Y. Hwu, "Correction of center of rotation and projection angle in synchrotron x-ray computed tomography." Scientific reports, vol. 8, no. 1. pp. 9884, 2018
[43] M. Xie, J. Liu, Y. Sun, W. Gan, B. Wohlberg, and U. S. Kamilov, "Joint reconstruction and calibration using regularization by denoising with application computed tomography," in Proc. IRFF Int. Conf. Comp. Vis. Workshops, Octoher 2021, pp. 4028-4037.
[44] S. Wang. Z. Su, L. Ying. X. Peng, S. Zhu, F. Liang. D. Feng, and D. Liang, "Accelerating magnetic resonance imaging via deep learning," in Proc. Int. Symp. Biomedical Imaging, April 2016, pp. 514-517.
[45] K. H. Jin, M. McCann, Froustey, and M. Unser, "Deep convolutional neural network for inverse problems in imaging." " IEEE Trans. Image Process., vol. 26, no. 9, pp. 4509-4522, Sep. 2017.
[46] E. Kang. J. Min, and J. Ye. "A deep convolutional neural network using directional wavelets for low-dose X-ray CT reconstruction." " Medical Physics, vol. 44, no. 10, pp. e360-e375, 2017.
[47] H. Chen, Y. Zhang, M. K. Kalra, F. Lin, Y. Chen, P. Liao, J. Zhou, and G. Wang. "Low-dose CT with residual encoder-decoder convolutional neural network," IEEE Trans. Med. Imag., vol. 36, no. 12, pp. 2524 2535, Dec. 2017.
[48] X. Xu, J. Pan, Y. Zhang, and M. Yang, "Motion Blur Kernel Estimation via Deep Learning." " IEEE Trans. on Image Process., vol. 27, no. 1. pp. 194-205, Jan. 2018.
[49] V. Monga, Y. Li, and Y. C. Eldar, "Algorithm unrolling: Interpretable, efficient deep lcarning for signal and image processing," IEEE Signal Process. Mag., vol. 38, no. 2. pp. 44, Mar. 2021.
[50] A. Bora, A. Jalal, F. Price, and A. G. Dimakis, "Compressed sensing using generative priors," in Proc. Int. Conf. Mach. Learn., Sydney, Australia, Aug. 2017, pp. 537-546.
[51] Zhang and B. Ghanem, "ISTA-Net: Interpretable optimization-inspreed deep network for image compressive sensing," " in Pruc. IEEE Conf. Comput. Vis. Puttern Recognit., 2018, pp. 1828-1837.
[52] A. Hauptmann, F. Lucka, M. Betcke, N. Huynh, J. Adler, B. Cox, P. Beard, S. Ourselin, and S. Arridge, "Model-hased learning for accelerated, limited-view 3-d photoacoustic tomography," IEFF Trans. Med. Imag., vol. 37, no. 6, pp. 1382-1393, 2018
[53] D. Gilton, G. Ongie, and R. Willett. "Deep equilibrium architectures for inverse problems in imaging." IEEE Trans. Comput. Imag., vol. 7, pp. 1 113--1133, 2021.
[54] J. Liu, X. Xu, W. Gan, S. Shoushtari, and U. S. Kamilov, "Online deep equilibrium learning for regulariza- tion by denoising." in Proc. Adv. Neural Inf. Process. Syst. New Orleans, LA, 2022.
[55] Y. Han and IEEE Trans. Med. Imag.. vol. 37, no. 6, pp. 1418-1429. 2018
Yc, "Framing U-Net via deep convolutional framelets: Application to sparse-view CT,"
[56] X. Peng, B. Sutton, F. Lam, and Z. Liang, "Deepsense: Learning coil sensitivity functions for sense reconstruction using deep learning." " Magn, Reson. Med., vol. 87, no. pp. 1894 1902, 2022.
[57] F. Bostan, R. Heckel, M. Chen, M. Kellman, and Waller, "Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network," Optica, vol. 7, no. 6, pp. 559-562, 2020.
[58] D. Ren, K. Zhang, Q Wang, Q. Hu, and W. Zuo, "Neural blind deconvolution using deep priors," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3341-3350.
[59] M. Asim, F. Shamshad, and A. Ahmed, "Blind image deconvolution using deep generative priors," IEEE Trans. Comput. Imaging, vol. 6. pp. 1493-1506, 2020.
12

---

[60] H. Chung. J. Kim, S. Kim, and J. Ye, "Parallel Diffusion Models of Operator and Image for Blind Inverse Problems," in Proc. IEEE Conf. Compud. Vis. Pattern Recognit., 2023.
[61] Y. Hu, W. Gan, C. Ying, T. Wang, C. Eldeniz, J. Liu, Y. Chen, H. An, and U. S Kamilov, "SPICE: Self- Self-supervised learning for mri with automatic coil sensitivity estimation," arXiv:2210002584 2022.
[62] A. Gossard and P. Weiss, "Training Adaptive Reconstruction Networks for Blind Inverse Problems," "
arXiv:2202. / 1342, 2022.
[63] Y. Huang. E. Chouzenoux, and J.-C. Pesquet. "Unrolled variational bayesian algorithm for image blind deconvolution," IEEE Trans. Image Process. vol. 32, pp. 430 1600-445.200 2022.
[64] Y. Li, M. Tofighi. J. Geng, V. Monga. and Y. C Eldar, "Efficient and interpretable deep blind image deblurring via algorithm unrolling," IEEE Trans. Comput. Imaging, vol. 6, pp. 666-681, 2020.
[65] U. S. Kamilov, H. Mansour, and B. Wohlberg, "A plug-and-plays priors approach for solving nonlinear imaging inverse problems." " IEEE Signal. Proc. Let., vol. 24, no. 12, pp. 1872-1876, Dec. 2017.
[66] R. Gribonval, "Should penalized least squares regression be interpreted as maximum a posteriori estima- tion?." IEEE Trans. Signal Process., vol. 59, no. 5. pp. 2405- 2410, May 2011.
[67] T. Tirer and Giryes. "Image restoration by iterative denoising and backward projections." JEEE Trans. Image Process., vol. 28, no. 3, pp. 1220-1223 1234, Mar. 2019.
[68] A. M. Teodoro, M. Bioucas-Dias, and M. A. Figuciredo, "A convergent image fusion algorithm using scene- scenc-adapted Gaussian-mixtures based denoising," IEEE Trans. Image Process., vol. 28, no. 1, pp. 451 451-463, Jan. 2019.
[69] Y. Sun, Z. Wu, B. Wohlberg, and U. S. Kamilov, "Scalable plug-and-play ADMM with convergence guarantees." IEEE Trans. Comput. Imag., vol. 7, pp. 849-863, July 2021.
[70] R. Cohen, M. Elad. and Milanfar, "Regularization by denoising via fixed-point projection (red-pro))" SIAM J. Imaging Sci., vol. 14, no. 3, pp. 1374-1406, 2021.
[71] M. Ljubenovié and M. A. Figuciredo, "Blind image deblurring using class-adapted image priors," in Proc. IEEÉ Int. Conf. Image Pruc. IEEE, 2017. pp. 490-494.
[72] M. Ljubenović and M. A. Figueiredo, "Plug-and-play approach class-adapted blind image deblurring." " Int. J. Doc. Anal. Recognit., vol. 22, no. 2. pp. 79-97, 2019.
[73] C. J Pellizzari, M. F Spencer, and C. A Bouman, "Coherent plug-ant and-play: digital holographic imaging through atmospheric turbulence using model-based iterative reconstruction and convolutional neural networks," IEFE Trans. Comput. Imaging, vol. 6, pp. 1607-1621, 2020.
[74] J. Wright, "Coordinate descent algorithms," Math. Program., vol. 151, no. 1, pp. 3-34, June 2015.
[75] F. Knoll et al., "fastMRI: A publicly available raw k-space and DICOM dataset of knee images for accelerated MR image reconstruction using machine learning." Radiology: Artificial Intelligence, vol. 2, no. I, Pp. c190007, 2020.
[76] M. Uecker, P. Lai, M. J. Murphy, P. Virtue, M. Elad, J. M. Pauly, S. S. Vasanawala, and M. Lustig, "ESPIRiT" an eigenvalue approach to autocalibrating parallel mri: where sense meets grappa." Magn. Reson, Med., vol. 71, no. 3. pp. 990-1001, 2014.
[77] O. Ronneherger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmenta- tion," Proc. Med. Image, Comput. Comput. Assist, Intervent., 2015, pp. 234-241.
[78] O. Kupyn, T. Martyniuk, J. Wu, and Z. Wang. "Deblurgan-v2. Deblurring feaders-of-maanituee faster and better," in Proc. IEEE Int. Conf. Comput. Vis., 2019, pp. 8878-8887.
[79] K. Zhang, L. V. Gool, and R. Timofte, "Deep unfolding network for image super-resolution."" in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2020, pp. 3217-3226.
[80] D. Martin, C. Fowlkes, D. Tal, and J. Malik, "A database of human segmented natural images and its application evaluating segmentation algorithms and measuring ecological statistics." in Proc. IEFE Int. Conf. Comp. Vis., Vancouver. Canada, July 7-14, 2001, pp. 416-423.
[81] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Deep image prior." in Proc. IEEE Conf. Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18-22, 2018 pp. 9446-9454.
[82] R. Gribonval and P. Machart, "Reconciling "priors" & "priors" without prejudice?," in Proc. Adv. Neural Inf. Process. Syst., Lake Tahoe, NV, USA, December 5-10, 2013, pp. 2193-2201.
[83] D. P. Bertsekas, "Incremental proximal methods for large scale convex optimization." Math. Program. Ser.
B. vol. 129, pp. 163-195. 195, 2011.
[84] J. Bolte, S. Sabach, and M. Teboulle, "Proximal alternating linearized minimization for nonconvex and nonsmooth problems," Math. Program., vol. 146, pp. 459444994499 494, 2013.
13

---

[85] J. Mairal, "Incremental majorization-miniiizztiio
minimizatiinn optimization with application to large-scale machine learning," SIAM J. Optim., vol. 25, no. pp. 829-855, Jan. 2015.
[86] E. K. Ryu and W. Yin, Large-Scale Convex Optimization: Algorithms and Analyses via Monotone Operators, Cambridge University Press, 2023.
[87] Kazerouni, U. S. Kamilov, Bostan, and Unser, "Bayesian denoising: From MAP to MMSE using consistent cycle spinning." IEEE Signal Procexs. Lett., vol. 20, no. 3, pp. 249-252. March 2013.
[88] R. Gribonval and M. Nikolova, "On Bayesian estimation and proximity operators," Appl. Comput. Harmon. Anal., vol. 50, pp. 49-72, Jan. 2021.
[89] D. Kingma and J. Ba, "Adam: A method for stochastic optimization." in Proc. Int. Conf. on Learn. Represent, 2015.
14


Reviews:
[{"review_id": "SeT5zq7fCg", "reviewer": "Reviewer_tTXT", "summary": "Develops a plug-and-play technique for blind (unknown forward model) imaging inverse problems. Key idea is to use two denoising algorithms: One as a prior on the image and one as a prior model on the measurement operator (e.g., blur kernel). Also analyzes the convergence of the algorithm. Successfully applies (in simulation) the proposed algorithm to MRI with unknown coil sensitive maps and deblurring with unknown blur kernel.\n", "questions": "Diffusion models are essentially denoisers by a different name, therefore a comparison with [60] seems obligatory. They are essentially solving the same problem (though the current method has more theory). How do they compare?\n", "limitations": "Adequately discussed", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Investigates important and understudied problem\n\nProposed technique is well-justified and seemingly effective\n\nI like the paper's approach to framing the blind image reconstruction problem (equation (10)).\n\nEasy to read\n\nProposed method seems generally applicable to lots of problems and opens up and opportunity for lots of follow-up work\n\nCode will be shared publicly", "weaknesses": "Lacks comparisons with closest competitors. Particulary [60].\n\nMissing related work: There's a line of work on PnP for holography with unknown phase errors (e.g., [A]) that may be worth mentioning.\n\n[A] Pellizzari, Casey J., Mark F. Spencer, and Charles A. Bouman. \"Coherent plug-and-play: digital holographic imaging through atmospheric turbulence using model-based iterative reconstruction and convolutional neural networks.\" IEEE Transactions on Computational Imaging 6 (2020): 1607-1621.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive comments on our work. Please see below for our point-by-point responses to your comments. The PDF with additional simulations is available within the Author Rebuttal panel above.\n\n**Comparison with [60]:** \nPrompted by your comment, we ran simulations using **BlindDPS** from [60] on the blind deconvolution problem considered in our paper. We report the quantitative and visual results in **Table 1** and **Figure 1** in the PDF attached to our response, respectively. Note how BlindDPS achieves the SSIM value of 0.734, while BC-PnP achieves that of 0.921. Based on the other results reported on DPS, we are confident that BC-PnP is competitive with BlindDPS in terms of traditional image quality metrics such as PSNR or SSIM, while BlindDPS can do better on perceptual metrics such as FID. For example, Table 7 in the original DPS paper (`arXiv:2209.14687`) reports the superior performance of the traditional PnP-ADMM over DPS on Gaussian deblurring. It is also worth highlighting that there is value in doing more careful comparisons between BlindDPS and BC-PnP that include possible finetuning/retraining of models for specific settings (*e.g.*, types and sizes of blur kernels), which would be infeasible within this rebuttal window. Nonetheless, we will publicly release our code, which will enable easy comparisons and reproducibility of our results in the future.\n\n**Citing related work:**\nThank you for sharing [A], which we will cite in the final paper.\n"}, {"review_id": "D1eSjIKrW0", "reviewer": "Reviewer_NG5f", "summary": "The paper generalizes the plug-and-play framework from single-block to a multi-block case by embedding unknown inverse operators. It provides convergence analysis for the proposed method and conducted various experiments on two imaging applications, i.e., parallel MRI and blind image deblurring. Numerical results are sufficiently convincing to show that the proposed method works better than the other comparable PnP methods in the blind measurement operator case. ", "questions": "In all the numerical experiments, are the assumptions 1-5 and conditions in Theorems 1-2 verified or checked? If yes, the details could be included in the supplement.", "limitations": "The authors have listed the limitations and provided some potential solutions in future works. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper reads well and is well-organized from theoretical discussions to numerical experiments. The proposed method is original with clear motivation of unknown inverse operators from the realistic setting. Thus, it may also shed some light on other imaging applications. ", "weaknesses": "1. The computational cost of this framework is not analyzed or even mentioned. Likewise, the running times for all the experimental results could be listed and compared. Some other end-to-end methods may also be tested. \n2. Some minor notational confusion exists, e.g., lines 198-199. \n3. Theorems 1-2 require square-summable condition for the error sequences to ensure the convergence of gradients. Further discussions may be added about how this condition can be satisfied or checked in practice. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive comments on our work. Please see below for our point-by-point responses to your comments. The PDF with additional simulations is available within the Author Rebuttal panel above.\n\n**Computational cost:** Prompted by your comment, we computed running times of BC-PnP and PnP for both blind image deblurring and CS-PMRI. The results are presented in **Table 2** of the attached PDF and will be included in the revised supplement. As expected, BC-PnP is slower than traditional PnP due to the need for updating the parameters of the measurement operators. The computational overhead of BC-PnP over PnP depends on the size of the measurement operator. For instance, in the context of blind image deblurring, where the variables of the measurement operators are 25 x 25 kernels, BC-PnP's computational overhead is about 7% over the traditional PnP. We will publicly release our code, which should enable others to easily compare the efficiency and performance of BC-PnP relative to other methods, including any specific end-to-end methods.\n\n**Clarifying the notations:** We will clarify the notation for modulo operator and its use in lines 198-199 in the revised paper.\n\n**Conditions for Theorems 1-2:** We agree with the reviewer that the paper would benefit from a more careful discussion of our assumptions. We will include the following discussion in the main paper after presenting our theory. Since Assumptions 1-3 correspond to the standard assumptions used in the literature, they are broadly satisfied in the context of inverse problems. Assumption 4 was introduced by us and states that the error of the approximate MMSE denoiser used for inference is bounded relative to the exact MMSE denoiser, which is reasonable when the approximate MMSE denoiser is a CNN trained to minimize the MSE. Assumption 5 is another assumption introduced by us to expand the region where the regularizer associated with the exact MMSE denoiser is smooth by including the range of the approximate MMSE denoiser. For example, this assumption is automatically true when the exact and approximate MMSE denoisers have the same range, which is reasonable when the approximate MMSE denoiser is trained to imitate the exact one. It is also worth mentioning that Theorems 1-2 are interesting even when the denoiser errors are **not** square-summable, since they provide explicit error bounds on convergence.\n"}, {"review_id": "WRqMJ59eJ0", "reviewer": "Reviewer_7kUv", "summary": "This paper presents a new plug-and-play algorithm for solving blind inverse problems, i.e., where the forward sensing operator is not fully known. The method leverages pretrained denoisers for both the signals and the unknown parameters of the sensing model. A theoretical analysis of the proposed algorithm is presented, showing that under certain conditions on the denoisers and data fidelity functions, the algorithm is guaranteed to converge to a local minimizer.", "questions": "- What is the performance of the proposed method on the blind deblurring problem with non-isotropic kernels? It would be good to show an example in this setting.", "limitations": "The paper discusses some of the limitations of the proposed method. However, there is no discussion related to the inherent large computational complexity of PnP methods, that require many more iterations than unrolled networks to reach convergence.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a novel plug-and-play algorithm for blind inverse problems, with some theoretical guarantees under relatively mild assumptions.\n\n- The experimental results show a good performance with respect to other competing methods in two different blind inverse problems.", "weaknesses": "I find it a bit surprising that the PnP method outperfoms other unrolled algorithms trained for a specific inverse problem. Even the authors express that \"While denoisers provide a convenient, principled, and flexible mechanism to specify priors, they are inherently self-supervised and their empirical performance can thus be suboptimal compared to priors trained in a supervised fashion for a specific inverse problem\"\nI wonder if the comparison presented in the paper is fair, do the competing methods use architectures that have a similar expressive power than the DRUNet denoiser used for PnP? Moreover, there exist end-to-end architectures that can estimate both the forward operator parameters and the underlying image (e.g. \"Deep algorithm unrolling for blind image deblurring\" by Li et al.), which haven't been evaluated here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive comments on our work. Please see below for our point-by-point responses to your comments. The PDF with additional simulations is available within the Author Rebuttal panel above.\n\n**Commentary on deep unrolling:**\nWe fully agree with the reviewer that in non-blind settings deep unrolling (**DU**) methods are expected to outperform PnP. Prompted by your comments, we tested this hypothesis using the DU method USRNet on image deblurring. We present the results in **Table 3** of the attached PDF. Note how USRNet-oracle that uses the true blur kernel outperforms PnP-oracle. However, PnP outperforms USRNet when the kernel used for inference is inaccurate. Since the prior in PnP is independent from the measurement operator, it is not surprising that PnP is more robust to inaccuracies in the measurement operator.\n\nPrompted by your comment, we compared BC-PnP against DUBLID (“Deep algorithm unrolling for blind image deblurring\" by Li et al, 2019). We report the results in **Table 4** of the attached PDF. The results show that BC-PnP can quantitatively outperform DUBLID in blind image deblurring. Note that the blur kernels predicted by DUBLID differ in size compared to those used in our experiments. As a result, we excluded the quantitative evaluation of the predicted kernel from the table. We will cite DUBLID in the revised paper.\n\n**Blind deblurring beyond Gaussian blur:**\nPrompted by your comment, we compared BC-PnP in blind image deblurring with motion kernels. We followed `github.com/LeviBorodenko/motionblur` to simulate motion kernels for training and testing. We report results of BC-PnP and other methods in **Table 6** in the attached PDF. The results show that BC-PnP can outperform PnP by using a prior over the measurement operator. It is also worth noting that given the short duration of the rebuttal window, these results are preliminary. Nevertheless, we are confident that BC-PnP's performance can be further improved through fine-tuning of hyperparameters.\n\nComputational cost: \nWe agree with the reviewer that PnP running over many iterations can have higher computational complexity than unrolled networks with a small number of steps. We will incorporate this perspective into the limitations section of the final manuscript.\n"}, {"review_id": "GntTWletD1", "reviewer": "Reviewer_8yHJ", "summary": "The paper proposes a block-coordinate optimization strategy for plug-and-play (PnP) methods to solve blind inverse problems in imaging. Empirical validation is performed with blind deblurring and parallel CS-MRI. Theoretical study on the convergence of the proposed BC-PnP was extended from the theoretical result of PnP with MMSE denoisers.", "questions": "Please see weaknesses", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. To the best of my knowledge, this is one of the first works to extend PnP methods to blind inverse problems\n\n2. Theory is sound and proves proper convergence properties of the proposed method.\n\n3. The method seems to be robust and generally applicable to inverse problems arising in imaging, as validated in the experiments.", "weaknesses": "1. I do not find much novelty in the theory of the paper. While the components are all sound, both components of the extension (extending to block coordinate update and extending to denoisers that are optimal only up to a constant error) seems to be quite trivial. I would appreciate it if the authors could elaborate on this point.\n\n2. BC-PnP seems to rely quite heavily on initialization for the imaging operators (i.e. sensitivity maps, blur kernels) that are close to ground truth. Such initialization strategy is not always possible, and for many cases even when it is possible, is often computationally demanding. Does BC-PnP converge properly also for cases when there is no such initialization strategy? Adding on this point, it might not be fair to compare other blind inverse problem solver that do not leverage any initialization strategy.\n\n3. For blind deblurring, it seems that the authors only tested on Gaussian blur kernels, which are known to be relatively easy to fit. Does the method scale to more complex blur kernels, such as in [1,2]?\n\n\n**Referneces**\n\n[1] Levin, Anat, et al. \"Understanding and evaluating blind deconvolution algorithms.\" 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009.\n\n[2] Chung, Hyungjin, et al. \"Parallel diffusion models of operator and image for blind inverse problems.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and positive comments on our work. Please see below for our point-by-point responses to your comments. The PDF with additional simulations is available within the Author Rebuttal panel above.\n\n**Novelty in the theory:**\nWe respectfully disagree with the reviewer's evaluation of the novelty in our theory. On the contrary, we believe that our work presents one of the most compelling theoretical and conceptual extensions of PnP to date. Our theory extends the existing work on PnP by considering (**a**) non-convex data fidelity terms, (**b**) expansive-inexact denoisers, and (**c**) partial processing of data. Our theory addresses several common misconceptions about PnP in the community, including that (**a**) PnP requires convex data fidelity terms; (**b**) CNN priors in PnP require spectral normalization to ensure convergence; and (**c**) effectiveness of learned denoisers as priors is limited to images.\n\n**Commentary on initialization:**\nIndeed, initialization is a deep and important topic in the context of nonconvex optimization. Since BC-PnP is designed for nonconvex blind inverse problems, it certainly benefits from the availability of a good initialization method. Prompted by your comment, we ran additional simulations on CS-PMRI to test the performance of BC-PnP with another initialization strategy. In the new setting, we used the inverse Fourier transform of the central low-frequency k-space data (termed **Low-K**) as an initialization strategy for BC-PnP. This new Low-K method is known to be less optimal compared to the ESPRiT method used in the main paper, but offers significantly reduced computational complexity. We report the results in **Table 5** and **Figure 2** in the attached PDF. We also report convergence of BC-PnP under the Low-K initialization in **Figure 3** of the attached PDF. Overall, these results show that (**a**) while the performance of BC-PnP is slightly inferior when initialized via Low-K compared to ESPRiT, it still maintains its superior performance over the baseline methods, and (**b**) the convergence behavior of BC-PnP is similar for both initialization strategies.\n\n**Blind deblurring beyond Gaussian blur:**\nPrompted by your comment, we compared BC-PnP in blind image deblurring with motion kernels. We followed `github.com/LeviBorodenko/motionblur` to simulate motion kernels for training and testing. We report results of BC-PnP and other methods in **Table 6** in the attached pdf. The results show that BC-PnP can outperform PnP by using a prior over the measurement operator. It is also worth noting that given the short duration of the rebuttal window, these results are preliminary. Nevertheless, we are confident that BC-PnP's performance can be further improved through fine-tuning of hyperparameters."}]

Author rebuttal:
Thank you all for providing us with valuable feedback. We provide detailed answers to all the comments below. To better address some of them, we ran additional simulations, albeit within the time constraints of the limited rebuttal window. The PDF file attached to this response reports the results of these simulations. We will update the paper and the supplementary material for the camera ready submission based on our responses.

Meta-review: In this paper, the authors generalize Plug-and-Play approaches to solving blind inverse problems.  The approach uses two denoising algorithms, one for the image prior and one as a prior for the unknown measurement operator.  The authors extend existing theoretical convergence results to the non-convexity that arises due to blindness, among other novelties.  The provides results show good performance of the propose method relative to baselines in two blind inverse problems.  The strength of this paper is the nontrivial extension of existing PnP techniques, including experimental and theoretical results, to the challenging nonconvex setting of blind inverse problems.  Weaknesses of the original submission involve some missing baselines, but those were addressed during rebuttal.
Decision: Accept (poster)