SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For convenience, we will define quantities  $\varepsilon^2 \coloneqq \max \{\varepsilon_1^2, \varepsilon_2^2, \dots\}$  and  $\overline{\varepsilon}_t^2 \coloneqq (1/t)\left(\varepsilon_1^2 + \dots + \varepsilon_t^2\right)$  that correspond to the largest and the mean squared distances between the inexact and exact denoisers. Assumption 4 states that the error of the approximate MMSE denoiser used for inference is bounded relative to the exact MMSE denoiser, which is reasonable when the approximate MMSE denoiser is a CNN trained to minimize the MSE.
It has been shown in the prior work [23, 66] that the function  $h$  in (11) is infinitely continuously differentiable over  $\mathrm{Im}(\mathsf{D}_{\sigma}^{*})$ . Our analysis requires the extension of the region where  $h$  is smooth to include the range of the approximate MMSE denoiser, which is the goal of our next assumption.
Assumption 5. Each regularizer  $h_i$  in (12) associated with the MMSE denoiser (8) is continuously differentiable and has a Lipschitz continuous gradient with constant  $M_i > 0$  over the set
$$
\left. \operatorname {I m} _ {\varepsilon} \left(\mathrm {D} _ {\sigma_ {i}} ^ {*}\right) := \left\{\boldsymbol {x} \in \mathbb {R} ^ {n _ {i}}: \| \boldsymbol {x} - \mathrm {D} _ {\sigma_ {i}} ^ {*} (\boldsymbol {z}) \| _ {2} \leq \varepsilon , \boldsymbol {z} \in \mathbb {R} ^ {n _ {i}} \right\}, \quad i \in \{1, \dots , b \}. \right.
$$
We will define  $M_{\max} \coloneqq \max \{M_1, \dots, M_b\}$  to be the largest Lipschitz constant and  $\operatorname{Im}_{\varepsilon}(\mathsf{D}_{\sigma}^{*}) \coloneqq \{x \in \mathbb{R}^n : x_i \in \operatorname{Im}_{\varepsilon}(\mathsf{D}_{\sigma_i}^*)$ ,  $i \in \{1, \dots, b\} \}$  to be the set over which  $h$  is smooth. Assumption 5 expands the region where the regularizer associated with the exact MMSE denoiser is smooth by including the range of the approximate MMSE denoiser. For example, this assumption is automatically true when the exact and approximate MMSE denoisers have the same range, which is reasonable when the approximate MMSE denoiser is trained to imitate the exact one.
Our first theoretical result considers the sequential updates, where at each iteration,  $i_k$  is selected as  $i_k = 1 + \bmod (k - 1,b)$  with  $\bmod (\cdot)$  being the modulo operator. We can then express any iterate  $ib$  produced by BC-PnP for  $i \geq 1$  as
$$
\boldsymbol {x} ^ {i b} = \left(\boldsymbol {x} _ {1} ^ {i b}, \dots , \boldsymbol {x} _ {b} ^ {i b}\right) = \left(\boldsymbol {x} _ {1} ^ {(i - 1) b + 1}, \dots , \boldsymbol {x} _ {b} ^ {i b}\right).
$$
Note that  $\pmb{x}^{ib} \in \mathrm{Im}_{\varepsilon}(\mathrm{D}_{\sigma}^{*})$  since each block is an output of the denoiser. We prove the following result.
Theorem 1. Run BC-PnP under Assumptions 1-5 using the sequential block selection and the step  $0 < \gamma < 1 / L_{\max}$ . Then, we have
$$
\min  _ {1 \leq i \leq t} \| \nabla f (\boldsymbol {x} ^ {i b}) \| _ {2} ^ {2} \leq \frac {1}{t} \sum_ {i = 1} ^ {t} \| \nabla f (\boldsymbol {x} ^ {i b}) \| _ {2} ^ {2} \leq \frac {C _ {1}}{t} (f (\boldsymbol {x} ^ {0}) - f ^ {*}) + C _ {2} \bar {\varepsilon} _ {t b} ^ {2},
$$
where  $C_1 > 0$  and  $C_2 > 0$  are iteration independent constants. If additionally the sequence of error terms  $\{\varepsilon_i\}_{i\geq 1}$  is square-summable, we have that  $\nabla f(\pmb{x}^{tb})\to \mathbf{0}$  as  $t\to 0$
Our second theoretical result considers fully random updates, where at each iteration,  $i_k$  is selected as i.i.d. random variables distributed over  $\{1, \dots, b\}$ . In this setting, we analyze the convergence of BC-PnP in terms of the sequence  $\{\mathsf{G}(\pmb{x}^k)\}_{k \geq 0}$ . Note that it is straightforward to verify that  $\mathrm{Zer}(\mathsf{G}) = \mathrm{Zer}(\nabla f)$ , which makes this analysis meaningful. We prove the following result.
Theorem 2. Run BC-PnP under Assumptions 1-5 using the random i.i.d. block selection and the step  $0 < \gamma < 1 / L_{\max}$ . Then, we have
$$
\min  _ {1 \leq k \leq t} \mathbb {E} \left[ \| \mathrm {G} (\boldsymbol {x} ^ {k - 1}) \| _ {2} ^ {2} \right] \leq \mathbb {E} \left[ \frac {1}{t} \sum_ {k = 1} ^ {t} \| \mathrm {G} (\boldsymbol {x} ^ {k - 1}) \| _ {2} ^ {2} \right] \leq \frac {D _ {1}}{t} (f (\boldsymbol {x} ^ {0}) - f ^ {*}) + D _ {2} \bar {\varepsilon} _ {t} ^ {2},
$$
where  $D_{1} > 0$  and  $D_{2} > 0$  are iteration independent constants. If additionally the sequence of error terms  $\{\varepsilon_i\}_{i\geq 1}$  is square-summable, we have that  $\mathsf{G}(\pmb{x}^t)\xrightarrow{\mathrm{a.s.}}\mathbf{0}$  as  $t\to \infty$
The expressions for the constants in Theorems 1 and 2 are given in the proofs. The theorems show that if the sequence of approximation errors is square-summable, BC-PnP asymptotically achieves a stationary point of  $f$ . On the other hand, if the sequence of approximation errors is not square-summable, the convergence is only up to an error term that depends on the average of the squared approximation errors. Both theorems can thus be viewed as more flexible alternatives for the convergence analysis in [21]. It is also worth mentioning that the theorems are interesting even when the denoiser errors are not square-summable, since they provide explicit error bounds on convergence. While the analysis in [21] assumes convex  $g$  and nonexpansive  $\mathsf{D}_{\sigma}$ , the analysis here does not require these two assumptions. It instead views the denoiser  $\mathsf{D}_{\sigma}$  as an approximation to the MMSE estimator  $\mathsf{D}_{\sigma}^{*}$ , where the approximation error is bounded by  $\varepsilon_{k}$  at every iteration of BC-PnP. This view is compatible with denoisers trained to minimize the MSE loss (9).
Figure 1: Illustration of the BC-PnP convergence using the sequential and random i.i.d. block selection rules on CS-PMRI with the sampling factor  $R = 8$ . Leftmost two plots: Evolution of the distance between two consecutive image and CSM iterates. Rightmost three plots: Evolution of the RMSE and SSIM metrics relative to the true solutions across BC-PnP iterations. Note how both block selection rules lead to a nearly identical convergence behaviour of BC-PnP in this experiment.
5 Numerical Validation
We numerically validate BC-PnP on two blind inverse problems: (a) compressed sensing parallel MRI (CS-PMRI) with automatic coil sensitivity map (CSM) estimation and (b) blind image deblurring. We adopt the traditional  $\ell_2$ -norm loss in (10) as the data-fidelity term for both problems. We will use  $\pmb{x}$  to denote the unknown image and  $\pmb{\theta}$  to denote the unknown parameters of the measurement operator. We use the relative root mean squared error (RMSE) and structural similarity index (SSIM) as quantitative metrics to evaluate the performance.
We experimented with several ablated variants of BC-PnP, including PnP,  $\mathrm{PnP - GD_{\theta}}$ , and PnP-oracle.  $\theta$ . PnP and  $\mathrm{PnP - oracle}_{\theta}$  denote basic variants of PnP that use pre-estimated and ground truth measurement operators, respectively.  $\mathrm{PnP - GD_{\theta}}$  is a variant of PnP based on [43], where  $\theta$  is estimated without any DL prior. It is worth noting that PnP-oracle is provided as an idealized reference in our experiment. As discussed in the following subsections, we also compare BC-PnP against several widely-used baseline methods specific to CS-PMRI and blind image deblurring.
5.1 Compressed Sensing Parallel MRI
The measurement operator of CS-PMRI consists of complex measurement operators  $A(\theta) \in \mathbb{C}^{m \times n}$  that depend on unknown CSMs  $\{\theta_i\}$  in  $\mathbb{C}^n$ . Each sub-measurement operator can be parameterized as  $A_i(\theta_i) = PF\mathrm{diag}(\theta_i)$ , where  $F$  is the Fourier transform,  $P \in \mathbb{R}^{m \times n}$  is the sampling operator, and  $\mathrm{diag}(\theta_i)$  forms a matrix by placing  $\theta_i$  on its diagonal. We used T2-weighted MR brain acquisitions of 165 subjects obtained from the validation set of the fastMRI dataset [75] as the fully sampled measurement for simulating measurements. We obtained reference  $\theta_i$  from the fully sampled measurements using ESPIRiT [76]. These 165 subjects were split into 145, 10, and 10 for training, validation, and testing, respectively. BC-PnP and baseline methods were tested on 10 2D slices, randomly selected from the testing subjects. We followed [75] to retrospectively undersample the fully sampled data using 1D Cartesian equispaced sampling masks with  $10\%$  auto-calibration signal (ACS) [76] lines. We conducted our experiments for acceleration factors  $R = 6$  and 8. We adopted DRUNet [12] as the architectures of  $\mathrm{D}_{\sigma}$  for training both the image and CSM denoisers. BC-PnP and its ablated variants are initialized using CSMs  $\theta_0$  pre-estimated using ESPIRiT [76] and images  $x_0 \gets A(\theta_0)^{\mathrm{H}}y$ , where  $A^{\mathrm{H}}$  denotes the Hermitian transpose of  $A$ .
We considered several baseline methods, including ENLIVE [35], ESPIRiT-TV [76], Unet [77], and ISTANet+ [51]. ENLIVE is an iterative algorithm that jointly estimates images and coil sensitivity profiles. ESPIRiT-TV is an iterative algorithm that applies TV reconstruction method in (1). Unet is trained to map raw measurements to desired ground truth without the knowledge of measurement operator. ISTANet+ denotes a widely-used DU architecture. We tested ESPIRiT-TV and ISTANet+ using CSMs pre-estimated using ESPIRiT.
Figure 1 illustrates the convergence behaviour of BC-PnP on the test set for the acceleration factor  $R = 8$ . Figure 2 illustrates reconstruction results for the acceleration factor  $R = 6$ . Table 1 summarizes the quantitative evaluation of BC-PnP relative to other PnP variants and the baseline methods. These results show that joint estimation can lead to significant improvements and that BC-PnP can perform as well as the idealized PnP-oracle $\theta$  that knows the true measurement operator.
Figure 2: Illustration of results from several well-known methods on CS-PMRI with the sampling factor  $R = 6$ . The quantities in the top-left corner of each image provide RMSE and SSIM values for each method. The squares at the bottom of each image visualize the error and the corresponding zoomed area in the image. Note how BC-PnP using a deep denoiser on the unknown CSMs outperforms uncalibrated PnP and matches PnP-oracle $_{\theta}$  that knows the true CSMs.
Table 1: RMSE and SSIM performance of several methods on CS-PMRI. The table highlights the best and second best results. The Calibration column highlights methods specifically designed to solve the blind inverse problem. Note how the use of a DL prior over the measurement operator enables BC-PnP to outperform PnP and  $\mathrm{PnP - GD}_{\theta}$  and approach the performance of the oracle algorithm.
Method	Calibration (Y/N)	R = 6	R = 8
RMSEx ↓SSIMx ↑RMSEθ ↓	RMSEx ↓SSIMx ↑RMSEθ ↓
ENLIVE [35]	✓	0.371	0.763	—	0.419	0.730	—
ESPIRiT-TV [76]	✓	0.218	0.884	0.256	0.361	0.818	0.356
Unet [77]	X	0.218	0.904	—	0.195	0.907	—
ISTANet+ [51]	X	0.110	0.946	—	0.140	0.928	—
PnP	X	0.111	0.950	0.256	0.171	0.924	0.356
PnP-GDθ [43]	✓	0.116	0.950	0.254	0.163	0.926	0.355
BC-PnP (Ours)	✓	0.091	0.961	0.247	0.122	0.946	0.337
PnP-oracleθ*	X	0.069	0.969	0.000	0.082	0.962	0.000
* not available in practice for blind inverse problems.
5.2 Blind Image Deblurring
The measurement operator in blind image deblurring can be modeled as  $A(\theta) x = \theta * x$ , where  $\theta$  is the unknown blur kernel,  $x$  is the unknown image, and  $*$  is the convolution. We randomly selected 10 testing ground truth image from CBSD68 [80] dataset. We generated  $25 \times 25$  Gaussian kernels with  $\sigma = 10^{1}$ . We tested the algorithms on 2 Gaussian kernels. We used a pre-trained image denoiser, as in the experimental setting of [12]. The kernel denoiser was trained on 10,000 generated kernels at several noise levels. We adopted DnCNN with 17 layers as the architectures of  $\mathsf{D}_{\sigma}$  for training kernel denoisers. BC-PnP and its ablated variants are initialized with the blur kernels  $\theta_0$  pre-estimated using Pan-DCP [39] and images  $x_0 \gets A(\theta_0)^{\top} y$ .
We compared BC-PnP against several baseline methods, including Pan-DCP [39], SelfDeblur [58], DeblurGAN [78], USRNet [79]. Pan-DCP is an optimization-based method that jointly estimates image and blur kernel. SelfDeblur trains two deep image priors (DIP) [81] to jointly estimate the blur kernel and the image. DeblurGAN is a supervised learning-based method that lacks the capability for kernel estimation, but can reconstruct images via direct inference. USRNet is a DU baseline that was tested using blur kernel estimated from [39]. The results of DeblurGAN and USRNet are obtained by running the published code with the pre-trained weights.
Figure 3 illustrates the reconstruction results with a Gaussian kernel. Figure 3 demonstrates that BC-PnP can reconstruct the fine details of the snake skin, as highlighted by the white arrows, while both Pan-DCP and PnP produce smoother reconstructions. Additionally, BC-PnP generates a more accurate blur kernel compared to the ground truth kernel, whereas Pan-DCP and SelfDeblur yield blur kernels with artifacts. Table 2 presents the quantitative evaluation of the reconstruction results using
<sup>1</sup>We used github.com/shangqigao/BayeSR for generating the kernels.
Figure 3: Illustration of results from several well-known methods on blind image deblurring with the Gaussian kernel. The squares at the top of each image show the estimated kernels. The quantities in the top-left corner of each image provide RMSE and SSIM values for each method. The squares at the bottom of each image highlight the error and the corresponding zoomed image region. Note how the BC-PnP using a deep denoiser on the unknown kernel significantly outperforms the traditional PnP method and matches the performance of the oracle PnP method that knows the true blur kernel. Note also the effectiveness of BC-PnP for estimating the unknown blur kernel.
Table 2: Quantitative evaluation of BC-PnP in blind image deblurring. We highlighted the best and second best results, respectively. The Calibration column highlights methods specifically designed to solve the blind inverse problem. Note how the use of a prior over the measurement operator enables BC-PnP to nearly match the performance of the oracle algorithm.
Method	Calibration (Y/N)	■	■
RMSEx ↓SSIMx ↑RMSEθ ↓	RMSEx ↓SSIMx ↑RMSEθ ↓
Pan-DCP [39]	✓	0.087	0.835	0.283	0.114	0.733	0.246
SelfDeblur [58]	✓	0.219	0.495	0.775	0.176	0.553	0.831
DeblurGAN [78]	X	0.090	0.823	—	0.118	0.716	—
USRNet [79]	X	0.106	0.855	—	0.114	0.769	—
PnP	X	0.082	0.857	0.283	0.106	0.763	0.246
PnP-GDθ [43]	✓	0.082	0.857	0.283	0.108	0.767	0.246
BC-PnP (Ours)	✓	0.055	0.921	0.097	0.098	0.794	0.107
PnP-oracleθ*	X	0.051	0.929	0.000	0.088	0.817	0.000
* not available in practice for blind inverse problems.
a Gaussian kernel, indicating that BC-PnP outperforms the baseline methods and nearly matches the SSIM and RMSE values of  $\mathrm{PnP}$ -oracle $_{\theta}$  that is based on the ground truth blur kernel.
The work presented in this paper proposes a new BC-PnP method for jointly estimating unknown images and unknown measurement operators in blind inverse problems, presents its theoretical analysis in terms of convergence and accuracy, and applies the method to two well-known blind inverse problems. The proposed method and its theoretical analysis extend the recent work on PnP by introducing a learned prior on the unknown measurement operator, dropping the convexity assumptions on the data-fidelity term, and nonexpansiveness assumptions on the denoiser. The numerical validation of BC-PnP shows the improvements due to the use of learned priors on the measurement operator and the ability of the method to match the performance of the oracle PnP method that knows the true measurement operator. One conclusion of this work is the potential effectiveness of PnP for solving inverse problems where the unknown quantities are not only images.
The work presented in this paper comes with several limitations. The proposed BC-PnP method is based on PnP, which means that its performance is inherently limited by the use of AWGN denoisers as priors. While denoisers provide a convenient, principled, and flexible mechanism to specify priors,
they are inherently self-supervised and their empirical performance can thus be suboptimal compared to priors trained in a supervised fashion for a specific inverse problem. PnP running over many iterations can also have higher computational complexity compared to some end-to-end alternatives, such as DU with a small number of steps. Our analysis is based on the assumption that the denoiser used for inference computes an approximation of the true MMSE denoiser. While this assumption is reasonable for deep denoisers trained using the MSE loss, it is not directly applicable to denoisers trained using other common loss functions, such as the  $\ell_1$ -norm or SSIM. Finally, as is common with most theoretical work, our analysis only holds when our assumptions are satisfied, which might limit its applicability in practice. Our future work will investigate ways to improve on the results presented here by exploring new PnP strategies for relaxing assumptions for convergence, considering end-to-end trained variants of BC-PnP based on DEQ [53, 54], and exploring BC-PnP using explicit regularizers [26-28].
The expected impact of this work is in the area of imaging inverse problems with potential applications to computational imaging. There is a growing interest in computational imaging to leverage pretrained deep models for estimating the unknown image as well as the unknown parameters of the imaging system. The ability to better address such problems can lead to new imaging tools for biomedical and scientific studies. While novel DL methods, such as the proposed BC-PnP approach, have the potential to enable new technological capabilities, they also come with a downside of being more complex and requiring higher-levels of technical sophistication. While our aim is to positively contribute to humanity, one can unfortunately envisage nonethical use of imaging technology.
Research presented in this article was supported in part by the NSF CCF-2043134. This work is also supported by the NIH R01EB032713, RF1AG082030, RF1NS116565, and R21NS127425.
[1] M. T. McCann, K. H. Jin, and M. Unser, "Convolutional neural networks for inverse problems in imaging: A review," IEEE Signal Process. Mag., vol. 34, no. 6, pp. 85-95, 2017.
[2] A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos, "Using deep neural networks for inverse problems in imaging: Beyond analytical methods," IEEE Signal Process. Mag., vol. 35, no. 1, pp. 20-36, Jan. 2018.
[3] G. Ongie, A. Jalal, C. A. Metzler, R. G. Baraniuk, A. G. Dimakis, and R. Willett, "Deep learning techniques for inverse problems in imaging," IEEE J. Sel. Areas Inf. Theory, vol. 1, no. 1, pp. 39-56, May 2020.
[4] S. V. Venkatakrishnan, C. A. Bouman, and B. Wohlberg, "Plug-and-play priors for model based reconstruction," in Proc. IEEE Global Conf. Signal Process. and Inf. Process., Austin, TX, USA, Dec. 3-5, 2013, pp. 945-948.
[5] S. Sreehari, S. V. Venkatakrishnan, B. Wohlberg, G. T. Buzzard, L. F. Drummy, J. P. Simmons, and C. A. Bouman, "Plug-and-play priors for bright field electron tomography and sparse interpolation," IEEE Trans. Comput. Imaging, vol. 2, no. 4, pp. 408-423, Dec. 2016.
[6] C. Metzler, P. Schniter, A. Veeraraghavan, and R. Baraniuk, "prDeep: Robust phase retrieval with a flexible deep network," in Proc. Int. Conf. Mach. Learn., Stockholm Sweden, Jul. 10–15 2018, pp. 3501–3510.
[7] K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Honolulu, USA, July 21-26, 2017, pp. 3929-3938.
[8] T. Meinhardt, M. Moeller, C. Hazirbas, and D. Cremers, “Learning proximal operators: Using denoising networks for regularizing inverse imaging problems,” in Proc. IEEE Int. Conf. Comp. Vis., Venice, Italy, Oct. 22-29, 2017, pp. 1799-1808.
[9] W. Dong, P. Wang, W. Yin, G. Shi, F. Wu, and X. Lu, “Denoising prior driven deep neural network for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 10, pp. 2305–2318, Oct 2019.
[10] K. Zhang, W. Zuo, and L. Zhang, "Deep plug-and-play super-resolution for arbitrary blur kernels," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Long Beach, CA, USA, June 16-20, 2019, pp. 1671-1681.
[11] K. Wei, A. Aviles-Rivero, J. Liang, Y. Fu, C.-B. Schonlieb, and H. Huang, “Tuning-free plug-and-play proximal algorithm for inverse imaging problems,” in Proc. Int. Conf. Mach. Learn., 2020.
[85] J. Mairal, "Incremental majorization-minimization optimization with application to large-scale machine learning," SIAM J. Optim., vol. 25, no. 2, pp. 829-855, Jan. 2015.
[86] E. K. Ryu and W. Yin, Large-Scale Convex Optimization: Algorithms and Analyses via Monotone Operators, Cambridge University Press, 2023.
[87] A. Kazerouni, U. S. Kamilov, E. Bostan, and M. Unser, "Bayesian denoising: From MAP to MMSE using consistent cycle spinning," IEEE Signal Process. Lett., vol. 20, no. 3, pp. 249-252, March 2013.
[88] R. Gribonval and M. Nikolova, “On Bayesian estimation and proximity operators,” Appl. Comput. Harmon. Anal., vol. 50, pp. 49–72, Jan. 2021.
[89] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. on Learn. Represent., 2015.