Single-Shot Plug-and-Play Methods for Inverse Problems
Authors: Yanqi Cheng
Lipei Zhang
Zhenda Shen
Shujun Wang
Lequan Yu
Raymond H. Chan
Carola-Bibiane Schönlieb
Angelica I Aviles-Rivero
Venue: Accepted by TMLR ()

Abstract: The utilisation of Plug-and-Play (PnP) priors in inverse problems has become increasingly prominent in recent years. This preference is based on the mathematical equivalence between the general proximal operator and the regularised denoiser, facilitating the adaptation of various off-the-shelf denoiser priors to a wide range of inverse problems. However, existing PnP models predominantly rely on pre-trained denoisers using large datasets. In this work, we introduce Single-Shot PnP methods (SS-PnP), shifting the focus to solving inverse problems with minimal data. First, we integrate Single-Shot proximal denoisers into iterative methods, enabling training with single instances. Second, we propose implicit neural priors based on a novel function that preserves relevant frequencies to capture fine details while avoiding the issue of vanishing gradients. We demonstrate, through extensive numerical and visual experiments, that our method leads to better approximations.

Published in Transactions on Machine Learning Research (10/2024)
Single-Shot Plug-and-Play Methods for Inverse Problems
Yanqi Cheng', Lipei Zhang", " Zhenda Shen?" Shujun Wang", Lequan Yu², Raymond H. Chan Carola-Bibiane Schönlieb', Angelica I Aviles-Rivero 6.1
Depprtment of Applied Mathematics and Theoretical Physics, University of Cambridge
2 Deportment of Mathemalics, Cily University of Hong Kong
3 Biomedical Engineering, Hong Kong Polytechnic University
Department of Statistics and Actuarial Science, The University of Hong Kong
School of Data Science, Lingnan University
Yau Mathematical Sciences Center, Tsinghua University
Reviewed OpenReview:
Abstract
The utilisation of Plug-and-Play (PnP) priors inverse problems has become increasingly prominent in recent years. This preference is based on the mathematical equivalence between the general proximal operator and the regularised denoiser, facilitating the adaptation of various off-the-shelf denoiser priors to wide range of inverse problems. However, existing PnP models predominantly rely on pre-trained denoisers using large datasets. In this work, we introduce Single-Shot PnP' methods (SS-PnP)), shifting the focus to solving inverse problems with minimal data. First, we integrate Single-Shot proximal denoisers into iterative methods, enabling training with single instances. Second, we propose implicit neural priors based on novel function that preserves relevant frequencies to capture fine details while avoiding the issue of vanishing gradients. We demoustrate, through extensive numerical and visual experiments, that method leads to better approximations.
1 Introduction
Inverse problems have long been a fundamental challenge in the field of mathematics and applied sciences, encompassing a wide range of applications from image reconstruction to signal processing (Devaney, 2012; Bertero et al., 2021). Traditionally, these problems have been approached through various analytical and numerical methods (Vogel, 2002; Jordan, 1881; Metropolis & Ulam, 1949) using either single multiple images (without deep net learning process). However, the advent of deep learning has revolutionised this domain. Deep inverse problems present a modern approach, offering new insights and solutions where conventional methods have limitations (McCann et al., 2017). This shift towards leveraging machine learning techniques marks a significant evolution in tackling inverse problems, opening doors to more sophisticated and efficient problem-solving techniques.
A popular framework in this deep inverse problem era is Plug-and-Play (PnP) methods tarisana et al., 2013; Zhang al., 2021; Chan et al., 2016; Ono, 2017). At the of this approach lies the mathematical equivalence of the proximal operator to denoising (Venkatakrishnan et al., 2013), concept that intertwines optimisation theory with modern denoisers. This equivalence paves the way for the integration of advanced deep learning-based denoisers into the inverse problem-solving process. The Plug-and-Play framework essentially allows for the seamless insertion of these denoisers into iterative algorithmss enhancing their ability to recover high-quality signals or images from corrupted observations (Zhang et al., 2021; 2019; Ahmad et al., 2020).
*Joint contribution.

---

Published in Transactions on Machine Learning Research (10/2024)
Although PnP methods have demonstrated outstanding results across wide range of inverse problems, they predominantly rely on the assumption of having substantial training datasets for the development of robust denoising models (Arridge et al., 2019). This requirement often becomes a significant bottleneck, especially in scenarios where data availability limited or the diversity of data is not sufficiently reprosentative. Morcover, PnP methods lypically necessitate retraining or fine-tuning to adapt effectively to varying distribbttinn and signal characteristics. To address these challenges, Single-Shot learning emerges as a promising alternative, offering a paradigm shift in how deep inverse problems can be trained with minimal data. Unlike traditional methods that require extensive datasets, Single-Shot learning aims to make significant inferences from single instance, or in some cases, a small set of instances. To the best of our knowledge, there is no enisting work on Single-Shot PnP methods. Our work thus opens the door to a novel research line for Pul, introducing the concepl of Single-Shol Plug-ond-Playy melhods.
Single-Shot techniques offer a viable solution to the current constraints in PnP methods, particularly their reliance on extensive training datasots. This shift enables us to delve into the recent advancements in signal representation, specifically, the emergence of neural implicit representations. Neural implicit representations (Strümpler et al., 2022; Saragadam et al., 2023; Tancik et al., 2020) are trained to learn contimuous functions that map coordinates to signal values or features, making this form of representation highly efficient in capturing intricate data details with compact network architecture. Its application in the context of Single-Shot PnP methods is particularly promising, as denoisers that can effectively operate with limited data inputs, aligning with the Single-Shot learning paradigm. We therefore use implicit neural representations as a way represent the prozimal denoiser in PnP techniques in a Single-Shot fashion. Our contributions are:
G We propose the concept of Single-Shot Plug-and-Play (SS-PnP) Methods to solve inverse problems, in which we highlight:
We introduce Single-Shot proximal denoisers into iterative methods for solving any inverse problem. Our scheme eliminates the need for a pre-trained model, enabling training with single instance.
We introduce implicit neural priors for Plug-and-Play methods that enables the network to preserve more details during training. Additionally, we provide a theoretical justification for our prior, emphasising how its continuity and differentiability play crucial role in mitigating the issuc of vanishing gradients during the training process and preserve fine details.
& We demonstrate, through extensive experiments on several inverse problems, that our technique leads to a better approximation capturing finer details, smoother edge features and better colour representation. The method is evaluated on inverse problems with both single operator task and multi-operator task like joint demosaicing and deconvolution task. With only one image input in the whole reconstruction procoss, it outperforms the classical methods and pre-trained models among all the lasks.
2 Related Work
In this section, we review the existing literature and the concepts closely related to our work.
Plug-and-Play (PnP) Methods. They have revolutionised the field of inverse problems by integrating advanced denoisers into iterative algorithms. This innovative approach, initiated by Venkatakrishnan et al. (2013), has undergone significant evolution. Meinhardt et al. (2017) showcased its effectiveness in diverse imaging tasks, marking turning point in PnP's development. The subsequent works (Ryu et al., 2019; Sun et al., 2019; Hurault et al., 2022) further refined PnP, enhancing its stability and convergence, thus broadening its applicability.
In addition, the works of that (Teodoro et al., 2018; Yuan et al., 2020; Zhang et al., 2017b; Ono, 2017; Sun et al., 2019) further expanded the scope of PnP, demonstrating its adaptability in complex imaging scenarios. A notable advancement. in the optimisation landscape came with the introduction of TPPnP (Tuning- Free
2

---

Published in Transactions on Machine Learning Research (10/2024)
Plug-and-Plng (Wei et al,, 2020), which innovatively eliminated the need for parameter tuning in PnP algorithms.
Previous methods have relied on data-driven pre-training, which becomes impractical in situations with limited data or on smaller devices due to the extensive size of the required models. Consequently, developing a method for Single-Shot image prior denoising emerges as a compelling solution for such resource-constrainee environments.
A wide range of denoisers has been used within the PuP framework. Classical denoisers such as BM3D (Dabov et al., 2007) have been the most prevalent. Other notable approaches include Teodoro et al. (2016) and Venkatakrishnan et al. (2013). These traditional denoisers are well-established and often require little or no data pre-training. On the other hand, another emerging family of denoisers leverages deep learning techniques (Meinhardt et al., 2017; Zhang et al., 2017b; Laumont et al., 2022). These deep learning- based denoisers have gained prominence for their ability to capture complex features and patterns in data, making them highly effective in PnP frameworks. The aim of this paper to further explore and advance the use of deep learning- reasning-based denoisers, particularly in scenarios with minimal data, through the proposed Single-Shot methodology.
Single-Shot Image Denoising, A crucial element in PnP methods is the denoiser model. During the last years, the denoiser technique in PnP has evolved remarkably, transitioning from traditional methods to advanced deep learning techniques. Pioneering works such as the BM3D algorithm (Dabov et al., 2007) and the Non-Local Means (NLM) algorithm (Buades et al., 2005) laid the groundwork, setting significant benchmarks. The introduction of deep learning marked paradigm shift, exemplified by the DnCNN model proposed by Zhang et al. (2017a) and its variant DnCNN-S (Zhang et al., 2018), which demonstrated the efficacy of convolutional networks in denoising. The Deep Image Prior (DIP) by Lempilsky et al. (2018) furthered this progression, utilising the structure of convolutional networks as a prior for denoising.
A notable advancement is the rising of sclf-supervised methods, which revolutionised the Single-Shot denoising field by eliminating the need for clean training data. The Noise2Void framework by Krull et al. (2019) and the Noise2Self algorithm (Batson & Royer, 2019) are pioneering examples, utilising concepts like blind-spot these, Self2Self (Quan et al., 2020), Noisc2Same (Xie et al., 2020), and Noise2Info (Wang et al., 2023) further explored self-supervision, offering unique strategies for leveraging the inherent properties of noisy images. Additional approaches like CycleTSP (Zamir et al., 2020), IRCNN (Zhang et al., 2017b), GCDN (Anwar & Barnes, 2020), and bayesian denoising with blind-spot networks (Laine et al., 2019) further enrich the landscape, cach contributing novel perspectives and solutions to the challenge of Single-Shot image denoising.
networks for effective denoising. Building
These advancements are not confined to noise reduction alone; many of the developed deep denoisers are inherently adaptable and can be generalised to tackle various inverse problems in imaging. Inverse problems, such as demosaicing, denoising, and deconvolution, share common traits with denoising. The underlying principles and network architectures developed for denoising can often be extended or finc-tuned to address these challenges (Romano et al., 2017; Akyūz et al., 2020). Furthermore, the concept of Plug-and- methods opens up new avenues. However, this progress highlights a gap: despite the evolution of Single-Shal Shot denoisers, there is currently no work on Single-Shot denoisers into iterative algorithms. Therefore, this work introduces the concept of Single-Shot Plug-and-Plas methods.
Despite the versatility of Single-Shot image denoising, CNN-based estimators frequently fail to capture continuously high-frequency details crucial for image reconstruction. Implicit neural representation (LNR) emerges as a solution, adept at addressing these high-frequency challenges in inverse problems.
Implicit Neural Representation, characterised as a fully connected network-based method, has seen a rise in popularity for solving inverse problems as highlighted by Sun et al. (2021a). Traditional activation functions like ReLU have exhibited limitations in representing high fighffrequecyy' features, as discussed by Dabov et al. (2007). This shortcoming has led to the exploration of nonlinear activation functions, such as the sinusoidal function (Sitzmann et al., 2020), enhancing reprosentational capabilitios. The adaptability of INR is evident in its diverse applications across medical imaging (Wang et al., 2022), image processing (Chen et al., 2021; Attal et al., 2021), and super-resolution (Saragadam et al., 2023).
3

---

Published in Transactions on Machine Learning Research (10/2024)
71
Prox
No.
Prox
2 K
A
Figure 1: The pipeline of Single-Shot Plug-and-Plaay methods (SS-PnP). The 2 blocks indicate the steps as in Algorithm 1, and the * indicates the fix in denoiser weight over the ADMM iterations, where there are K iterations in total (i.e. e (0.1,..,KK1))
A distinct advantage of 1NR lies in its independence from pre-training, attributed to its rapid training capability, as indicated by Saragadam et al. (2023). This feature makes INR particularly suitable for Single-Shot image denoising tasks, where a single image suffices for effective learning, bypassing the need for extensive data-driven pre-training. The ability of INR to learn from minimal data points underscores its potential in resource-limited scenarios.
3 Methodology
Inverse imaging problems have emerged as a crucial cornerstone in computer vision and various related domains. A conventional forward model reads:
A(x) +
(1)
where y e JRM is the known observation, x e R is the unknown targel interest, e R the measurement. noise, and RR T is the forward measurement operator, which varies across different tasks.
The approximation of gives rise to highly ill-posed problem, necessitating regularisation. Consequently, the fundamental objective for any imaging inverse problem is the minimisation of:
min D(x) 7R(x),
(2)
ZERN
(1:) refers the data fidelity term, usually taking the form D(x) 1 (1=)) R(x) denotes the regularisation term, which encodes prior knowledge about The parameter y serves the regularisation weight, determining the trade-off between data. fidelity and regularisation. widely used approach to solve equation 2 is the family of first-order optimisation methods (Beck & Teboulle, 2009b; Boyd et al., 2011; Chambolle & Pock, 2011).
3.1 Single-Shot Proximal Denoiser
We introduce Single-Shot Plug-and-Play, methods (SS-PnP) demonstrated in Figure 1. The optimisation of equation 2 typically exhibits non-smooth characteristics due to R. A widely adopted strategy for addressing this problem is to employ first-order methods such as the alternating direction method of multipliers (ADMM). Given a function F(.), we define the proximal operator of at o with step size as:
Proxer(v) = argmin { I- 1/12 +8F(w).
(3)
Considering ADMM, one can express PDP-ADMM as:
k+1 Prox,3n (2 k x² = Hor (z =
k+1 Prox 1 D (e t
(4)
4 k+1 = rkt k+1 kt1
4

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1: Single-Shot Plug-and-Play
Forward model: y A(x) +E
Input: ye IR-
Output:
e R.N
Step 1 Train the Single-Shol Shot Denoiser
Choose noise strength e, and initialise fso for i (0. 1} do fe.. g(
& is independent of
Train the Single-Shol Shot denoising prior
end
fo fo,
Step 2
ADMM Iteration
Let H. = fø and x² AT
*A" denotes adjoint of A
Choose noise strength a: [ox]0 k- Initialisc 2d for h (0,1,2 10,1,,..,,K 13 do
and penalty parameter μæ: (4k)-- K-1
k+1 = Hop (z* - )²) k+1 Prox D (ckttl x²)
ktl x + kttl 2kt ktl
end
= =r
where Prox.qa(()) is the proximal operator the regularisation with noise strength 0g and Prox D mg is to enforce the data consistency (Ryu et al., 2019) with penalty parameter μ in the k-th iteration, for & é (0, 1. K 111. From equation * we can observe that Plug-and-Play (PnP) methods leverage the equivalence between the proximal operator Prox,gn() and denoiser Ho, with the denoising parameter og 0.
Single-Shot Denoiser is All You Need for PnP. PnP techniques primarily rely on denoisers, often trained on extensive datasets and using off-the-shelf deep denoisers (Ryu et al., 2019; Sun et al., 2019; Hurault et al., 2022). However, an unexplored question whether Single-Sho Shot denoisers can be used in iterative methods and what their properties are. To our knowledge, there are no eristing works on this.
In our Single-Shot denoising stage, we utilise the observed image with noise y to train the denoiser, enabling it to distinguish and mitigate complex noise and distortions specific to the example. The neural network fe aims to transform the noisy and corrupted image into its less corrupted counterpart. This is achieved through an optimisation process given by:
i argmin ((fe.y+E),
(5)
where L is loss function that evaluates the difference between the network's output and the corresponding observed image with noise. Refer to Step in Algorithm 1.
This pre-trained model subsequently integrated into a Plug-and-Play (PnP) framework as a prior for regularisation. Let H = fg. the trained denoiser serves as guiding force in the itcrative reconstruction process, enhancing the ability to recover high-quality images from corrupted observations. By embedding this Single-Shot learning model into the PuP framework, we establish potent approach for tackling range
5

---

Published in Transactions on Machine Learning Research (10/2024)
of inverse problems, particularly in situations involving severely corrupted images. To promote the synergy between Single-Shot learning and PnP methods, the so utilise nonlinear INR denoiser. Refer to Step 2 in Algorithm 1.
In this work, we consider ADMM justified by its well-known outstanding performance (Venkatakrishnan et al., 2013; Wei et al., 2022) in comparison to existing methods, its well-establiseed convergence properties (Ryu et al., 2019), and its modular approach, which collectively ensure its effectiveness across various inverse problem scenarios.
3.2 Implicit Neural Prior for Houg-andPlaa
Implicit neural representations enable the learning of continuous functions from signal. State-of-the-arr performance relies deep denoisers like UNet (Ronneberger al., 2015) and FFDNet (Zhang al., 2018). While convolutional-based denoisers have shown impressive results, there is currently the use of implicit neural representation (INR) as a prior for PnP methods. INR can be used as a Single-Shot approach in equation 4, we then open the door to new research direction for PnP.
research exploring
INR boasts remarkable expressive power and inductive bias, paving the way for innovative denoising approaches. Another significant contribution of this work the introduction new Single-Shot framework for PnP. particular, we present novel INR prior. Unlike the majority of existing works, we provide solid theoretical justification for its properties and behavior.
Zooming into our Prior. We next define our proposed implicit neural representation prior.
Our INR Prior
We define nonlinear activation function and form our implicit neural representation, which reads:
$ (x) exp( (aix bi) sin(aga b2)
(6)
exp(-(arx b1))
Our proposed activation function is nonlinear. Moreover, it has key properties of differentiabiliy and continuity.
Proof.
Let g(x) expl-(a1æ+ b and f(x) sin(a + b2). For Voe R,
-z) x-C f(x)g(x) [(DD(()--((D0))++((())))))))))))
4-c
f(x) (())-(()) g(x))( f() g(x) 9(c)
I-C
I-c
(7)
Consequently,
(z)) (f9)(c)
lim
n-c
--C [fxx- g(x) f(c) g(x) - g(c)
lim x-c
F-C
f(x)-f(c) F-C
lim (x) f(c) lim g(x) g((c)
lim
2-c
I
(8)
Because both g(x) and fxx are continuous and differentiable, lim (f))(e) exists for all e R. Hence, exp(-(a1æ ) sin + by) is continuous and differentiable for all ce R. Consequently, the entire activation function the of two differentiable functions, making it both continuous and differentiable. 7
I-c

---

Published in Transactions on Machine Learning Research (10/2024)
Why these properties interesting? The nonlinearity, contimuity, and differentiability brought by formula make it more representative compared to traditional activation functions, enabling the network to preserve more details during training, Additionally, the continuity and differentiability of our formula play a crucial role in mitigating the issue of vanishing gradients during training.
Convergence of our Prior
Consider the function (()) in equation then it holds that:
{1
((x)
x
I
This behavior ensures that (x) remains bounded and avoids gradient explosion, making it a stable function for optimisation.
Proof. For y> 0, Ve 0, 3ý V max I0 when y 1
exp (v") sin(agx b2) exp (-y²)
max 1, }
=xxp
(10)
exp(min(O,,In))) < exp((Ine) =
Meanwhile, the function (-y)-T converges to 1 when x co. Consequently, the function is convergent to when x + co.
We underline that previous proof that the convergence results present specifically in terms of the behaviour of the prior within our proposed framework. By leveraging the convergence and bounded nature of the activation function, we can significantly mitigate the issue of exploding gradients. Moreover, as the model converges to 0, it also effectively reduces the impact of outliers, enhancing the overall robustness of our model.
4 Experiment
In this section, we describe the experiments undertaken to framework.
validate our proposed Single-Shot Plug-and-Play
4.1 Experimental Setting
In our image preprocessing, we utilised dual resizing strategies. We sourced the images with Creative Commons Licenses and resized to 512 x 384, and in the meantime tested on the selected data in Bevilacqua et al. (2012) and Zeyde et al. (2012), without resizing. We remind to the reader, the experiments on Single-Shot Plug-and-Play methods (SS-PnP) considered only a single image input in the whole pipeline.
Training Scheme. During the initial implicit ncural representation (INR) pro-training phaso, Gaussian noise with a standard deviation in the range of [0.001,0.3] 0.5] was explored with 0.1 was set for all the experiments. The training conducted over 100 iterations, with network configuration comprising 2 hidden layers and 64 features per layer. The learning rate was set to 0.001. We then reconstruct the image for 5 ADMM iteration steps with dynamic noise strength and penalty parameter chosen by logarithmic descent. that gradually decreases value between 35 and 30 over steps.
For comparative analysis, the Noisc2Self pre-training scheme (Krull al., 2019), was applied three networks, each undergoes 100 training iterations with a learning rate of 0.01. The DrCNN (Zhang et al., 2017a) and FFDNet (Zhang et al., 2018) architectures, configured with 8 hidden
Evaluation Protocol.
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: The performance (PSNR(dB)) comparison of the Single-Shot deep denoising priors (DnCNN, FFDNet, UNet and our proposed *) on super resolution (SR) task with 2x and 4x upscalings the Plug-and-Play framework.
SS-PnP
Fractal
Wolf
Dog
Peacock
Tiger
Bird
(Backbone)
2x
4x
2x
4x
2x
4x
2x
4x
2x
4x
2x
4x
DrCNN
19.29
17.45
20.71
17.16
18.29
14.31
19.03
16.79
19.22
14.28
21.90
18.68
FFDNet
22.04
19.59
21.12
22.85
24.02
21.98
22.31
18.99
23.02
19.11
26.34
21.86
UNet
18.58
14.18
21.78
15.83
18.71
14.34
19.56
16.25
18.87
15.52
21.82
16.53
25.43
21.26
27.21
23.30
25.80
22.00
22.46
19.27
23.12
19.45
26.44
22.06
GT (x*)
Observed (y)
PSNR:
25.80
DrCNN
FFDNet
UNet
PSNR: 18.30
PSNR: 24.02
PSNR: 18.71
Figure 2: The comparative visualisation of super resolution with 2x upscaling task on 'Dog' example among the Single-Shot deep denoising priors (with * denotes our proposed prior) in Plug-and aan--and-Paay framework. The zommed-in view is provided at the right hand side of each result.
layers and 64 feature maps per layer. Conversely, the UNet (Ronneberger al., 2015) architect employed 4-times downsampling strategy and initiated with 32 feature maps in first convolutional operation. For fair comparison, we fixed the sctting for the optimisation step used in our proposed strategy.
The empirical studies are trained and tested on NVIDIA A10 GPU with 24GB RAM. The Plug-and-Play phase ensued with the v-Prox toolbox (Lai et al., 2023), adopting the default settings for all tasks. For evaluating our methods, we employed Peak Sigual-to- Noise Ratio (PSNR) and Structural Similarity Index (SSIM), Higher PSNR and SSIM scores signal superior reconstruction quality.
4.2 Results & Discussion
This section shows all the numerical and visual results that support our method.
' Super-resolution (SR). It refers to the process of reconstructing high-resolution image (HR) from or more low-resolution observations (LR). The forward measurement operator for super-resolution is given by: A(x) (k @ x) l., where & represents the kernel for convoluting the image, and 4. is the downscaling operation with scale 8. Here, kernel operator was set as size 5 with standard deviation of the Gaussian distribution as
In evaluating the Single-Sho Shot super-resolution efficacy of our method, we utilise six varied categories of images, with upscaling factors of 2x and 4x ensure fair comparison against well-established techniques such
8

---

Published in Transactions on Machine Learning Research (10/2024)
GT (x*)
Observed (y)
PSNR: 23.24
DOONN
FFDNet
UNet
PSNR: 15.32
PSNR: 23.19
PSNR: 14.37
Figure 3: The visualisation comparison of super resolution with 4x upscaling task on "Fox' example between the 4 Single-Shot deep denoising priors performing within Plug-and-Play framework, with detailed comparison zoomed in. * denotes our prior.
GT (x*)
Observed (y)
PSNR:
33.66
DrCNN
FFDNet
UNet
PSNR: 33.31
PSNR: 33.05
PSNR: 33.57
Figure 4: Visual comparison of the deep denoising priors (include our proposed as *) on deconvolution task in Single-Shot Plug-and-Play (SS-PnP) strategy for 'Giraffe" example, with zoomed-in region exhibiting intricate details.
DrCNN, FFDNet, and UNet, which are all trained with the Noise2Self pre-training scheme. The quantitative results, as detailed in Table 1, reveal our method's outstanding performance, outshining the benchmarks with significant margins. Notably, our approach demonstrates a marked improvement in PSNR values, with particularly pronounced enhancements in the upscaling scenario. These advancements indicate our model's robustness and substantial improvement forward in Singlo-Shot super resolution.
Our method demonstrates exceptional preservation of texture and detail complexity, shown in Figures 2, 3. Our technique achieves high-resolution enhancement while intricately reconstructing fine details, evident from the minimised artifacts such as grid patterns and spurious spots in Figure 2. In Figure 3, DnCNN introduces noticeable colour distortions in both the background and the foreground. The UNet generates numerous undesirable grids. Although our approach presents a comparable visual quality to FFDNet, achieves a superior PSNR, suggesting a quantitatively and qualitatively improved performance. Collectively, our results indicate that our method not only holds promise for practical application but also a new standard for super-resolution tasks.

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Evaluation comparison of the Single-Shot deep denoising priors within the Plug-and- aodg-and-Plaas and-Play framework (SS-PnP setting) on deconvolution task measured with PSNR(dB), and SSIM. We denote proposed Single-Shot denoising prior as *
SS-PnP
Racoon
Turtle
Tiger
Bird
Head
Monarch
(Backbone)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
DrCNN
27.43
0.88
26.24
0.89
23.75
0.84
30.67
0.93
29.87
0.87
28.26
0.93
FFDNet
19.81
0.49
25.03
0.84
19.62
0.62
16.72
0.55
26.50
0.72
28.09
0.92
UNet
27.06
0.88
26.17
0.90
23.72
0.84
31.06
0.94
29.90
0.87
28.49
0.94
27.56
0.89
26.30
0.90
23.76
0.84
31.07
0.94
29.92
0.87
28.55
0.94
GT (x**)
Observed (y)
TV
PSNR: 24.48
PSNR: 16.72
Pretrained -Utt
Pretzained FFDNet
Noise2Sell-UNe UNeb
PSNR: 18.50
PSNR: 23.30
PSNR: 24.42
PSNR: 16.11
Figure 5: The comparative visualisation of joint deconvolution and demosaicing tasks on the "Fractal' example. The comparison is made across different denoising priors in Play-and-Play framworks, inclusing Single-Shot deep denoising priors (our proposed prior-*, Noise2Self- UNET, and Noise25olf-FFDNee pre-trained deep denoising priors (Pretrained- UNet and FFett)) and classical denoising priors (TV).
' Image Deconvolution. This is a computational technique aimed at reversing the effects of blur on photographs. Mathematically, the observed image, y, is the result of convoluing the true image, 2, with a forward measurement operator: A(x) = k x. The goal of deconvolution is estimate the original image æ by deconvoluting the observed image y with the gaussian kernel Here, the kernel size was as 15 with standard deviation 5 of Gaussian distribution.
The performance comparison of various Single-Shol deep denoiser algorithms deconvolution task, shown in Table 2, evaluated using both PSNR and SSTM metrics well. The comparison includes our method alongside established techniques like DnCNN, FFDNet, and UNet across other six image categories. Our approach consistently achieves competitive PSNR scores, surpassing others the 'Turtle' and 'Monarch' categories, and still shows parity improvements in SSTM values. This indicates not only enhanced accuracy image reconstruction but also improved perceptual quality. These results underscore our mothod's effectiveness in noise reduction and sharpness, demonstrating its potential for practical deconvolution applications.
In Figure 4, we illustrate qualitative comparison of our deconvolution algorithm four leading Single-Shot deep denoising priors. The visual fidelity of our method is apparent, particularly in its capacily to reconstruct intricate details, such as the giralle's fur that reflected in the zoomed views. This attention to detail extends to the preservation edge sharpness and the subtle gradations, which contribute more natural and cohesive image composition. Contrastingly, other methods exhibit varying degrees of blurring and artifact introduction. This qualitative advancement underscores our algorithm's potential set remarkable benchmark for image deconvolution.
'Giraffe' image against
10

---

Published in Transactions on Machine Learning Research (10/2024)
GT (x**)
Observed (y)
PSNR: 27.24
PSNR: 14.00
Pretrained Bett
Pretrained-FFDDA
Noise2Self- UNet
Noime2eef-PFDNet
PSNR: 16.32
PSNR: 22.05
PSNR: 26.76
PSNR: 20.82
Figure 6: The visualisation comparison of joint deconvolution and demosaicing task on 'Squirrel' example between Single-Shot deep denoising priors (our proposed prior *, Noise2Self UNET and Noise25elf-FFDDNC pre-trained deep denoising priors (Protrained-UNet and Pretrained-FFDNeet), and classical denoising prior (TV) for denoising in Plug-and-Play algorithm.
Pretrained UNet
Pretrained FFDNet
Noise2Self- UNet
Noise2Self FFDNet
200
100
PSNR: -13.24
PSNR: -10.92
PSNR: -5.19
PSNR: -0.49
PSNR: -6.58
Figure 7: The difference the error maps between the proposed Single-Shot deep denoising prior * and the other compared methods Figure on the joint deconvolution and demosaicing task on 'Squirrel' example.
& Joint Demosaicing and Deconvolution. Demosaicing is an algorithmic process that reconstructs a full-colour image from the incomplete colour samples output by image sensor overlaid with colour filter array (CFA).
The joint demosaicing and deconvolutional process can be represented as: A(x) k @ (M e 2), where is the full-colour image, M is the colour filter array (CFA), @ denotes element-wise multiplication. The setting of I was same with kernel in deconvolution task.
Based on performances from previous tasks, we note that FFDNet and UNet produce good results. In the more challenging tasks it is joint demosaicing and deconvolution, these methods, along with classical Total Variation (TV) (Jordan, 1881), are further compared with the data-driven pre-trained denoising priors, and Single-Shot denoising priors training under Noise2Self strategy against our proposed method.
Table 3 illustrates that our method outperforms all methods across all categories, achieving the highest PSNR and SSIM scores in most cases. Notably, the 'Woff and 'Monarch' categories, our method significantly leads, reflecting substantial improvement both reconstruction accuracy and image quality, as indicated by the PSNR and SSIM metries respectively. This demonstrates the effectiveness of our approach handling complex image restoration (asks.
In Figure 5 and 6, our method outperforms other established algorithms in joint deconvolution and demosaicing, particularly in mitigating chromatic aberrations such as red grids or spots. These artifacts, which degrade image quality, are significantly reduced in our approach, leading to a visually coherent result that aligns more closely with the ground truth. Competing methods, including pre-trained networks and classical denoising techniques, often introduce or inadoquately suppress such distortions, resulting in inferior colour and contrasts outcomes. Our method advances the visual quality of image restoration, effectively preserving the natural colour and detail fidelity.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Table The performance (PSNR(dB), SSTM) comparison on joint deconvolution and demosaicing of Single- Shot deep denoising priors *, Noisc25olf-UNE,, and Noisc25elf-FFDNo. pre-trained deep denoising priors Pretrainned-UNe and Preirained-PFDNel), and classical denoising priors (TV) in
following (Lai et al., 2023)
Hiu--and-Play framework.
* denotes proposed denoising prior in the SS-PnP strategy.
Wolf
Beach
Mushroom
Bird
Head
Monarch
PSNR
SSTM
PSNR
SSTM
PSNR
SSTM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Classic
TV
17.27
0.45
16.69
0.48
17.83
0.53
18.72
0.63
17.53
0.58
16.35
0.41
UNet
17.54
0.36
18.02
0.48
18.09
0.49
19.54
0.65
19.12
0.57
17.10
0.41
Pretrain
FFDNet
25.66
0.81
24.06
0.75
25.02
0.82
24.30
0.82
24.65
0.77
21.82
0.63
UNet
30.54
0.90
25.52
0.77
26.86
0.87
27.75
0.87
27.96
0.81
26.31
0.89
SS-PnP
FFDNet
18.45
0.28
16.17
0.25
18.78
0.34
18.15
0.45
18.68
0.29
15.39
0.26
*
30.90
0.91
27.37
0.85
27.00
0.88
28.20
0.89
28.02
0.82
26.48
0.89
We also present visualisation comparing the error maps of our proposed prior with other denoising priors within the Plug-and-Play framework in Figure 7. While there are some noticeable visual differences, the distribution of errors across the image suggests that many discrepancies are not easily detectable by the human eyc. However, our method demonstrates a significant numerical improvement over the alternatives, despite these subtle visual distinctions.
Table The performance (PSNR(dB)) comparison of the im- plicit neural priors on deconvolution (Deconv), super resolution (SR) with 2x and 4x upscalings, and joint doconvolution and demosaicing (Joint) tasks. The * denotes our proposed implicit neural prior, which we compared with the SOTA implicit neural
4.3 Ablation study
We provide a further empirical study for comparing our proposed implicit neural network (INR) with the existing classical INR, SIREN (Strümpler al., 2022).
representation, SIREN.
Example
INR Prior
Deconv
SR (2x)
SR (4x)
Joint
We follow the same experimental settings as described in 4.1, and measure the per- formance based on Peak Signal-to-Noise Ratio (PSNR). In Table 4, our proposed INR outperforms SIREN in all the 4 tasks: deconvolution, super resolution 4x, and joint deconvolution and demo- saicing. This performance is significant in the super resolution with average improvement. of 0.8dB. The spatial com- pactness brought by our INR guarantees the representativeness of different levels of feature that pushes the performance of super resolution tasks.
SIREN
33.05
26.08
22.26
30.83
Wolf
*
33.07
27.21
23.30
30.90
SIREN
30.71
25.72
21.83
28.13
Bird
31.07
26.44
22.06
28.20
and
SIREN
29.82
23.69
19.57
27.79
Head
*
29.92
24.24
20.61
28.02
Table 5:
The performance (PSNR(dB)) comparison of BM3D (Dabov el al., 2007), DIP (Sun el. al., 2021b) and our proposed * priors on resouuttnolution (SR) task with 2x and 4x upscalings in the Plug-and- framework.
We also performed comparison with other classical implicit neural representa- tion (INR) methods, such as WIRE (Sara- gadam et. al., 2023). However, WIRE showed inconsistent performance and lacked stability across the various tasks. In the Bird example, our method demon- strated superior PSNR results compared
PnP
Fractal
Tiger
Bird
Framcwork
2x
4x
2x
4x
2x
4x
BM3D
12.62
12.35
15.59
14.66
11.89
11.46
DIP
22.92
18.51
22.14
18.34
23.79
20.49
*
25.43
21.26
23.12
19.45
26.44
22.06
12

---

Published in Transactions on Machine Learning Research (10/2024)
to WIRE, with 26.44dB compared to 24.88dB for 2x super-resolutio, 22.06dB compared to 20.59dB for 4x super-resolution, and 28. 20dB compared to 27.83dB for the joint deconvolution and demosaicing task.
Deconv
SR (2x)
SR (4x)
Joint
Observed (*)
We can observe from Figure 8 that our INR excels in reconstructing the image with smoother edge features and enhanced colour representation. The difference is more clearly reflected in the error maps, with noticeable variation around the edge features across all four tasks. The performance over iteration curves demonstrate our leading advances over the optimi- sation iterations, meanwhile show the carly iteration steps is effective for achieving the best. performance. This is commonly observed in iterative optimisation techniques (Wei et al., 2020; 2022), where initial iterations tend to improve the quality of the recon- struction, while beyond a certain point, the model begin to introduce artifacts or oversmooth the image, which leads a degradation in performance and consequently drop in PSNR.
Diff
SIREN
PSNR (dB)
GT
(x*)
We also conducted comparative analysis of our proposed Single-Shot prior against other state-of- the-art priors within Blug-and-Plas frameworks, as
shown in Figure 5. The BM3D (Dabov et al., 2007) and DIP (Sun al., 2021b) priors were each run for 240 iterations- ten times the iterations used for our Singlo-Shot approach. Despite this, the classic BM3D method's performance markedly lower compared to both DIP and our method. While the DIP prior within the Plug-and-Play framework demonstrated results closer to ours, a significant performance gap remains, underscoring the superior efficacy of our approach in both 2x and 4x super-resolution tasks.
Figure 8: The visualisation comparison of the implicit neural priors (Ours * and STREN) deconvolution (Deconv), super resolution(SR) with 2x and 4x upscal- ings, and joint deconvolution and demosaicing (Joint) tasks on "Bird" example. The difference of the error maps of * and SIREN are provided in the third row. The plot indicates the performance (PSNR) with both implicit neural priors changing over the ADMM ilera- tion on the joint deconvolution and demosaicing task.
5 Conclusion
Our work pioneers the Single-Shot Plug-and-Play (SS-PnP) method, transforming the use of PnP priors in inverse problem-solving by reducing reliance on large-scale pre-training of denoisers, and using a single instance. We also propose Single-Shot proximal denoisers via implicit neural priors allowing for superior approximation quality for solving inverse problems using only a single instance. We propose a novel function for implicit neural priors that has desirable theoretical guarantees. We also showed that our work generalises well across different tasks, showing empirical stability for single and multiple operators. We then open the door to new research line for Single-Shot PnP. Whilst this work uses mainly PnP-ADMM due to the well-know properties, and performance in comparison with other algorithms. Future work will include to evaluate our new rescarch line on Single-Shot PnP on different algorithms including but not limited to FISTA (Beck & Teboulle, 2009a). HQS (Geman & Yang, 1995), and Primal-dual ( (Dantzig el al., 1956) algorithiss. Another additional side insight could be to explore the result over randomising 20 in step 2 of Algorithm Though the primary focus of this work not the convergence of the proposed implicit neural prior, the convergence of the broader PnP framework is significant and intricate topic that is valuable to analysis in the future research work.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Acknowledgements
This project supported with funding from the Cambridge Centre for Data-Driven Discovery and Accelerate Programme for Scientific Discovery, made possible by a donation from Schmidt Futures. YC is funded by an AstraZeneca studentship and a Google studentship. The work of RHC was partially supported by HKRGC GRF grants CityU1130992,, CRF grant C1013-21GF and HKITF MHKJFS Grant MHP MHP/005 /054/22. CBS acknowledges support from the Philip Leverhulme Prize, the Royal Society Wolfson Fellowship, the EPSRC advanced career fellowship EP/ RPP/V22942//LL 1,, EPSRC grants EP EP/S0200000 /S026045/ and /PTT033553/////// EP/NO14588/ 1,, PP/TO17961/1 the Wellcome Innovator Awards 215733/ /9//3//////// and 221633 ZZ/20/Z, the European Union Horizon 2020 research and innovation programme under the Marie Skodowska-Curi grant agreement No. 777826 NoMADS, the Cantab Capital Institute for the Mathematics of Information and the Alan Turing Institute. ATAR acknowledges support from CMIH (EP TEP/TO17961, /1) and CCIMI, University of Cambridge. This work was supported in part by Oracle Cloud credits and related resources provided by Oracle for Research. Also, EPSRC Digital Core Capability.
References
Rizwan Ahmad, Charles A Bouman, Gregery T Buzzard, Stanley Chan, Sizhuo Liu, Edward Reehorst, and Philip Schniter. Plug-and-play methods for magnetic resonance imaging: Using denoisers for image rocovery. IEEE signal processing magazine, 37(1):105-116, 2020. 1
Ahmet Oguz Akyüz et al. Deep joint deinterlacing and denoising for single shot dual-iso hdr reconstruction. IEEE Transactions on Image Processing, 29:7511-7524, 2020. 3
Saced Anwar and Nick Barnes. Densely residual laplacian super-resolution IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1192 -2144 2020. 3
Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb. Solving inverse problems using data-driven models. Acta Numerica, 28:1-174, 2019. 2
Benjamin Attal, Eliot Laidlaw, Aaron Gokaslan, Changil Kim, Christian Richardt, James Tompkin, and Matthow O' Törf: Timo-of-flihh radiance fields for dynamic scene view synthesis. Advances in neural information processing systems, $4:26289-26333 2021. 3
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. International Conference on Machine Learning, pp. 324-533. PMLR, 2019.
A Beck and M Teboulle. A fast iterative shrinkage-thresoolding algorithm with application to wavelet-based image deblurring. In IEEE, 2009a. 13
Amir Bock and Mare Teboulle. A fast iterative shrinkage-thresholling algorithm for linear inverse problems. SIAM journal on imaging sciences, 2(1):183 202, 2009b.
Mario Bertero, Patrizia Boccacci, and Christine De Mol. Introduction to inverse problems in imaging. 2021.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Maric Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. 2012.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and TrendsB in Machine learning, 311):111 122, 2011.
Antoni Buades, Bartomeu Coll, and J-M Morel. A non-local algorithm for image denoising. In 2005 IEEE computer society conference on computer vision and pattern recognation (CVPR 055, volume 2, pp. 60-65. Ieee, 2005. 3
Antonin Chambolle and Thomas Pock. first-order primal-dual algorithm for convex problems with applications to imaging. Journal of mathematical imaging and vision, 40:120-145, 2011. 4
14

---

Published in Transactions on Machine Learning Research (10/2024)
Stanley H. Chan, Xiran Wang, and Omar A Elgendy. Plug-and-play admm for image restoration: Fixed-point convergence and applications. IEEE Transactions on Compulational Imaging, 3(1 3(1))84-98) 99, 2016. 1
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628 8638, 2021.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d dransoorm-domain collaborative filtering, IEEE Transactions image processing, 16(8):2080-2085) 2007.
3, 12, 13
G. B. Dantzig, Jr Ford, L. R., and D. R. Fulkerson. A primal-dual algorithm. A Primal Dual Algorithm, 1956. 13
Anthony J Devancy. Mathematical foundations of imaging, tomography and wavefield inversion. Cambridge University Press, 2012.
Geman and Yang. Nonlinear image recovery with half-quadratic regularivation. IEEE brunsactions on image processing publication of the IEEE Signal Processing Society, 1995. 13
Samuel Hurault, Arthur Leclaire, and Nicolas Papadakis. Proximal denoiser for convergent. plug-and-play optimization with nonconvex regularization. In International Conference on Machine Learning, pp. 9483- 9505. PMLR, 2022. 2, 5
Camille Jordan. Sur la series de fourier. CR Acad. Sci., Paris, 92:228- 230, 1881.
Alexander Krull, Tim-Oliver Buchholz, and Florian Jug. Noise2void denoising from single noisy images. In Proceedings of the IEFE/CVF conference on computer vision and paltern. recognilion, pp- 2129 2137, 2019. 3,
Zeqiang Lai, Kaixuan Wei, Ying Fu, Philipp Härtel, and Felix Heide. -prox: Differentiable proximal algorithm modeling for large-scale optimization. ACM Transactions on Graphics (TOG), 42(4):1-19, 2023. 8, 12
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. Advances in Neural Information Processing Systems, 32, 2019.
Rémi Laumont, Valentin De Bortoli, Andrés Almansa, Julie Delon, Alain Durmus, and Marcelo Pereyra. Bayesian imaging using plug & play priors: when langevin meets tweedie. SIAM Journal on Imaging Sciences, 15(2):701 737, 2022.
Victor Lempitsky, Andrea Vedaldi, and Dmitry Ulyanov. Deep image prior. In 2018 IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 3446-9454 IEEE, 2018. 3
Michael McCann, Kyong Hwan Jin, and Michael Unser. Convolutional neural networks for inverse problems in imaging: A review. IEEE Signal Processing Magazine, 34(6):85 95, 2017.
Tim Meinhardt, Michael Moller, Caner Hazirbas, and Daniel Cremers. Learning proximal operators: Using denoising networks for regularizing inverse imaging problems. In Proceedings of the IEEE International Conference on Compuler Vision, pp. 7781-1790, 2017. 2,3
Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American statistical association, 44(247):335 341, 1949.
Shunsuke Ono. Primal-dual plug-and-play image restoration. IEFE Signal Processing Letters, 2408):1108-1100
2017. 1, 2
Yuhui Quan, Mingqin Chen, Tongyao Pang, and Hui Ji. Sclf2self with dropout: Learning self-supervised denoising from single image. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 1890-1898 2020.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 10(4):1804-1844, 2017. 3
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assister International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234-241. Springer, 2015. 6,
Citrrunntion-MMICO. 2015: 18th
Ernest Ryu, Jialin Liu, Sicheng Wang, Xiaohan Chen, Zhangyang Wang, and Wotao Yin. Plug-and-play methods provably converge with properly trained denoisers. In International Conference on Machine Learning, pp. 5546-556600 5557. PMLR, 2019. 2, 5, 6
Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18507 18516, 2023. 2, 3, 4, 12
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in newral information processing systems, 33: 7462 7473, 2020. 3
Yannick Strümpler, Janis Postels, Ren Yang, Lue Van Gool, and Federico Tombari. Implicit neural represen- tations for image compression. In European Conference on Compuler Vision, pp. 74-91. Springer, 2022. 2, 12
Yu Sun, Brendt Wohlberg, and Ulugbek S Kamilov. An online plug-and play algorithm for regularized image reconstruction. IEEE Transactions on Computational Imaging, 5(3):395-40) 2019. 2, 5
Yu Sun, Jiaming Liu, Mingyang Xie, Brendt Wohlberg, and Ulugbek S Kamilov. Coil: Coordinate-based internal learning for imaging inverse problems. arXiv preprint arXiv:2102.055: 051881, 2021a. 3
Zhaodong Sun, Fabian Latorre, Thomas Sanchez, and Volkan Cevher. A plug-and deep image prior. ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 8103 8107. IEEE, 2021b. 12, 13
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537- -74777
2020. 2
Afonso M Teodoro, José M Bioucas-Dias and Mário AT Figueiredo. Image restoration and reconstruction using variable splitting and class-adapted image priors. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 3518-3522. IEFE, 2016. 3
Afonso M Teodoro, José M Bioucas-Dias, and Mário AT Figuciredo. A convergent image fusion algorithm using scene-adapted gaussian- 451-463, 2018. 2
gaussian-mixture mixture-based denoising. IEEE Transactions Image Processing, 28(1):
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference signal and information processing, pp. IEEE, 2013. 2, 6
Curtis R Vogel. Computational methods for inverse problems. STAM, 2002. 1
Jiachuan Wang, Shimin Di, Lei Chen, and Charles Wang Wai Ng. Noise2info: Noisy image to information of noise for self-supervised image denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16034 16043, 2023.
Yuchao Wang, Yonghao Long, Siu Hin Fan, and Qi Dou. Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 431-441. Springer, 2022.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Kaixuan Wei, Angelica Aviles-Rivero, Jingwei Liang, Ying Fu, Carola-Bibiana Schönlieb, and Hua Huang. Tuning-free plug-and-play proximal algorithm for inverse imaging problems. In International Conference on Machine Learning, pp. 10158 10169. PMLR, 2020. 3, 13
Kaixuan Wei, Angelica Aviles Rivero, Jingwei Liang, Ying Fu, Hua Huang, and Carola-Bibiane Schönlieb. Tfpnp: Tuning-free plug-and-play proximal algorithms with applications to inverse imaging problems. The Journal of Machine Learning Research, 23(1):699 746, 2022. 6, 13
Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2same: Optimizing self-supervised bound for image denoising. Advances in neural informalion processing systems, 33:20320- 20330,, 2020. 3
Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai. Plug-and--laay algorithms for large-scale snapshot compressive imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1447-1457, 2020.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuai Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In IEEE/CVP conference on computer vision and pottern recognition, pp. 2996-2705, 2020.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparss-represenaaaionn. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pp. 711 730. Springer, 2012. 7
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep cun for image denoising. IEEE transactions image processing, 26(7):3142 3155, 2017a.
3, 7
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. Learning deep cmn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognilion, pp- 3929 3938, 2017b. 2, 3
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Fidnet: Toward a fast and flexible solution for enn-based image denoising. IEEE Transactions on Image Processing, 27(9):4608-498 4620, 2018. 3,
Kai Zhang, Wangmeng Zuuo, and Lei Zhang, Deep plug-and-play super-resolution for arbitrary blur kernels. Proceedings of the IEEE/CC /CFF Conference on Computer Vision and Pattern Recognition, pp. 1671-1681, 2019.
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-aud-play image restoration with deep denoiser prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (10):6360-6376, 2021.
17


Reviews:
[{"review_id": "j5hZLPcvzX", "reviewer": "Reviewer_foFv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To the best of my knowledge, this is the first paper to consider the use of IRL denoisers within a plug-and-play framework for image restoration. This is an original research direction that will certainly interest many TMLR readers and stimulate further research on the topic. The submission is clear, well organised, and easy to read. The figures and tables are also easy to read, are correctly referenced in the paper and have clear captions.\n\nWith regards to weaknesses, I would point out that there are many other plug-and-play methods available that do not require any form of training data, including many of the first plug-and-play methods which relied on patch-based image denoisers such as BM3D or non-local Bayes. Moreover, there are also several plug-and-play methods that rely on denoisers represented by Gaussian mixture models on the image patches; these denoisers can be trained directly from a single noisy image or alternatively with a small dataset of images. Furthermore, one can also consider other small architectures (see e.g., 10.1109/TCI.2023.3306100) that can be trained with a reduced number of images and in addition guarantee convexity as well as other key mathematical properties. Similarly, while this is the first paper to use IRL denoisers within a plug-and-play framework, the use of IRL architectures for image restoration is studied in detail in arXiv:2304.10250. Lastly, it is not clear to me that (6)-(9) guarantee the convergence of the iterates of Algorithm 1. Establishing the convergence of the iterates of plug-and-play algorithms is highly non-trivial and I do not believe that this has been properly addressed in the submission.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission presents a new plug-and-play methodology for solving high-dimensional inverse problems, with a focus on image restoration tasks and on applications with limited training data available in mind. Most modern plug-and-play methods rely on denoisers encoded by neural networks, which require abundant data for training. The main novelty of this paper is to consider implicit neural representation (INR) denoisers, which require significantly less data than standard approaches (e.g., DnCNN and U-Net architectures). The proposed approach is demonstrated empirically with a range of experiments related to image super-resolution, deblurring, and demosaicing with limited training data available, where the proposed method outperforms alternative approaches based on DnCNN and U-Net architectures, the learning-free strategy total-variation regularisation, as well as the alternative IRL strategy SIREN architecture.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In my opinion, the following critical changes are required before this submission can be considered for publication:\n1) While this is the first submission to consider IRL denoisers within a plug-and-play framework, it is certainly not the first to consider plug-and-play methodology with denoisers that require little or no training. It is therefore important to remove all statements that suggest that this is the first paper to consider, propose, or introduce \"single-shot\" plug-and-play methodology or plug-and-play methodology with single-shot denoisers. \n2) The introduction needs to be significantly revised to present a more complete description of the literature and place the contribution of this submission within the context of prior research. As I pointed out previously, there is a large literature related to plug-and-play image restoration with patch-based image denoisers that can be implemented without any training data available. In addition, some approaches such as the ones leveraging Gaussian mixture models on image patches can perform well in situations where there is a very small amount of data available. These should be discussed in the introduction.\n3) The numerical experiments should be revised to include comparisons with 1) a plug-and-play image restoration method implemented with the BM3D denoiser; 2) a plug-and-play image restoration method implemented with the non-local Bayes denoiser; 3) a trained Gaussian mixture model on a patch representation as proposed in Teodoro et al. (10.1109/TIP.2018.2869727); 4) a trained convex ridge regulariser (10.1109/TCI.2023.3306100); and 5) the alternative IRL strategy proposed in arXiv:2304.10250. The relative advantages and drawbacks of these approaches relative to the proposed approach should be discussed in detail.\n4) The statements following (6) regarding the convergence of the scheme should be significantly revised. The current submission does not establish the convergence of Algorithm 1. Instead, I would recommend mentioning the analysis of the convergence of Algorithm 1 as an important perspective for future work. \n\nMoreover, I also recommend revising the introduction to mention stochastic plug-and-play Bayesian imaging methods (see for example https://doi.org/10.1137/21M140634) and providing some pointers to this literature.", "broader_impact_concerns": "I have no concerts about the ethical implications of this submission, other than the fact that some of the claims regarding prior research are inaccurate. The changes that I request above address these concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gY7nVtc9by", "reviewer": "Reviewer_dodD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper adequately introduces the current literature on both INR and PnP priors. \n2. The numerical evaluation is clear with different examples.\n3. The paper is clearly written with each section well defined. It is easy to navigate.\n\nWeakness:\n1. There are several problems in the methodological part.\n    1.1 What happens if dim(y) > dim(x) ? How can the denoiser for y be applied to denoise z - x in the PnP part if the observation lives in a different space than the observation? \n    1.2 The activation function presented resembles closely the activation function called WIRE (from the paper cited in this work, namely Saragadam et al. (2023)). Indeed, it seems to me it corresponds to the complex part of the WIRE activation $+(exp(-a_1x + b_1)) + 1)^{-1}$.\nAm I mistaken? If not, why does it does not appear on the paper somewhere? \n    1.3 The question \"Why are these properties interesting\" is not really answered. Why is differentiability and continuability more representative? Why is it important to have the asymptotic behavior proposed?  Does it make a difference? does the term  $+(exp(-a_1x + b_1)) + 1)^{-1}$ actually bring something numerically rather than theoretically (ok, the representation theorem for neural networks needs this kind of hypothesis, but if this is your motivation, it should be mentioned).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper contributions can be divided into two main contributions:\n1. It introduces a new activation function for Implicit Neural representation (INR), with an analysis of some of its properties such as derivability and asymptotic behavior.\n2. It shows how INR can be used for PnP inverse problem solving by first adding a single shot denoiser learning part based only on the observed variable.\n\nIt then evaluates the performance on several imaging tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Comparison with competitors.\n    1.1 First of all, I think a comparison with the WIRE activation would greatly enhance the paper. Indeed, is the term $+(exp(-a_1x + b_1)) + 1)^{-1}$ actually worth adding?\n    1.2 There are other methods that do not need self-supervised training or pretraining, such as [1]. Comparison to such methods would greatly enhance the ability of the community to address the proposed method.\n\n2. Confidence intervals would be greatly appreciated in all experiments.\n\n[1] Sun, Zhaodong, et al. \"A plug-and-play deep image prior.\" ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xZ7J76hS0m", "reviewer": "Reviewer_k4Bh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is mostly well-written and presentation is clear. \n\n* The concept of Single-Shot PnP is both novel and interesting. \n\n**Weaknesses**\n\n*  Some of the claims are concerning. For example, the statement: \"With only one image input in the whole reconstruction process, it outperforms the classical methods and pre-trained models across all tasks,\" seems to be too broad. This would only makes sense if the authors have compared against tradition PnP methods such as  DPIR and provided better performance (More details are included in the requested changes).\n\n* While Sections 1-2-3 are written clearly and presented well, some important details regarding the experimental results are missing. (More is included in the requested changes) \n\n* The experiments are incomplete in fully demonstrating  the performance  of the proposed methods. For instance, a direct comparison is needed to evaluate the performance against traditional PnP (with pretrained deep denoisers trained in a supervised manner).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper introduces Single-Shot PnP (SS-PnP), a novel method for solving inverse problems. Unlike traditional approaches, SS-PnP does not rely on supervised training of deep denoisers as PnP priors. Instead, it is trained on a single image during the testing phase.\n\n2. The authors propose an activation function for the implicit neural prior, which is integrated into the PnP-ADMM. They also provide a compact theoretical justification for the advantages of this function, such as the differentiability of the network and continuity.\n\n3. Empirical results are presented to demonstrate the effectiveness of the proposed method on several image restoration problems.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "***Major Points***\n\n1. Clarification of Experimental Results:\n\n     * Could the authors please clarify what is meant by the \"pretrained Unet\"? Is the Unet trained in a supervised manner and then used in PnP-ADMM? If so, could the authors include details regarding the training dataset? If these models are trained using only a single image, could this be explicitly mentioned?\n\n    * If the pretrained Unet and FFDNet are not trained deep denoisers, the authors should include a comparison with supervised training of the PnP priors. While this reviewer does not expect that a self-supervised, single-shot prior would perform equally or better than a supervised deep prior, it would be informative to see the performance gap.\n\n\n   * On a related note, the numbers reported in Table 3 seem unusual. It is surprising that a supervised pretrained Unet and FFDNet perform worse than the single-instance training. This would only make sense if those networks were trained with only one image, which would create an unfair comparison due to the scale of the Unet and FFDNet networks  compared to the limited data available for training.\n\n\n2. Discussion on PSNR Decrease:\n    * Could the authors include a discussion on why there is a decrease in PSNR in Figure 7? Understanding this drop in performance would be valuable.\n\n3. Visual Quality Claims:\n    * The claims regarding visual quality made throughout the paper are not always accurate. For example, in Figure 7, the differences between SIREN and the proposed method are not distinguishable to the human eye. Similarly, in Figure 6, Noise2Self-Unet and the proposed method are difficult to differentiate visually. The authors should either revise these claims or include error maps that could provide more informative comparisons regarding visual quality.\n\n4. Clarification of Equation (2):\n    * What happened to the parameter  $\\lambda $ in Equation (2)? Could the authors provide the relationship between \n$\\lambda $, $\\sigma_k$, and $\\mu_k$\t\n ? Alternatively, the authors could simplify the equation by dissolving $\\lambda$ into $R$ and directly providing the relationship between   $\\sigma_k$ and $\\mu_k$. \n\n\n***Minor Points***\n1. Typos such as in the first paragraph of Section 4.1.\n\n2. Please include some information regarding the dynamic noise strength parameter setting. \n\n3. The authors should include a discussion regarding the effect of noise on step 1 of the algorithm and training the prior. \n\n4. The last paragraph on page 10, starting with \"Based on performances...\", is unclear. The phrase \"under data-driven pre-trained and Noise2Self pre-training conditions against our proposed method\" needs clarification. Please rewrite this paragraph for clarity.", "broader_impact_concerns": "Can the authors include the limitation of the proposed method?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The development of new and efficient denoisers that require limited training data is of significant interest in the field of inverse problems. As such, this paper aligns well with the scope of TMLR.
Decision: Accept (poster)