Early Stopping for Deep Image Prior
Authors: Hengkang Wang
Taihui Li
Zhong Zhuang
Tiancong Chen
Hengyue Liang
Ju Sun
Venue: Accepted by TMLR ()

Abstract: Deep image prior (DIP) and its variants have shown remarkable potential to solve inverse problems in computational imaging (CI), needing no separate training data. Practical DIP models are often substantially overparameterized. During the learning process, these models first learn the desired visual content and then pick up potential modeling and observational noise, i.e., performing early learning then overfitting. Thus, the practicality of DIP hinges on early stopping (ES) that can capture the transition period. In this regard, most previous DIP works for CI tasks only demonstrate the potential of the models, reporting the peak performance against the ground truth but providing no clue about how to operationally obtain near-peak performance without access to the ground truth. In this paper, we set to break this practicality barrier of DIP, and propose an effective ES strategy that consistently detects near-peak performance across several CI tasks and DIP variants. Simply based on the running variance of DIP intermediate reconstructions, our ES method not only outpaces the existing ones---which only work in very narrow regimes, but also remains effective when combined with methods that try to mitigate overfitting.

Published in Transactions on Machine Learning Research (12/2023)
Early Stopping for Deep Image Prior
Hongkang Wang
wong9881 Gumm edu
University of Minnesota, Twin Cifies
Taihui Li
lics5027@ummn edu
University of Minnesota, Twin Cities
Zhong Zhuang
2huam1430vmmn edu
University of Minnesota, Twin Cities
Tiancong Chen
chen6271 @umn. edu
University of Minnesota, Twin Cilies
Hongyuc Liang
Gung656@uumrr edu
University of Minnesota, Twin Cities
Ju Sun
jusun Besun.ummnccm edu
University of Minnesota, Twin Cifies
Reviewed OpenReview: https: https://openreviel/ openreview. net/ forum? id= 23122rLC8X
Abstract
Deep image prior (DIP) and its variants have shown remarkable potential to solve inverse problems in computational imaging, needing no separate training data. Practical DIP models are often substantially overparameterized. During the learning process, these models first learn the desired visual content and then pick up potential modeling and observational noise, i.e. performing early learning then overfitting (ELTO). Thus, the practicality of DIP hinges on early stopping (ES) that can capture the transition period. In this regard, most previous DIP works for computational imaging tasks only demonstrate the potential of the models, reporting peak performance against ground truth, but providing no clue about how to operationally obtain near-peak performance without access to ground truth. In this paper, we set out to break this practicality barrier of DIP and propose an effective ES strategy that consistently detects near-pes peak performance in various computational imaging tasks and DIP variants. Simply based on the running variance DIP intermodiate roconstructions, our ES method not only outpaces the existing ones which only work in very narrow regimes, but also remains effective when combined with methods that try to mitigate overfitting. The code to reproduce our experimental results is available at httts:://s/t thtbb/gtttuuu...//// thub. com/ st//suuuumn umn/ Sarlyy_Stopinggfor.
1 Introduction
Inverse problems (IPs) are prevalent in computational imaging, ranging from basic image denoising, super- resolution, and deblurring, to advanced 3D reconstruction and major tasks in scientific and medical imag- ing (Szeliski 2022). Despite the disparate scttings, all these problems take the form of recovering a visual object x from = f(x), where I models the forward physical process to obtain the observation y. Typically, these visual IPs are not determined a unique way: æ cannot be determined uniquely from y. This ex- accrbated by potential modcling (c.g., linear to approximate a nonlincar process) and observational (c.g.. Gaussian or shot) noise, i.c., y == (x)) To overcome nomuniqeeness and improve noise stability, rescarchers often encode variety of problem-speeifie priors on æ when formulating IPs.

---

Published in Transactions on Machine Learning Research (12/2023)
Traditionally, IPs are phrased as regularized data fitting problems:
min f())( + XB(x) fly, f())) data-fitting loss, R(x): regularizer
(1)
where À is the regularization parameter. Here, the loss l is often chosen according to the noise model, and the regularizer R encodes priors on æ. The advent of deep learning has revolutionized the way IPss are solved. On the radical side, deop neural networks (DNNs) are trained to directly map any given y to an æ; on the mild side, pre-trained or trainable deep learning models are taken to replace certain nonlinear mappings in iterative numerical algorithms for solving EFq. (1) (e.g. plug-and-play and algorithm unrolling); see recent surveys Ongie al. (2020); Janai al. (2020) on these developments. All of these deep-learning based methods rely large training sets to adequately represent the underlying priors and/or noise distributions. This paper concerns another family of striking ideas that do not require separate training data.
Deep image prior (DIP) Ulyanov et al. (2018) proposes parameterizing x as a = Gø(z), where Ga is a trainable DNN parameterized by and x is a frozen or trainable random seed. No separate training data other than y are used! Plugging the reparametrization into Eq. (1) we obtain
min Kly, I Go(z)) + AR o Go(z).
(2)
0
Go is often "overparameterized" containing substantially more parameters than the size of x, and "structured"-ea" consisting of convolution networks to encode structural priors in natural visual objects. The resulting optimization problem is solved using standard first-order methods (e-g., (adaptive) gradient descent). When æ has multiple components with different physical meanings, can naturally parametrize a using multiple DNNs. This simple idea has led to surprisingly competitive results in numerous visual IPs, from low-level image denoising, super-resolution, inpainting (Ulyanov et al. 2018 Heckel & Hand 2019 Liu
let al.
2019) and blind deconvolution (Ren et al.) 2020 Wang et al. 2019 Asim et al. 2020) Tran et al.
2021
Zhuang et al. 2022a),, to mid-level image decomposition and fusion (Gandelsman et al. 2019 Ma
et al.
2021), and to advanced computational imaging problems (Darestani & Heecel 2021 Hand et al.
2018
Williams et al. 2019 Yoo et al. 2021
Baguer et al., 2020 Cascarano et al. 2021 Hashimoto & Ote,
2021
Gong et al. 2022
Veen et al. 2018
Tayal et al. 2021
Zhuang ct al. 2022b the survey Qayyum
et al.
2021
Overfitting issue in DIP A critical detail that we have glossed over is overfitting. Since Go is of- ten substantially overparameterized, Gz(z) can rep- resent. arbitrary elements in the æ domain. Global optimization of equation 2 would normally lead to
26
24
PSNR
22
20
PSNR
y fo Gz(z), but Gz(z) may not reproduce x,
18
e.g., when f is non-injective, or y f(xx) so that Go(z) also accounts for the modeling and obser- vational noise. Fortunately, DIP models and first- order optimization methods together offer a bless- ing: in practice, Go(z) has a bias towards the de- sired visual content and learns it much faster than learning noise. Therefore, the quality of reconstruc- tion climbs to peak before the potential degrada- tion due to noise; see Fig. 1 This "carly-learning- then-overfiuiing (ELTO) phenomenon has been re- peatedly reported in previous work and is also sup- ported by theories on simple Go and lincar f (Heckel & Soltanolkotabi] 2020b(a). The successes of the DIP models claimed above are on the premise that appropriate early stopping (ES) around performance peaks can be made.
16
14
12-
10-
500
1000
1500 2000 2500 Iteration
Figure 1: The "early-learning- thenrroiintteee--ovriiitinn (ELTO) phenomenon in DIP for image denoising. The quality of the estimated image climbs first to a peak and then drops once the noise is picked up by the model Go(z) also.
2

---

Published in Transactions on Machine Learning Research (12/2023)
Is ES for DIP trivial? Natural ideas trying to perform ES can fail quickly. (1) Visual inspection: This subjective approach is fine for small-scale tasks involving few problem instances, but quickly becomes infeasible for many scenarios, such as (a) large-scale batch processing, (b) recovery of visual contents tricky to visualize and/or examine by eyes (e.g. 3D or 4D visual objects), and (c) scientific imaging of unfamiliar
objects (e.g., MRI imaging of rare tumors and microscopic imaging of new virus species); (2) Tracking
full-reforence/mee noorrferece image quality metrics (FR/NR-IQMSs) or fitting loss: Without the ground truth æ; computing any FR-IQM and thereby tracking its trajectory (e.g., the PNSR curve in Fig. 1 is out of the question. We consider the tracking of NR-IQMs the performance is much worse than ours. We also explore the possibility of using the loss curve for ES here, but are unable to find correlations between the trend of the loss and that of the PSNR curve, shown in Fig. 15 (3) Tuning the iteration number: This ad-hoc solution is taken in most previous work. But since the peak iterations of DIP vary considerably across images and tasks (see, e.g., Figs. 4 and 29 and Appendiees A.7.3 and A.775)), this could entail numerous trial-and-crron steps and lead to suboptimal stopping points; (4) Validation-based ES: ES easily reminds us of validation-based ES in supervised learning. The DIP approach to IPs, summarized in Eg. (2) does not belong to supervised learning, as it only deals with a single instance y, without separate (æ, y) pairs as training data. There are recent ideas (Yaman et al.) 2021 Ding et al.) 2022) that hold part of the observation y out as a validation set to emulate validation-based ES in supervised learning, but they quickly become problematic for nonlinear IPs due to the significant violation of the underlying i.idd. assumption; see Sec. 3.4
a family of baseline methods in [Sec. 3.1,
Prior work addressing the overfitting There are three main approaches to counteracting the overfitting of DIP models. (1) Regularization: Heckel & Hand (2019) mitigates overfitting by restricting the size of
Go to the underparamet riiraaattrizaiion rization regime. Metzler et al.
2018 Shi et al. (2022) Jo et al. (2021); Cheng et al. (2019) control the network capacity by regularizing the layer-wise ise weights or the network Jacobian. Liu et al.
(2019); Mataev et al.] (2019); Sun (2020); Cascarano et al.] (2021) use additional regularizer(s) R(Ga
such as the total-variation norm or trained denoisers. These methods require the right regularization level which depends on the noise type and level to avoid overfitting; with an improper regularization level, they can still lead to overfitting (see Fig. 4 and Sec. 3.1. Moreover, when they do succeed, the performance peak is postponed to the last iterations, often increasing the computational cost by several folds. (2) Noise modeling: You et al. (2020) models sparse additive noise as an explicit term in their optimization objective, Jo et al. [021) designs regularizors and ES criteria specific to Gaussian and shot noise. Ding et al. (2021) explores subgradient methods with diminishing step size schedules for impulse noise with the & loss, with preliminary success. These methods do not work beyond the types and levels of noise they target, whereas our knowledge of the noise in a given visual IP is typically limited. (3) Early stopping (ES): [Shi et al.)
(2022) tracks progress based on ratio of no-reference blurriness and sharpness, but the criterion only works for their modified DIP models, as acknowledged by the authors. Jo et. al. (2021) provides noise-speeific regularizer and ES criterion, but it is not clear how to extend the method to unknown types and levels of noise. Li et al. (2021) proposes monitoring DIP reconstruction by training a coupled autoencoder. Although its performance is similar to ours, the extra autoencoder training slows down the whole process dramatically; see Sec. 3. Yaman et al. (2021): Ding et al.] (2022) emulate validation-based ES in supervised learning by splitting elements of y into "training" and "validation" sets so that validation-based ES can be performed. But in IPs, especially nonlinear ones (e.g., in blind image deblurring (BID), y v k x where is the linear convolution), elements of y can be far from being i.i.d., and so validation may not work well. Moreover, holding out part of the observation in y can substantially reduce the peak performance; see Sec. 3.4
Our contribution We advocate the ES approach the iteration process stops once a good ES point is detected, as (1) the regularization and noise modeling approaches, even if effective, often do not improve peak performance but push it until the last iterations; there could be 2 10x more iterations spent than climbing to the peak in the original DIP models; (2) both need deep knowledge about the noise type/level, which is practically unknown for most applications. If their key models and hyperparameters are not set appropriately, overfitting probably remains, and ES is still needed. In this paper, we build a novel ES critcrion for various DIP models simply by monitoring the trend of the running variance of the reconstruction sequence. Our ES method is (1) Effective: The gap between our detected and the peak performance, i.e. the detection gap, is typically very small, as measured by standard visual quality metrics
3

---

Published in Transactions on Machine Learning Research (12/2023)
Table 1: Summary of performance of our DIP+ES-WMI and competing methods on image denoising and blind image deblurring (BID). V: working reasonably well (PSNR 2 2dB less of the original DIP peak); -: not working well (PSNR & 2dB less of the original DIP peak): N/A: not applicable (i.e., we do not perform comparison due to certain reasons). Note that DF-STE, DOP, and SB are based modified DIP models.
Image denoising
BID
Gaussian
Impulse
Speckle
Shot
Real world
Low
High
Low
High
Low
High
Low
High
Low
High
DIP+ES- ES-WWV (Ours) DIR+NR-
N/A
N/A
DIP+SV-ES
V
V
V
V
V
N/A
N/A
+IP+
V
V
V
V
V
DF-STE
V
V
N/A
N/A
N/A
N/A
V
N/A
N/A
DOP
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
SB
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
(PSNR and SSIM). Our method works well for DIP and its variants, including sinusoidal representation networks (Sitzmann et al., 2020, SIREN) and deep decoder (Heckel & Hand 2019). on different noisy types/levels and in 5 visual IPs, including both lincar and non-linear ones. Furthermore, our method help several regularization-bue based methods, e.g., Gaussian process-DIP (Cheng al 2019 GP-DIP), DIP with total variation regularization (Liuu et al. 2019 Cascarano et al. 2021 DIP-TV) to perform reasonable ES when they fail to prevent overfitting; (2) Efficient: The per-iter inr-iteration overhead is a fraction for the standard version in Algorithm 1 or negligible for the variant in Algorithm 2] relative to the per-iteration cost of Eqq. (2) (3) Robust: Our method is relatively insensitive to the two hyperparameters, i.e. window size and patience number. We keep the same hyperparameters for all experiments [Secs. 2 and 3 except for the ablation study. In contrast, the hyperparameters of most of the methods reviewed above are sensitive to the noise type/level. We summarize the performance of our DIP+ES method against competing methods
for image denoising and BID in Tab. we present the detailed results in Sec. 3
Remarks on diffusion models for IPs Recently, diffusion-based models (DBMs) have shown great promise
0.08
-0.065
PSNR
24
MSE
0.07
0.060
in solving linear IPs Wang et al. 2022 ; Zhu et al. (2023).
22
Variance
0.06
0.055
However, we note three things about these ideas: (1) their performance seems to be sensitive to the match of noise type and level between the training of the diffusion mod- cls and those in the actual IPs. Mismatch can lead to miserable results,
20
0.05
0.050
18
Detection
0.04
0.045
0.040
0.03
0.035
14-
0.02
0.030
demonstrate in Tabs. and g
0.01
0.025
(2) DBMs in solving IPs can suffer from overfitting is- sues similar to that in DIP also (sce Appendix A. (3) there has been limited success in tackling nonlinear IPs
10
0.00
0.020
500 1000 1500 2000 2500 3000 (Iteration)
Figure 2: Relationship between the PSNR, MSE,
by DBMs so far, see, e.g., the very recent attempt Chung] le al. (2023). It remains to be seen how effective these ideas can be general nonlinear IPs.
and VAR curves. Our method relies on the VAR
curve, whose valley is often well alignod with the MSE valley that corresponds to the PSNR peak.
2 Our Early-Stopping (ES) Method
Intuition for our method We assume that æ is the unknown groundtruth visual object of size N, {0*), )221 is the iterate sequence and {#t) the reconstruction sequence where x² = Go (2). Since we do not know æ, we cannot compute the PNSR any FR-IQM curve. But observe from Fig. 2 that the MSE (resp. PSNR; recall PSNR(=") 10 log / curve follows a U (resp. bell) shape: || - æ=| initially drops rapidly a low level and then climbs back due the noise effect, i.c. the ELTO phenomenon Sec. T. we hope to detect. the valley of this U-shaped MSE curve.
4

---

Published in Transactions on Machine Learning Research (12/2023)
Then how to gauge the MSE curve without knowing x! We consider the running variance (VAR):
W VAR(() = W Σ + 1/W
W-1
(3)
Initially, the models quickly learn the desired visual content, resulting in a monotonic and rapidly decreasing MSE curve (see Fig. 2p. So we expect the rumning variance of (x' h221 to also drop quickly, as shown in Fig.
When the iteration is near the MSE valley, all x² are near, but scattered around æ. So W Eis x ~ and VAR(I) 2 back, leading to a similar bounce back in the VAR curve as the x' sequence gradually moves away from x. Table 2: ES- WMV (our method) on real-world image denoising for 1024 images: mcan and (std) the images. (D: detected)
1 W Ewz ||a - x: lp Afterward, the noise effect kicks in and the MSE curve bounces
This argument suggests U-shaped VAR curve and the curve should follow the trend of the MSE curve, with approximately aligned valleys, which in turn aligned with the PSNR peak. To quickly verify this, we ran- domly sample 1024 images from the RGB track of the NTIRE 2020 Real Image Denoising Challenge (Abdel-) hamed et al., 2020), and perform DIP-based image de- noising (i.e. min ((()) where y denoles the noisy image). Tab. 2 reports the average detected PSNR/SSIM and the average detection gaps based on our ES method (soe Algorithm 1) that trics to detect the valley of the VAR curve. On average, the dctection gaps are 0.95 in PSNR and 0.02 in SSIM, and the difference in visual qualities is typically barely noticeable by eyes! Furthermore, we provide histograms of the PSNR and SSIM gaps in Fig. 25 For more than 95% of the images, our ES method obtains a PSNR gap less than 2dB.
(loss)
PSNR (D) PSNR Gap SSIM (D) SSIM Gap
MSE
34.04 (3.68) 0.92 (0.83)
0.92 (0.07) 0.02 (0.04)
6
33.92 (4.34)
0.92 (0.59)
0.93 (0.05)
0.02 (0.02)
Huber
33.72 (3.86) 0.95 (0.73)
0.92 (0.06) 0.02 (0.03)
Detecting transition by running variance Our lightweight mothod only involves computing the VAR curve and numerically delecting its valley the iteration stops once the valley is detected. To obtain the curve, we set a window size parame- ter W and compute the windowed moving variance (WMV). To robustly detect the valley, we introduce a patience number P to tolerate up to P consecutive steps of variance stagnation. Obviously, the cost is dominated by the calculation of variance per step, which is O(WN) (N the size of the visual object). In comparison, typical gradient update step for solving Eq. (2) costs at least 1((0) where [0] is the number of parameters in the DNN Gø- Since ( is typically much larger than W (default: 100), our running VAR and detection incur very little compu- 10:
Algorithm 1 DIP with ES WMV
Input: random seed z, randomll--iiitialzzed 00, win- dow size W, patience P, empty queue Q, iteration counter k 0, VAR
Output: reconstruction x² while not stopped do update 0 via Eq. (2) to obtain 0h+1 and #+1 push k+1 to 2, pop queue if [ > W
1:
2:
3:
4:
if | = W then
5:
compute VAR of elements
via Eq. (3)
6:
if VAR < &VARmia then VARmin + VAR, x² k+1
7:
end if 8:
9:
if VARmin stagnates for P iterations then
stop and return x²
end if tational overhead. Our entire algorithmic pipeline 11:
is summarized in Algorithm I To confirm the effec- 12:
end if
liveness, we provide qualitative samples in Figs. 13:
k K+11
and 4 with more quantitative results included in the experiment part (Sec. 3 see also Tab. 2). Fig. 3
end while
shows that for image denoising with different noise types/levels, our ES method can detect ES points that achieve near-peak performance. Similarly, our method remains effective in several popular DIP variants, as shown in Fig. 4, Note that although our detection for DIP-TV in Fig. 4 is a bit far from the peak in terms of iteration count (as the VAR curve is almost flat after the peak), the detection gap still small (~ 1.29dB).
Scemingly similar idcas Our running variance and its U-shaped curve are reminiscent of the classical U- shaped bias- vaaianariance tradeoff curve and therefore validation-based ES (Geman al. 1992 Yang et al. 2020). But there are crucial differences: (1) our learning setting is not supervised; (2) the variance in supervised
5

---

Published in Transactions on Machine Learning Research (12/2023)
Gaussian
Impulse
Shot
Speckle
Low-level noise
21:3
3323
373
1:00
0223
200
6/6
High-level noise
Figure 3: Our ES-WMV method on DIP for denoising "F16' with different noise types and levels (top: low-level noise; bottom: high-level noise). Red curves are PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES point.
learning is with respect to the sample distribution, while our variance here pertains the {ar) sequence. As discussed in Sec.. 1] we cannot directly apply validation-basee based ES, although is possible heuristically emulate it by splitting the elements in y (Yaman al., 2021 Ding et al. 2022 whiic might be problematic for nonlinear IPs. Another line of related ideas is the detection of Pariance-based online change points in time series analysis (Aminikhanhhahi & Cook 2017], where the running variance is often used to detect shifts in means under the assumption that the means are piecewise constant. Here, the piecewise constancy assumption does not hold for our (>"h21
Deep Decoder
GP-DIP
DIP-TV
SIREN
Low-level noise
410
1000
3001
High-level noise
2000
Figure 4: ES-WMV deep decoder, GP-DIP, DIP-TV, and SIREN for denoising "F16" with different levels of Gaussian noise (top: low-level noise; bottom: high-level noise). Red curves PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES point. (We sketch the details of the above DIP variants in Appendix A.5)
Theoretical justification We can make our heuristic argument in Sec. more rigorous by restricting oursclves to additive denoising, that is, y = n, where the noise N (0, 2 (2/n- I). and appcaling
the popular linearization strategy (i.e. neural tangent kernel Jacot (2018): Heckel Soltanolkotabl
(020bb) in understanding DNN. The idea is based the assumption that during DNN training does

---

Published in Transactions on Machine Learning Research (12/2023)
not move much away from initialization 0% that the learning dynamic can be approximated by that of linearized model, L.e. suppose that we take the MSF loss,
ly - -o(z))) R |1 Ggo(z) Jc (0)) 11 f(0),
(4)
where Ja(0%) is the Jacobian of G with respect to O at 00, and Jc (0 00) is the first-order Taylor approximation to Gø(z) around 0t F(O) simply a linear least-squares objective. We can directly calculate the running variance based on the linear model, as shown below.
Theorem 2.1. Let os's and w, 's be the singular values and left singular vectors of Ja(0"), and suppose that we run a gradient descent with step size y, on the linearized objective f(0) to obtain {0'} and {x²} with x² = (z) + Ja (e)) ).). Then, provided that n, I 1/ max;(o?),
VAR(t)= = E Cw.n, (w, (1 n)??)² 2t
(5)
where ÿ = (2), and OW.n. 2 0 depend only on W, n, and as for all i.
The proof can be lound in Appendix A.2 Theorem 2.1 shows that if the learning rate (LR) 17 sufficiently small, the WMV of {œt) decreases monotonically. We can develop complementary upper bound for the WMV that has U shape. To this end, make use of Theorem 1 of Heckel & Soltanolkotabi (2020b), which can be summarized (some technical details omitted;; precise statement is reproduced in Appendix A.3) as follows: consider the two-layer model GC = ReI.U(UBC)), where C B models 1 x trainable convolutions, upsampling operation, and B is the fixed random seed. Let J be reference Jacobian matrix solely determined by the upsampling operation U, and d; r's and w, 's the singular values and left singular vectors of J. Assume that æ E span (w)...., wp). Then, when is sufficienlly small, with high probability,
Exact Calculation Theorem 2.1
35
Upper Bound Theorem 2.2
2.5
2.0
1.0
0.5
Iexx contains fixed weights, U is an
Iteration
Figure 5: The exact and upper bounds predicted by Thhorrms 2.1 and 2.2
|Go (B) - 2112 S (1 (1- no;)) ||0||2 E(n) + +
(6)
where >> is small scalar related to the structure of the network and E(n) is the error introduced by noise: (m) = (1=((( - n)3)²² by (1-10a)) ||a|| when is small and then by F(m) when I is large. However, since the former decreases and the latter increases as f grows, the upper bound has a U shape with respect to t. On the basis of this result, we have the following.
1)² n. So, if the gap op/Tpt > 1, ||G (B) æll2 is dominated
Theorem 2.2. Assume the same setting as Theorem of Hecket & Soltanolkotabl (20206). With high probability, our WMV is upper bounded by
12 W
x 2 (1
-1- ?1?
2t
+ 12 Σ (11-
-1
(w. n)² (w) + 12e 11
(7)
The exact statement and proof can be found in above, we can conclude that the upper bound in Theorem 2.2 also has a U shape. To interpret the results, Fig. shows the curves (as functions of t) predicted by Theorems 2.1 and 2.2 The actual VAR curve should be between the two curves. These results are primitive and limited, similar to the situations for many deep learning theories that provide loose upper and lower bounds; we leave a complete theoretical justification for future work.
Appendix A.3 Using a reasoning process similar to that
7

---

Published in Transactions on Machine Learning Research (12/2023)
A memory-efficient variant While Algorithm is already lightweight and effective in practice, modify it slightly to avoid maintaining Q and therefore saving memory. The trick is to use exponential moving variance (EMV), together with exponential moving average (EMA), shown in Appendix A.4 The hard window size paramcter W is now replaced by the soft forgetting factor a: the larger the a, the smaller the impact of the history, and hence smaller effective window. We systematically compare ES-WMV with ES-EMV in Appendix A.7.13 for image denoising tasks. The latter has slightly better detection due the strong smoothing effect (a 0.1). For this paper, we prefer to remain simple and leave systematic evaluations of ES-EMV other IPs for future work.
3 Experiments
We test ES-WMV for DIP in image denoising, inpainting, demosaicing, super-resolution, MRI reconstruction, and blind image deblurring, spanning both linear and nonlinear IPs. For image
denoising, we also systematically evaluate ES- for main DIP variants, including deep decoder HHeckel
& Hand 2019) DIP-TV (Cascarano et al. 2021 GP-DIP (Cheng et al. 2019), and demonst rate ES- WMV
as a reliable helper to detect good ES points. Details of the DIP variants are discussed in
Appendix A.5
We
also compare US-WWVV with the main competing methods, including DF-STE (Jo et al. et al. 2021). DOP YYou al., 2020), SB (Shi et al. 2022), and VAL Yamman et al. 2021
2021 SV-ES ES (Li
Ding et al. 2022).
Details of the main ES-based methods assess reconstruction quality and report PSNR and SSIM gaps (the difference between our detected and peak mumbers) as indicators of our detection performance. Common acronyms, pointers to external codes, detailed experiment settings, real-world denoising, image inpainting, and image demosaicing
be found in Appendix A.6 We both PSNR and SSIM to
are in Appendices A.1 A.7.1 A.7.2 A.7.7 A.7.8 and A.7.10, respectively.
3.1 Image denoising
Prior work dealing with DIP overfitting mostly focuses on image denoising and typically only evaluates their methods on one or two kinds of noise with low noise levels, c-g., low-level Gaussian noise. To stretch our evaluation, we consider 4 types of noise: Gaussian, shot, im- pulse, and speckle. We take the classical 9- image dataset (Dabov et al. 2008). and for each noise type, generate two noise levels, low and high, i.e., level and 4 ofedendsssks & Di- etterich (2019). respectively. In Tab. and Ap-
28
12
12
Krbdd
L00
Andlo
Granies
Inpabe rabe
PENR-CAP 12
pendix A.7.7 we also report. the performance of our ES-WMMV on real-world denoising eval- uated large-scale datasets. In addition, we also compare DIP-based denoising with a state-of-the-art diffusion-modeel-annee denois-
PAN
Popper
Loae
ing in Tab. 9
Slot 00.5
Specil
Comparison with baseline ES methods It is natural to expect that NR-IQMs, such as the classical BRISQUE (Mittal et al. 2012
Figure 6: Baseline ES vs our ES- WMV on denoising with low-level noise. For NIMA, we report both technical qual- ity assessment (NIMA-q) and aesthetic assessment (NIMA-
NIQE (Mittal et al. 2013), and modern DNN- based NIMA (Esfandarani & Milanfar 2018
a). Smaller PSNR gaps are better.
can be used to monitor the quality of intermediate reconstructions and hence induce natural ES criteria. Therefore, we set baseline methods using BRISQUE, NIQE, and NIMA, respectively, and seek the optimal x² using these motrics. Fig. 6 presents the comparison (in terms of PSNR gaps) these methods with our ES- WMV on denoising with lowelevel noise by using DIP; results on high-level noise and also as measured by SSTM included in Appendix A.7.4] Visual comparisons ween our ES- FS-WMV and the baseline mods
8

---

Published in Transactions on Machine Learning Research (12/2023)
are shown in Figs. 7 and 16 While our method enjoys favorable detection gaps ( 2) for most tested noise types/lever /levels (except for Baboon, Kodak1, Kodak2 for certain noise types/levels, DIP itself is suboptimal in terms of denoising such images with substantial high-frequency components), the baseline methods can see huge detection gaps up to 10.
BRISQUE
NIQE
NIMA-q
NIMA-a
ES-WMV
Peak
Figure 7: Visual comparisons of the detected images by NR-IQMs and ES- BS-WMV. From top to bottom: shot noise (low), shot noise (high), speckle noise (low), speckle noise (high).
Competing methods DF-STE (Jo et al. 2021 is specific for Gaussian and Poisson de- noising, and noise variance is needed for thoir tuning parameters. Fig. 8 presents the com- parison of our method with DF-STE in terms of PSNR; SSIM results are in Appendix A.7.5 Here, directly report the final PSNRs ob- tained by both methods. For low-level noise, there is no clear winner. For high-level noise, ES-WMV outperforms DF-STE by considerable margins. Although the right variance level provided DF-STE order to tune their regularization parameters, DF-STE stops after only very few epochs, leading very low performance and almost zero standard deviations since they return almost the noisy input. However, we do not perform any parameter tuning for ES-WMM Fur- thermore, we compare the two methods on the CBSD68 dataset in Appendix A.7.5 that leads to similar conclusion.
10
30.
20-
15-
Kustad
Kuter?
Ganoiaa moiee
Sker 2ole
Figure 8: Comparison of DF-STE and ES- WM- for Gaus- sian and shot noise in terms of PSNR.
We report the results SV-ES in Appendix A.7.3 since ES- WMV performs largely comparable to SV-ES. However, RS-WMY much faster in wall-clock time, as reported in (Tb.. 3 for each epoch, the over- head of our ES-WMV is less than 3/4 of the DIP update itself, while SV-ES around 25x of that.

---

Published in Transactions on Machine Learning Research (12/2023)
There is surprise: while method only needs to update the running variance of (#')21 each time, SV-ES needs to train a coupled autoencoder which is extremely expensive.
Table 3: Wall-clock time (secs) of DIP and three ES methods per epoch on NVIDIA Tesla K40 GPU: mean and (std). The total wall clock time should contain both DTP and a certain ES method.
DOP designed specifically just for impulse noise, we compare EV-WWV with DOP on
DIP
SV-ES
ES- WMV
ES-EMV
impulse noise (see Appendix A.7.5). The loss is changed to & to account for the sparse noise. In terms of the final PSNRs, DOP outperforms DIP with ES-WMV by a small gap, but even the peak PSNR of DIP with & lags behind DOP by about 2dB for high noise levels.
Time 0.448 (0.030)
13.027 (8.872) 0.301 (0.018) 0.003 (0.003)
Table 4: Comparison between ES- WMV and SB for image denoising on the CBSD68 dataset with varying noise level a. The higher PSNR detected and carlier detection are better, which are in red: mean and (std).
VAL
Vil
25
25
50
PSNR
Epoch
PSNR
Epoch
PSNR
Epoch
WMV 28. 7(3.2) 3962(2506) 27. (72.) 3068(2150) 21.3(2.3) 1548(1939)
Comparison of VAL and ES- WMV for Gaussian and impulse noise in terms of PSNR.
Figure 9:
SB 29.0(s.1) (198) 27.3 22.3(..2) 5099(1776) 23.0(1.0) 5765(1248)
The ES method in SB is acknowledged by its authors to fail for vanilla DIP (Shi let al.; 2022).. Moreover, their modified model still suffers from the overfitting issue beyond very low noise levels, as shown in Fig. 22 Their ES method fails to stop at appropriate places when the noise level is high. Hence, we test both ES-WMV and SB on their mod- ified DIP model in (Shi el al. 2022 based on the two datasets they test: the classic 9- image dataset (Dabov et al.] 20088) and the CBSD68 dataset (Martin et al., 2001p. Qual- itative results on 9 images are shown in Ap- pendix A.7.5) detected PSNR and stop epochs on the CBSD68 dataset are reported in [Tb.. 4 For SB, the detection threshold parameter set to 0.01. ods have similar detection performance for low noise levels, but ES- WMV outperforms SB when the noise level is high. Also, ES- WMV tends to stop much carlier than SB, saving computational cost.
4.5
4.5
40-
40
35-
POME-GAP
30
30-
25
10-
0.5
45
Loop Jecrekr
5.0
4.0-
3.5
3.0
2.5
2.0
1.5
1.0
evident that both meth-
0,0-
dll
DIIIPPTV
SIREN
Figure 10: Performance of ES- on deep decoder, GP- DIP, DIP-TV, and SIREN for Gaussian denoising in terms of PSNR gaps. L: low noise level; H: high noise level.
We compare VAL with our ES-WMV on the 9-image datasct with low-/high-lovel Gaussian and impulse noise. Since Ding et al. (022) takes 90% pixels to train DIP and this usually decreases the peak performance, we report. the final PSNRs detected by both methods (see Fig. Dh. The two ES methods perform very comparably in image denoising, probably due to mild violation of the i..d. assumption only, and also to a relatively low degree of information loss due to data splitting. The more complex nonlinear BID in Sec. 3.4 reveals their gap.
ES- ES-WMV as helper for DIP variants Deep decoder, DIP-TV, and GP-DIP represent different regularization strategies to control overfitting. However, critical issue setting the right hyperparameters for them so that overfitting is removed while peak-level performance is preserved. Therefore, practically,
10

---

Published in Transactions on Machine Learning Research (12/2023)
these methods are not free from overfitting, especially when the noise level is high. Thus, instead of treating them competitors, we test whether ES-WMV can reliably detect good ES points for them. We focus on Gaussian denoising and report the results in Fig. 10 (a)-(c) and [Appendix A.7.6 ES-WMV is able to attain I 1 PNSR gap for most cases, with few outliers; we provide a detailed analysis about some of the outliers in Appendix A.g
WS-WMV as a helper for implicit neural representations (INRs) INRs, such as Tancik et al. 2020 and Sitzmann et al. (0020), use multilayer perceptrons to represent highly nonlinear functions in low-dimensional problem domains and have achieved superior results in complex 3D visual tasks. We further exlend our WS to help the INR. family and take SIREN (Sitzmann et al., 2020) as an example. STREN parameterizes a as the discretization of a continuous function: this function takes in spatial coordinates and returns the corresponding function values. Here, we test SIREN, which is reviewed in Appendix A.5. replacement for DIP models for Gaussian denoising and summarize the results in Fig. 10 anand Fig. 24 ES-WMV is again able to detect near-peak performance for most images.
3.2 Image Super- Resolution
In this task, we try to recover a clean im- age xy from a noisy downsampled ver- sion y = (xo) + e, where D.(+) [0, 1]sxtHxo - [0, 1]3xH*W is down- sampling operator that resizes an im- age by the factor and models ex- tra additive noisc. We consider the fol- lowing DIP-reparametrized formulation ming {(0) [GD)())())) - vI where Go is a trainable DNN parameterized by 0 and is a frozen random seed. Then we conduct experiments for 2x super- resolution with low-level Gaussian and impulse noise. We test our ES-WMV for DIP and state-of-the-ar zero-shot method based on pre-trained diffusion
Table 5: Comparison of ES-WMV for DIP and DDNM+ Wang et al. (2022) for 2X image super-resolution with low-level Gaussian and impulse noise: mean and (std). The highest PSNR and SSIM for each task are in red. In particular, we set the best hyperparameter for DDNM+ (o, = 0.12), which is unfair for the DIP + ES- combination as we fix its hyperparameter setting.
PSNR
SSIM
Gaussian
Impulse
Gaussian
Impulse
DIP (pcak)
22.88 (1.58) 28.28 (2.78) 0.61 (0.00) 0.88 (0.05)
DIP + ES-EEM WS-
22.11 (1.90)
26.77 (5.76)
0.54 (0.11)
0.86 (0.96)
DDNM+ (ay .12)
25.37 (2.00)
18.50 (0.68)
0.74 (0.11)
0.50 (0.06)
DDNM+ (a, .00
16.91 (0.42)
16.59 (0.34)
0.31 (0.08)
0.49 (0.06)
model DDNM+ Wang et al. (2022) on the standard super-resolution dataset Set14 Zeyde et al. 2012 as shown in Tab. 5. from large external training datasets, while DIP does not. We observe that (1) Our ES- WS- is again able to detect near-peak performance for most images: the average PSNR gap is < 1.50 and the average SSIM gap is < 0.07; (2) DDNM+ is sensitive to the noise lype and level: from Tab. 5] DDNM+ trained assuming Gaussian noise level 0g = 0.12 outperforms DIP and DIP+ES-WMV when there is Gaus- sian measurement noise at the level ag = 0.12, which is unrealistic in practice, as the noise level is often unknown beforehand. When the noise level is not set correctly, e.g., as 0g = in the DDNM+ (a,, = .0) row of Tab. 5| the performance of DDNM+ is much worse than that of DTP and DIP+ES-WWV. Also, for super-resolution with impulse noise, DIP is also a clear winner that leads DDNM+ by a large margin; and (3) in [Appenixx A.8 we show that DDNM+ may also suffer from the overfitting issue and our ES-WMV can help DDNM+ to stop around the performance peak as well.
Fig. 11 and Appendix A.7.9. We note that DDNM+ relies on pre-trained models
3.3 MRI reconstruction
We further test ES-WMV on MRI reconstruction, clas- sical linear IP with a nontrivial forward mapping: y ~ F(z), where F is the subsampled Fourier operator, and we use & to indicate that the noise encountered in practi- cal MRI imaging may be hybrid (e.g., additive, shot) and uncertain. Here, we take the 8-fold undersampling and
Table 6: ConvDecoder on MRI reconstruction for 30 cases: mean and (std). (D: Detected)
PSNR(D) PSNR Gap SSIM(D) SSIM Gap
32.63 (2. (2:36)
0.23 (0.32)
0.81 (0.00) 0.01 (0.01)
11

---

Published in Transactions on Machine Learning Research (12/2023)
DIP Peak
DIP+ES-WMV
DIP (Overfitting)
DDNM+ (oy .12)
DDNM+ (ay .00)
Figure 11: Visual comparisons for 2x image super-resolution task with additional noise. Top row: with additional low-level Gaussian noise; Bottom row: with additional low-level impulse noise.
parameterize æ using "Conv-Decoder" DDaeetaani & Heckel 2021). a variant of deep decoder. Due to the heavy over-parameeterization, overfitting occurs and ES is needed. Darestani & Heckel (2021) directly sets the stopping point at the 2500-th epoch, and we run our ES- WMV. We visualize the performance on two random cases (C1: 1001339 and C2: 1000190 sampled from Darestani & Hockel (2021). part of the MRT datatset (Zbontar al. 2018 in Fig. 29 (quality measured in SSIM, consistent with Darestani & Heckel (2021)). It is clear that ES- WMV detects near-peak performance for both cases and is adaptive enough to yield comparable or better ES points than heuristccally fixed ES points. Furthermore, test our CS- on ConvDecoder for 30 cases from the fastMRI dataset (sce Tab. 6), which shows the precisc and stable detection of ES-WMV.
3.4 Blind image deblurring (BID)
In BID, a blurry and noisy image is given, and the goal is to recover a sharp and clean image. The blur is mostly caused by motion and/or op- tical non-ideality in the camera, and the forward process is often modeled as = k x + n, where k is the blur kernel, n models additive sensory noise, and * is linear convolution to model the spa- tial uniformity of the blur effect (Szeliski 2022) BID a very challenging visual TP due to bilin- earity: (k) + k w. Recently, Ren al.
PSNR
WLL Overtt
VAL
LAU, Pesk
Overtt
2020 Wang et al. 22019;; Asim et al. (2020): Tran et al. (2021) have tried to use DIP models to solve BID by modeling & and x as two separate DNNs, i.e., ming..e, |yy Go, * (z.))= + A|VGo, (zx) (xx))l//[VGG, VGg (22) where the regular- izer al. (2023b) is to promote sparsity in the gradient domain for the reconstruction of x, as stan-
Image Comhitation
Insge-Kemel Combiiation
Figure 12: Top left: ES-WMM in BID; top right: vi- sual results of ES-WMV; bottom: quantitative results of ES- CSV and VAL, respectively.
dard in BID. We follow Ren al. (2020) and choose
12

---

Published in Transactions on Machine Learning Research (12/2023)
multilayer perceptron (MLP) with softmax activation for Ges: and the canonical DIP model (CNN-based encoder-decoder architecture) for (2x). We change their regularizer from the original || |V (z,)ll 1I to the current, as their original formulation is tested only at a very low noise level o = 10 and no overfitting is observed. We set the test with a higher noise level a = 10-3 and find that its original formulation does not work. The benefit of the modified regularizer on BID is discussed in Krishnan et al. (2011)
First, we take 4 images and 3 kernels from the standard Levin dataset QLevin et al. 2011p, re- sulting in 12 image-kernng kernel combinations. The high noise level leads to substantial overfitting, as shown in Fig. 12 (top left). However, ES-
natiensanuuberr
100
Dep
Window size
WMV can reliably detect good ES points and lead to impressive visual roconstructions (sce Fig. 12 (top right)). We systematically com- pare VAL and our ES-WMV on this difficult. nonlinear IP, as we suspect that nonlinearity can break down VAL as discussed in infscc.1 1 and subsampling the observation y for training- validation splitting may be unwise. Our results (ig. 12 (bottom left/rightt) confirm these pre- dictions: the peak performance detected by VAL is much worse after 10% of the clements in y are removed for valida- tion. In contrast, our ES-WMV returns quan- titatively near-peak performance, much better than leaving the process to overfit. In Tab. 13 we further test both low- and high-level noise on the entire Levin dataset for completeness.
PSNR: GAP
Wath
Levrring rabe
(a)
(b)
Patiencc number
Depth
Window size
PSWK
124
With
Loaminn rose
Iteratioon
(c)
(d)
Figure 13: (a) Histogram of PSNR gaps for 243 different hyperparamter combinations; (b) radar plot of the cases whose PSNR gaps are larger than 2 dB; (c) radar plot of the cases whose PSNR gaps are larger than 2 dB and window size = 100; (d) an example case to visualize VAR curves with different window sizes. (For radar plots, there are three grid values for each hyperparameter. The farther away from the center, the higher the value.)
3.5 Ablation study
The window size W (default: 100) and the pa- tience number P (default: 1000) are the only hyperparameters for our ES-WMV. Moreover, in this abalation study, we also include the key DIP hyperparameters, which obviously can also affect our ES performance- in our experiments above, have used the default published DIP hyperparameters for each IP, as our ES method works under the condition that DIP performs reasonably well for the IP under consideration. To this end, we select the lcarning rate, which typically determines the learning pace and peak performance of DIP, and the depth/width of the network, which rules the network capacity.
Our base task is Gaussian denoising on the classic 9-image dataset (Dabov et al. 2008) with medium-level noise. We take the same default U-Net backbone model as the experiments in Fig. 3 and perform experiments the following hyperparameter grid: window size {500, 100, 10), patience number {5000, 1000, 100), DIP learning rate (0.01, 0.001, 0.0001}, DIP model width (256, 128, 64), and DIP model depth (,,5, 3}. resulting in total of 243 hyperparameter combinations. For each combination, we calculate the mean PSNR gap, on which our subsequent analysis is based. First of all, we see from Fig. 13(a) that for most (2 83%) hyperparameter combinations, the mean PSNR gap falls below 2 dB. For cases larger than 2dB, we use the radar plot Fig. 13(b) to explore the dcciding factors and find that most of these cases tend have small (10) or medium (100) window sizes. This is not surprising, as small window size can lead to a very fluctuating VAR curve, as shown in Fig. 13).). d). To further explore other deciding factors,
focus on the subset with
mean PSNR gap 2 2dB and window size = 100 and plot their settings in Fig. 13(c). We find that these cases invariably have small patience number (100), which can trap our valley detection algorithm into a local fluctuation. So, overall, it seems that our window size and patience number are deciding factors for failures, relative to DIP hyperparameters such as learning rate and network capacity.
Hence, we next look closely at the combined effect of window size (W) and patience number (P) on ES performance. For this, we plot 9 histograms for the 9 different (W, P) combinations in Fig. 14 (i.e., each
13

---

Published in Transactions on Machine Learning Research (12/2023)
histogram is the 27 DIP hyperparameters). The trend is clear: the larger the patience number, the smaller the PSNR gaps; the larger the window size, the smaller the detected PSNR gaps. The average PSNR gap of our default hyperparameter combinations is below 1 dB (the center one). If we further increase the patience number and the window size, the average PSNR gap can even be lower than 0.5 dB (top left corner). Overall, this confirms again our above observation that window size and patience number are the deciding factors for the detection performance of our method. Also, this suggests that our ES method can operate well with small PSNR gaps over a wide range of combinations (W, P), unless both are very small.
4 Discussion
We have proposed a simple yet effective ES detection method (ES-WMM) and the ES-EMV variant) that works robustly multiple visual IPs and DIP variants. In comparison, most competing ES methods noise- or DIP-moodelsppecffc and only work for limited scenarios; Li et al. 2021 has comparable performance
Mean PSNR-GAP: 0.45
Mean PSNR-GAP: 0.80
Mean PSNR-GAP: 1.50
W: 500, P: 5000
W: 100, P: 5000
W: 10, P: 5000
8
Frequency
Frequency
Frequency
6
10
10
PSNR-GAP
PSNR-GAP
PSNR- GAP
Mean PSNR-GAP: 0.49
Mean PSNR-GAP: 0.85
Mean PSNR-GAP: 1.71
W: 500, P: 1000
W: 100, P: 1000
W: 10, 1000
Frequency
8
Frequency
Frequency
PSNR-GAP
PSNR-GAP
PSNR-GAP
PSNR-GAP:
500,
100, 100
W: 10, 100
Frequency 6
Frequency
Frequency
PSNR-GAP
PSNR-GAP
PSNR-GAP
Figure 14: histograms for 9 different hyperparameter combinations of our ES method. In each histogram, show the frequencies of PSNR-GAPs for 27 various DIP hyperparameter possibilities. W: window size; P: patience number.
14

---

Published in Transactions on Machine Learning Research (12/2023)
but slows down the running speed too much; validation-based ES DDng et al. 2022 works well for the simple denoising task while significantly lags behind our ES method on nonlinear IPs, e.g. BID.
As for limitations, our theoretical justification is only partial, sharing the same difficulty of analyzing DNNs in general; our ES method struggles with images with substantial high-frequency components; our detection is sometimes off the peak in terms of iteration numbers when helping certain DIP variants, e-g. DIP-TV with low-level Gaussian noise FFig. 4), but the detected PSNR gap is still small. DIP variants typically do not improve peak performance and also do not necessarily avoid overfitting, especially for high- level noise. We recommend the original DIP with our ES method for visual IPs discussed in this paper for the best performance and overall speed. Besides ES, there are other major technical barriers to making DIP models practical and competitive for visual IPs. A major one is efficiency: one needs to train a DNN using iterative methods for every instance; our recent works Li et al. 2023c d) have made progress on this issue.
Acknowledgements
Zhong Zhuang, Hengkang Wang, Tiancong Chen and Ju Sun are partly supported by NSF CMMI 2038403. We thank the anonymous reviewers and the associate editor for their insightful comments that have sub- stantially helped us to improve the presentation of this paper. The authors acknowledge the Minnesola Supercomputing Institute (MST) at the University of Minnesota for providing resources that contributed to the research results reported within this paper.
References
AbdeIrahman Abdelhamed, Mahmoud Afifi, Radu Timofte, Michael Brown, Yue Cao, Zhilu Zhang, Wang- meng Zuo, Xiaoling Zhang, Jiye Liu, Wendong Chen, Changyuan Wen, Meng Liu, Shuailin Lv, Yunchao Zhang, Zhihong Pan, Baopu Li, Teng Xi, Yanwen Fan, Xiyu Yu, Gang Zhang, Jingtuo Liu, Junyu Han, Errui Ding, Songhyun Yu, Bumjun Park, Jechang Jeong, Shuai Liu, Ziyao Zong, Nan Nan, Chenghua Li, Zengli Yang, Long Bao, Shuangquan Wang, Dongwoon Bai, Jungwon Lee, Youngjung Kim, Kyeongha Rho, Changyeop Shin, Sungho Kim, Pengliang Tang, Yiyun Zhao, Yuqian Zhou, Yuchen Fan, Thomas S. Huang, Zhihao Li, Nisarg A. Shah, Wei Liu, Qiong Yan, Yuzhi Zhao, Marcin Mozejko, Tomasz Latkowski, Lukasz Treszczokko, Michal Szalraniuk, Krzysztof Trojanowski, Yanhong Wu, Pablo Navarrete Michelini, Fengshuo Hu, Yunhua Lu, Sujin Kim, Wonjin Kim, Jaayeon Lee, Jang- Choi, Maganiya Zhussip, Azamat Khassenov, Jong Hyun Kim, Hwechul Cho, Priya Kansal, Sabari Nathan, Zhangyu Ye, Xiwen Lu, Yaqi Wu, Jiangzin Yang, Yanlong Cao, Siliang Tang, Yanpeng Cao, Matteo Maggioni, Ioannis Marras, Thomas Tanay, Gregory G. Slabaugh, Youliang Yan, Myungjoo Kang, Han-Soo Choi, Kyungmin Song, Shusong Xu, Xiaomu Lu, Tingniao Wang, Chunxia Lei, Bin Liu, Rajat Gupta, and Vineet Kumar. NTIRE 2020 challenge on real image demoising: Dataset, methods and results. In 2020 IEEE/CH /CVF Conference on Computer Vision and Pattern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 2077-2088 Computer Vision Foundation / IEFE, 2020. doi: 10. 10. 11109//VRR500098... 200...20200000222
Samaneh Aminikhanghahi and Diane J. Cook. A survey of methods for time series change point detection. Knowl. Inf. Syst., (1(2):339 367, 2017. doi: 10.
Muhammad Asim, Fahad Shamshad, and Ali Ahmod. Blind image deconvolution using deep generative priors. IEEE Trans. Computational Imaging, 6:1493 1506, 2020. doi: 00.108//CC1.022.3333377
Daniel Otero Baguer, Johannes Leuschner, and Maximilian Schmidt. Computed tomography reconstruction using deep image prior and learned reconstruction methods. CoRR, abs/2003. 04989, 2020.
Khosro Bahrami and A. C. Kot. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Process. Lett., 21(6):751 155,, 2014. 2314487.
doi: 10.1109/LSP:2014.
Pasquale Cascarano, Andrea Scbastiani, Maria Colomba Comes, Giorgia Franchini, and Federica Porta. Combining weighted total variation and deep image prior for natural and medical image restoration via ADMM. In 2021 s1 International Conference on Computational Science and Its Applications (ICCSA),
15

---

Published in Transactions on Machine Learning Research (12/2023)
Cagliari, Ital,, September 13-16, 2021 Workshops, pp. 39-46. IEEE, 2021. doi: 10.1109/ICCSA554 2021 .00016.
Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective the deep image prior. In TEEE Conference on Computer Vision and Pallern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 5443 5451. Computer Vision Foundation / LEEE, 2019. doi:
D0.109/CCVRR2011.
Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations, 2023. URL ittps://openrevieet net wttp:/r.wwee.ooorrdddddaaaaaaaa.
Frederique Crete, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. The blur effect: perception and estimation with a new no-reference perceptual blur metric. In Bernice E. Rogowitz, Thrasyvoulos N. Pappas, and Scott J. Daly (ods.), Human Vision and Flectronic Imaging XII, San Jose, CA, USA, January 29 February 1, 2007, volume 6492 of SPIE Proceedings, pp. 64920I. SPIE, 2007. doi: 10.1117/12 70.77900.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen O. Egiazarian. Image restoration by sparse 3d dramsform-domain collaborative filtering. In Jaakko Astola, Karen O. Egiazarian, and Edward R. Dougherty (eds.), Image Processing: Algorithms and Systems VI, San Jose, Culifornia, USA, January 28-29, 2008, volume 6812 of SPIE Proceedings, pp. 681207. SPIE, 2008, doi: 10.1117/12.766355.
Mohammad Zalbagi Darestani and Reinhard Heckel. Accelerated MRI with un-trained neural networks. IEEE Trans. Computational Imaging, 7:724-733, 2021. doi: 10. 2CT2TTT22222119775580
Lijun Ding, Liwei Jiang, Yudong Chen, Qing Qu, and Zhihui Zhu. Rank overspecified robust matrix recovery: Subgradient method and exact recovery. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6- 6-14, 2021, virtual, pp. 08767-26776 2021.
Lijun Ding, Zhen Qin, Liwei Jiang, Jinxin Zhou, and Zhihui Zhu. A validation approach to over- parameterized matrix and image recovery. CoRR, Abs//220..10655 2022. doi: 10.48550/ar /at00//rXiv.200000 1067.
Hossein Talebi Esfandarani and Peyman Milanfar. NIMA: neural image assessment. IEEE Trans. Image Process., 3998) 4011, 2018. doi: 11199//1P.000..2880
Yossi Gandelsman, Assaf Shocher, and Michal Irani. 'double-dip': Unsupervised image decomposition via coupled seep-image-priorss In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11026-11035. Computer Vision Foundation IEEE, 2019. doi:
201....1..........
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Newral Comput., 4(1):1-58, 1992. doi:
Kuang Gong, Ciprian Catana, Jinyi Qi, and Quanzheng Li. Direct reconstruction of linear parametric images from dynamic PET using nonlocal deep image prior. IEEE Trans. Medical Imaging, 41 41(3):680-66) 689, 2022. doi: 10.1109/TMT.200 202.. -329133..
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retrieval under a generative prior. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Biande Bianchi, and Roman Garnett (ods.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Moniréal, Canada, pp. 9154 9164, 2018.
Fumio Hashimoto and Kibo Otc. Direct PET image roconstruction incorporating deep image prior and a forward projection model. CoRR, abs/2109.000 2109.0009, 2021.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non- convolutional networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.ned, 2019.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Reinhard Heckel and Mahdi Soltanolkotabi. Compressive sensing with un-trained neural networks: Gradient descent. finds a smooth approxination. Tn Proceedings of the 37th International Conference on Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4149-4158 PMLR, 2020a.
Reinhard Heckel and Mahdi Soltanolkotabi. Denoising and regularization via exploiting the structural bias of convolutional generators. In 8th International Conference on Learning Represenlations, TCLR 2020, Addis Ababo, Ethiopia, April 26-30, 2020. OpenReview.nes, 2020b.
Dan Hendrycks and Thomas G. Dictterich. Benchmarking neural network robustness to common corrup- tions and perturbations. In 7th International Conference Learning Represenlations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Arthur Jacot, Clément Hongler, and Franck Gabriel. Neural tangent kernel: Convergence and generaliza- tion in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: An- nual Conference on Newral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018.
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehicles: Problems, datasets and state of the art Found. Trends Compul. Graph. Vis., 12(1-3):1 308, 2020. doi: (0.1561/060000000
Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 5067-5076. IEEE, 2021. doi: D0.119//CCV449922220..000
Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using a normalized sparsity measure, In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp. 233 240. IEEE Computer Sociely, 2011. 5995521.
doi: 10.119/CVPR.2011.
Anat Levin, Yair Weiss, Frédo Durand, and William T. Freeman. Understanding blind deconvolution algo- rithmss IEEE Trans. Pattern Anal. Mach. Intell., 33(1))22354 2367, 2011. doi: 10.119//PAML..011.44..
Taihui Li, Zhong Zhuang, Hengyue Liang, Le Peng, Hengkang Wang, and Ju Sun. Self-validation: Early stopping for single-instance deep generative priors. In 32nd British Machine Vision Conference 2021, BMVC 2021, Online, November 22-25, 2021, pp. 108. BMVA Press, 2021.
Taihui Li, Anish Lahiri, Yutong Dai, and Owen Mayer. Joint demosaicing and denoising with double deep image priors. CoRR, abs/ 22229.009922 2023a. doi: 21155//aaXiv..20.009222 URL htttps://oi.onn org/
10. 48554/555/500 48550/arXiv.... arwaa.aivv..... 2309. 0309.09426 09426
Taihui TA,, Hengkang Wang, Le Peng, Xian'e Tang, and Ju Sun. Robust Autoencoders for Collective Cor- ruption Removal. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, June 2023b. doi: 10.1109 ALL.0/IICASSP4995..0000 20057..0223.0005000 URL https://ieeexpporeeee ieee. org/ htt.:///r......eeeooeaaaaaaaaaoa abstractt/dcmmett//099966
ISSN: 2379-190X.
Taihui Li, Hengkang Wang, Zhong Zhuang, and Ju Sun. Deep Random Projector: Accelerated Deep Image Prior. In 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 18176 18185, Vancouver, BC, Canada, June 2023c. IEEE. ISBN 9798350301298,
doi: 10. 20.109//VVPR32229.2222
01743. URL https: :tttps://ieeexporrs ieee. org/ arg//dccueen//10252766..
Taihui Li, Zhong Zhuang, Hengkang Wang, and Ju Sun. Random Projector: Efficient Deep Image Prior. In ICASSP 2023 2023 JEEE International Conference on Acoustics, Speech and Signal Pro-
cessing (ICASSP), pp. 1-5, Rhodes Island, Greece, June 2023d. IEEE. ISBN 978-1-72816-327-7. doi: 10.1109/TCASSP44920 22023. 10097088. URL https://ieeexppore... ieee.
htt.:///pppppp.......oooooooea/a org//docmmen///0097008/
17

---

Published in Transactions on Machine Learning Research (12/2023)
Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S. Kamilov. Image restoration using total variation regularized deep image prior. In IEFE International Conference on Acoustics, Speech and Signal Pro- cessing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019, pp. 7715 7719. IEEE, 2019. doi: 10.1109/ICASSP... 20....0099....
Xudong Ma, Alin Achim, and Paul R. Hill. Unsupervised image fusion using deep image priors. CoRR, abs/2110.094900 2021.
David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik. database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statis- tics. In Proceedings of the Eighth International Conference On Computer Vision (ICCV-01), Vancoueer, British Columbia, Canada, July 7-14, 2001 Volume 2, pp. 416 425. IEEE Computer Society, 2001. doi: (0.1109/ICCV) htt///C...000...0000 2001.9365..
Gary Mataev, Peyman Milanfar, and Michael Elad. Deepred: Deep image prior powered by red. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Christopher A. Metzler, Ali Mousavi, Reinhard Heckel, and Richard G. Baraniuk. Unsupervised learning with stein's unbiased risk estimator. CoRR, abs/ abs/1805.10531, 2018.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality assessment, in the spatial domain. IEEE Trans. Image Process., 21(12):4695 4708, 2012. doi: 10.1109 (L1099/TIP.0010 20112.2012........ 2e..22211400 14050.
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making 'completely blind" image quality analyzer. doi: 10.1009/LSP:2012222277 IEEE Signal Process. Lett., (20)))09922222 2013.
Gregory Ongie, Ajil Jalal, Christopher A. Metzler, Richard G. Baraniuk, Alexandros G. Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39-55 56, 2020. doi: http://jaat.222.29900
Adnan Qayyum, Inaam lahii, Fahad Shamshad, Farid Boussaid, Mohammed Bennamoun, and Junaid Qadir. Untrained neural network priors for inverse imaging problems: A survey, TechRaiv, mar 2021. doi: http:////occrxi.140000
Dongwei Ren, Kai Zhang, Qilong Wang, Qinghua Hu, and Wangmeng Zuo. Neural blind deconvolution using deep priors. In 2020 IEEE/CVE Conference on Computer Vision and Pallern Recognition, CVPR 2020, Seattle, WA, USA, June 18-19, 2020, pp. 3338 0488-337..20 Computer Vision Foundation IEEE, 2020. doi: 10.1109 10.1109/CVPP /C/////VRR422000000000 200.22600000200034400
Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G. M. Snock. On measuring and control- ling the spectral bias of the deep image prior. Int. J. Comput. Vis., 130(4):885 908, 2022. doi:
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Marc' Aurelio Ran- zato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Zhaodong Sun. Solving inverse problems with bybrid deep image priors: the challenge of preventing overfit- ting. CoRR, abs/2011.01744 2020.
Richard Szeliski. Computer Vision Algorithms and Applications, Second Edition. Texts in Computer Science. Springer, 2022. ISBN 030-333---- doi: 10.1007/076-33-00--33332
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florrna Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33:
18

---

Published in Transactions on Machine Learning Research (12/2023)
Annual Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
Kshitij
Tayal, Raunak Manekar, Zhong Zhuang, David Yang, Vipin Kumar, Felix Hofmann, and Ju Sun. Phase retrieval using single-instance deep generative prior. Colll, abs/2106.04812, 2021.
Phong Tran, Anh Tuan Tran, Quynh Phing, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 11956-11965. Computer Vision Foundation / TEEE, 2021. doi: N0.1109/CVPR444337 2021.01178.
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 9446-9454. Computer Vision Foundation / TEEE Computer Society, 2018. doi: R110099/CVR.0088..
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse re- covery. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Allch--Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouser, BC, Canada, pp. 2968 2979, 2019.
David Van Veen, Ajil Jalal, Eric Price, Sriram Vishwanath, and Alexandros G. Dimakis. Compressed sensing with deep image prior and learned regularization. CoRR, abs/1806.00444 2018.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-Shot Image Restoration Using Denoising Diffusion Null- wwwp://.wwr///b//22....000. arXiv:2212.00490 [cs].
Space Model, December 2022. URL
Zhunxuan Wang, Zipei Wang. Qiqi Li, and Hakan Bilen. Image deconvolution with deep image and kernel priors. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, JCCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019, pp. 980-989. IEEE, 2019.
Francis Williams, Teseo Schneider, Cláudio T. Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep ge- ometric prior for surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 10130 10139. Computer Vision Foundation / IEEE, 2019. doi: 00A00///VP.22090110000
Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: A new benchmark. CoRR, abs/1104.02603, 2018.
Burhaneddin Yaman, Seyed Amir Hossein Hosseini, and Mehmet Akcakaya. Zero-shot physics-guided deep learning for subject-specifie MRI reconstruction. In NewrIPS 2021 Workshop on Deep Learning and Inverse Problems, 2021.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bins-variance trade-off for generalization neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10777- PMLR, 2020.
Jaejun Yoo, Kyong Hwan Jin, Harshit Gupta, Jérôme Yerly, Matthias Stuber, and Michael Unser. Time- dependent deep image prior for dynamic MRI. JEEE Trans. Medical Imaging, 40(12):3337-3348, 2021. doi: 10.1109/TMI.20211 20113333333..88
Chong You, Zhihui Zhu, Qing Qu, and Yi Ma. Robust recovery via implicit bias of discrepant learning rates for double over parametrrization. In Hugo Larochelle, Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (ods.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Jure Zbontar, Florian Knoll, Amuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Marc Parente, Kraysziof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal, Adriana Romero, Michael G. Rabbat, Pascal Vincent, James Pinkerton, Duo Wang, Nafissa Yakubova, Erich Owens, C. Lawrence Zitnick, Michael P. Rocht, Daniel K. Sodickson, and Yvonne W. Lui. fastmri: An 2018
open dataset and benchmarks for accelerated MRI. CoRR, abs/1811.08$
Roman Zeyde, Michael Flad, and Matan Protter. On Single Image Scale-Up Using Oparse-Repressntattee In Jean-Daniel Boissonnat, Patrick Chenin, Albert Cohen, Christian Gout, Tom Lyche, Marie-Laurence Mazure, and Larry Schumaker (ods.), Curves and Surfaces, volume 6920, pp. 711-730. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-661--24112-11 978-3-642-27413-8. 118-3-642-27411-22 47. URL http://link. springer - tle: Lecture Notes in Computer Science.
doi: 10.1007/
Series Ti-
Yuanzhi Zhu, Kai Zhang, Jingyun Liang, Jiczhang Cao, Bihan Wen, Radu Timofte, and Luc Van Gool. Denoising Dillusion Models for Plug-and-Play Image Restoration, May 2023. URL http://arxiv.orgg arXiv:2305.08995 [cs, eess].
Zhong Zhuang, Taihui Li, Hengkang Wang, and Ju Sun. Blind image deblurring with unknown kernel size and substantial noise. CoRR, abs/2208 0.0443,,, 2022a. doi: 10.48550/ W.iv.2208.094
Zhong Zhuang, David Yang, Felix Hofmann, David Barmherzig, and Ju Sun. Practical phase retrieval using double deep image priors. arXiv preprint arXiv:2211.0077 2022b.
A Appendix
A.1 Acronyms
List of Common Acronyms (in alphabetic order)
CNN
convolutional neural network
DIP
deep image prior
DIP-TV
DIP with total variation regularization
DNN
deep neural network
ELTO
heely-earriing-hhn-overfffeeee
ES
early stopping
EMA
exponential moving average
EMV
exponential moving variance
FR-IQM
full-reference image quality metric
GP-DIP
Gaussian process DIP
INR
implicit neural representations
IP
inverse problem mean squared error no-reference image quality metric
MSE
NR-IQM
PSNR
peak signal-to-noie ratio sinusoidal representation networks
STREN
VAR
variance windowed moving variance
WMV
A.2 Proof of 2.1
Proof. To simplify the notation, we write ÿ = y Ggo(2), = =Jc(f)). and c 0-00% So, the least-squares objective Eq. (4) equivalent to.
- -Jcll
(8)
20

---

Published in Transactions on Machine Learning Research (12/2023)
and the gradient update reads
c' =ccc-l- (Je ÿ),
(9)
where c² and x' Jc (g)(()) The residual time t can be computed as
r¹ y - Je ÿ - J (c -1jJ (J0'-1-1)
(10)
(11)
(I JJJJ)) (ÿ - Jcc'
(12)
(I nj J (g Jc Jc =
(13)
[I JJJ (g Jc') (using c² = 0)
(14)
[I JJJ g.
(15)
Assune that the SVD of J is as WEVT, Then
r¹ = [I W1 0= no?)² wT. yws,
(16)
and so
Jc = ÿ - (1- no?)") w. Tw..
(17)
Consider a set of W vectors V = [UI,... Uw We have the empirical variance.
1 W W
VAR(V) = Σ U
Vj
V
(18)
W
j=1
1
Therefore, the variance of the set {-²,
x
same as the variance of the set
(Je', Jettl ...... W--, can be calculated as
W-1
(w Ø)) 2 (1 (1 no? no?))tuu)
(wT
2
Σ 1- (1 - (1- ++w
(19)
W22
W-1
W-1
27
1
2 (wT8²
4)
W Σ
[ (1 (tw)) 2
Σ 1- (1 m7??) +
(20)
W
w=0
[(w2+ W (1 - no? (1-no3) (1 (1 no? 2
2W (1 110 (
(1 no?
Σ (w
W
- (1- - ?oo
?o!
W 2W (1 - 1177)(11 (1-107)'(1 - (1
m
(1 - (1- noo))1
(1) 1
(1
(ootu Jn
(nm 3
(21)
no?
W2 Σ (w; y)² (1 - -
(1 - m2? W
(1 - (1 (1- no?)
(22)
2 110
?oll
So the constants OW.y.o. 's are defined as
W (1 - ?
(1 - (1 - no? )
OW..o = W2no?
(23)
?o!?
21

---

Published in Transactions on Machine Learning Research (12/2023)
To they are nonnegative, it is sufficient to show that W (1 no?)2
(1- (1 no?) )
2 20
2 7
- no (1 ( - (2- n1)) (1 m0?)) 0. (24)
Now consider the function.
WC, W/) - (1-1)))))21) (2 {) - (1 - EE [0, W1.
(25)
First, one can easily check that ow Ww,, () 2 0 for all W 2 and all { [0,1], that is, h(c, W) increases
monotonically with respect to W. Thus, to prove Cw.n.. 2 0, it suffices to show that h(g, 1) 0. Now
1(g,1) 5(1 5(1 (1 5) - (2 -- = 0.
(26)
completing the proof.
A.3 Proof of 2.2
We first re-state Theorem 2 in Heckel & Soltanolkotabi (020b)
Theorem A.1 (Heckel & Soltanolkotabi (2020b)). Let x E E. be a signal in the span of the first p trigonometric basis functions, and consider a noisy observation = x
= + n, where the noise n
N (0, < /nnnnnn I). To denoise this signal, we fit a two-layer generator network Gc ReLU(UBBC) where [1...., 1, -1 -1].....1l//k. and B miid (N, 1), and U is upsumpling operator that implements circu- lar convolution with given kernel u. Denote o = ||u||| Fy(( @ u/ u/|lu|/[)]12 where g(t) = (1 coo --s--ooss- ct)- (t)/m)t and @ denote the circular convolution. Fix any E (0,op//1], and suppose that k: 2 Cun/E where Cn > O is a constant depending only on u. Consider gradient descent with step size nj S || (Fu is the Fourier Then, for all iterations obeying
transform of u ) starting from Co rid N (0, w²) entries lull vn
< 100 the reconstruction error obeys no?
|G (B) -x1l2 S (1 no?)) || +
Σ ((1 (1-ma7)² 1) 1)<(wnn) n)² + +ellull2
with probability at least 1 - exp (-k2) n
Note that since B wii N( 1) and hence is full-rank with probability one, the original Theorem 1 & 2 of Heckel & Soltanolkotabi (2020b) rename BC to C" and state the result directly on C", that is, assume that the model is ReLU(UC") v. It is easy to see that the original theorems imply the version stated here.
With this, we can obtain our Theorem 2.2, stated full technical form here:
Theorem A.2. Let æ E Rnn be signal in the span of the first p trigonometric basis functions, and consider a noisy observation y = x + n, where the noise n ~ -NNNN0. (0, E /n I). To denoise this signal, we fit two-layer generator network Gc(B) = ReLU(U BC),, where v =
-1/vkk, and B rii N 1), and U an upsampling operator that implements circular convolution with a given kernel u. Denote a ||ul|| Fg(u@u/llul 1-/2 where g(t) ((t)) (1-cos cos (t)/m)) and @ denotes the circular convolution. Fix any E (0, (0.op//1), and suppose 1e 2 Cun/e", where > Cu descent with step size n S ||Fu| (Fuu (Fu is the Fourier transform of u starting from Co mad N (0,w²), entries x livll,
constant only depending on u. Consider gradient
Then, for all iterates t obeying < no?
100
our WMV obeys
Vn
12 W
x212 2 (1 (1-11) ?1?2) 2
+ 12 Σ (1-mo?)) ++ -1)((00,77) (w!n)²² +00++2255 -12c211v13
11
WMV <
(27)
(1 -)oo)-=
with probability at least 1 exp (-k2) -2
22

---

Published in Transactions on Machine Learning Research (12/2023)
Proof. We make use of the basic incquality: ||a dimension. We have
bll2 s + for any two vectors b of compatible
1
| (B)
Ga 1
(28)
W
W
j=0
W-1
W-1
1
|G (B) - x +
G (B)
(29)
W
W 1=
w==
W
W-1
2
|G (B) - =012
+
+2
Σ
Gouz (B)1
(30)
W
W
j=0
2
W
1
2
W-1
| (B) =
Σ
c
=x1
(31)
W
j=0
(z +
2|
x l/ convex and Jensen's inequality)
W-
4
| (B)-
(32)
W
In view of Theorem A.1
Σ (1--n0)) 1-- -1) nw + +322|
12
GG (B) - x012 x112 3
| | +3
(33)
Thus,
W
(B) 2x113
W-1
W-
< 3
Σ w=0
(1 110 )) 20+2w
+3 3 Σ
Σ
(1- wtw -1) (w/n)² + +3
(34)
w=0 i=1
(1 no??
21
(1 - - (1 )
+3W Σ (I-nof) 1) nw 3W llyl?
n
< 3|
(35)
1- (1 - noo2)
< <3|00 - - no2z + 3W Σ (11 no?) -1) 2 (w.n)² + 3W ll
(1 - ?0,)
-1
(36)
completing the proof.
A.4 ES-EMV algorithm
The exponential moving variance version of our method is summarized in Algorithm 2
A.5
More details on major DIP variants
Deep Decoder (Heckel & Hand 2019) differs from DIP mainly in terms of network architecture: It is typically under-parameterized network consisting mainly of 1 x 1 convolutions, upsampling, ReLU and whae normalization layers, while DIP uses an over-porrumelerized, U-net like convolutional network.
GP-DIP
CCeng et al. 2019) uses the original DIP (Uyanov et al. 2018) network and formulation, but replaces stochastic gradient descent (SGD) by stochastic gradient Langevin dynamics (SGLD) in the gradient update step. ie., for the generic gradient step for optimizing Eq. (2) reads:
0 0 (V ((o((z)) + +A(GG(())) +
(37)
where 7) is zero-mean Gaussian with isotropie variance level t.
23

---

Published in Transactions on Machine Learning Research (12/2023)
Algorithm 2 DIP with ES EMV
Input: random seed 2, counter k: = 0, EMA = 0, FMV" 0, FMV min 00
Go, forgetting factor (0,1), 1), patience number P, iteration
Output: reconstruction x²
1: while not stopped do
2: update 0 via Eq. (2) to obtain 0 and =
EMA 8+1 (1 a EMA* + #+1
EMV (1 ax) EVM + arl - a) a))l EMA
if EMVK+1 EMV min then
EMV min EMVK+1 x f- x ++1 end if
7:
8:
if EMV min stagnales for P iterations then
stop and return x² 10: end if
9:
11: k le ++1
12: end while
DIP-TV Cascarano et. al.j 2021 uses the original DIP (Ulannov et al. 2018) network, with a Total Variation (TV) regularizer added. Then, the proposed objective is solved with the Alternating Direction Method of Multipliers (ADMM) framework.
SIREN Sitzmann et al. 2020g treats the object directly as contimuous function on R² or R$ (or higher- dimensional spaces depending on the application) and hence parameterizes it as a multi-layer perceptron (MLP): 1) the input to SIREN is the 2D/3D coordinate of each pixel instead random values, and 2) the network uses a sinusoidal activation function instead of the commonly used ReLU. When substituting the DIP network with SIREN and solve
Eq. (2) problems, similar overfitting issue is still observed.
A.6 More details major ES methods
Here, provide more details the main competing methods.
Spectral Bias (SB) Shi al. [2022 operates on deep decoder models and proposes two modifications to change the spectral bias: (1) controlling the operator norm of the weight for each convolutional layer by normalization
w
w
(38)
max (1.1/ollop/AA)
ensuring that ||w'llop S X, which in turn controls the Fourier spectrum of the underlying function represented by the layer; (2) performing Gaussian upsampling instead of the typical bilinear upsampling to suppress the smoothness effect of the latter. These two modifications with appropriate parameter setting (A, and in Gaussian filtering) can improve the learning of the high-frequency components by deep decoder, and allow the bluriness-over-shapppee stopping criterion.
Ar(w')
Σ
Σ
(aw--W-wo
(39)
where r(w') = B(x')) (((')) and B-)) and S(-) are the blurriness and sharpness metrics in Crete et al. (2007)
and Bahrami & Kot (014), respectively. In other words, the criterion Eq. (39) measures the change ratios in consceutive windows of size W, and small changes indicate
the average Shurinncss-vvr--saapppe
good ES points. But, as mentioned, this criterion only works for modified DD models and not for other DIP
variants, acknowledged by the authors in Shi al. (0022) and confirmed in our experiment (see Sec. 3.1
24

---

Published in Transactions on Machine Learning Research (12/2023)
DF-STE Jo et al. (2021) targets Gaussian denoising with known noise levels (i.e. y + n, where is the i.i.d. Gaussian noise) and considers the objective.
o² tr (z) 1 ly Goly)l n
min
+ n²
(40)
where Ir Ja. (y) is the trace of the network Jacobian with respect to the input, that is, the divergence term in Jo et al. (2021). The divergence term is proxy for controlling the capacity of the network. The paper then proposes a hcuristic zcro-crossing stopping criterion that stops the itcration when the loss starts to cross zero into negative values. Although the idea works reasonably well on Gaussian denoising with low and known noise level (the variance level o² explicitly needed the regularization parameter ahead of the divergence term), starts break down when the noise level increases even if the right noise level is provided; see Sec. 3.1 Also, although the paper has extended the formulation to handle Poisson noise, it is unclear how to generalize the idea for handling other types of noise, well as how to move beyond simple additive denoising problems.
SV-ES
Li let al.
2021 proposes training an autoencoder online using the reconstruction sequence (w') h2)
min 00,00
Σ CAE('', Dwo o Ev Ex()))
(41)
Any new x' passes through the current autoencoder and the reconstruction error EAR is recorded. They observe that the error curve typically follows U-shaped shape and that the valley of the curve is approx- imately aligned with the peak of the PNSR curve. Therefore, they design an ES method by detecting the valley of the error curve. This method works reasonably well for different IPs and different DIP variants. A major drawback is efficiency: the overhead caused by the online training of the autoencoder is on an order of magnitude higher than the cost of the DIP update itself, as shown in Tab. 3
DOP You et al. (0200) considers only additive sparse noise (e.g,, salt and pepper noise) and proposes modeling the clean image and noise explicitly in the objective:
min yy Go(z) ( ho holl 0.
(42)
where the overparameterized term - ho h (o denotes the Hadamard product) meant capture sparse noise, where similar idea has been shown to be effective for sparse recovery in Vaskevicius et al. (2019). Different. properly tuned learning rates for the clean image and sparse noise terms are necessary for success. The downside includes the prolonged running time, as it pushes the peak reconstruction to the very last iteration, and the difficulty to extend the idca to other types of noisc
A.7 Additional experimental details & results
A.7.1
External codes
DIP: http:p/s//ttaaaaooooe//////////r
Deep decoder: http:///gthhhh.ooo/////haaahaaaa _ere__deee_______ deeeeeddeeeeepeeeeeerrreeeree
DIP-TV: http::///tthaaiccc///ssaaa/////2
GP-DIP: http:://aapgglleee.emeeeeeeeeaar
DF-STE: http:://ghhunnoo///ssssssa//aaad
SV-ES: httpfe///Attttticccooommmmmmaaaa
DOP: http:///tttiiccco/oogooo///o/oor
SB: http:p/shhhhcllliicaaaaaaalalaar
CRSD68: httpe///fthccccc////aamaaaaassas
25

---

Published in Transactions on Machine Learning Research (12/2023)
A.7.2 Experiment Settings
Our default setup for all experiments is as follows. Our DIP model is the original from Ulyanov et al. (0018) the optimizer is ADAM with a learning rate 0.01. For all other models, we use their default architectures, optimizers, and hyperparameters. For ESSWWM, the default window size W = 100, and the patience number P 1000. We use both PSNR and SSIM to access the reconstruction quality and report PSNR and SSIM gaps (the difference between our detected and peak numbers) as an indicator of our detection performance. For most experiments, repeat the experiments 3 times to report the mean and standard deviation; when not, explain why.
Noise generation Following the noise generation rules of Hendrycks & Dictterich (001 we simulate four types of noise and three intensity levels for each type of noise. The detailed information is as follows, Gaussian noise: mcan additive Gaussian noise with variance 0.12, 0.18 and 0.26 for low, medium and high noise levels, respectively; Impulse noise: also known salt-and-pepper noise, replacing each pixel with probability p E [0, in a white or black pixel with half chance each. Low, medium and high noise levels correspond to p = 0.3, 0.5, 0.7, respectively; Speckle noise: for each pixel x E [0, 1], the noisy pixel is x11+ E), where e is zero-mean Gaussian with a variance level 0.20, 0.35, 0.45 for low, modium, and high noise levels, respectively; Shot. noise: also known as Poisson noise. For each pixel, E [0, I], the noisy pixel is Poisson distributed with the rate λz, where X is 25, 12, for low, medium, and high noise levels, rospectively.
Gaussian
Impulse
Shot
Speckle
Low-level noise
232
202
115
823
352
873
090
584
180
2320
1200
1000
200
2000
300
48460
2002 2000
High-level noise
14-
1 2.0
400
300
150
1000
400 500
200
Figure 15: Our ES- WMV method on DIP for denoising "F16' with different noise types and levels (top:
low-level noise; bottom: high-level noise). Red curves are PSNR curves, and brown curves are loss curves.
A.7.3 Denoising examples
We explore the possibility of using the fitting loss for ES here, but we are unable to find correlations between the trend of the loss and that of the PSNR curve, shown in ig. 15
A.7.4 Comparison with baseline methods
To further compare with baseline methods, we report the PSNR gaps in high-level noise cases and the SSTM gaps in low- and high-level noise cases in Fig. Fig. 18 and F Fig. 19 respectively, which show a trend similar to the results of PSNR gaps. The detection gaps of our method are very marginal (< 0.02) for most types and levels of noise (except Baboon and Kodak1 for certain lypes / levels of noise), while the baseline methods
26

---

Published in Transactions on Machine Learning Research (12/2023)
HRISQUIF
NIQE
NIMA-q
NIMA-B
FS-WM
Penk
Figure 16: Visual comparisons of NR-IQMs and ES-WMV. From top to bottom: Gaussian noise (low).
Gaussian noise (high), impulse noise (low), impulse noise (high).
27

---

Published in Transactions on Machine Learning Research (12/2023)
20
NIQU
BRESQUE
25
15-
NIMA
NIMA--
20-
PSNR-GAP
PSN--GAP
15
10-
10-
Honme
Poppecs
Rahaog
Keddek
Kodak3
Kodak 2
Home
Pypper
Lett
Bahos
F16
Kodad
Kodad
Kodad
Kedak
Gaussian noise
Impulee noise
25
25
20
20
PSNR-GAI
15
PSNR-GAP
15
10-
10-
Rome
Popport
Babova
Fi6e
Kudbal
Kodak
Kedda
Kodal1
Hnaed
Popeon
Lesa
Babkng
Fi6
Kodakl
Kodak
Kodak3
Kodakf2
Shot
Speokle noise
Figure 17: High-level noise detection performance in terms of PSNR gaps. For NIMA, report both technical quality assessment (NIMA-q) and aesthetic assessment (NIMA-a). Smaller PSNR gaps are better.
0.8
0.9
0.7
NIMA
0.6
NIMA-I
0.7
Oura
0.6
5SIM-GAP
0.5
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.0
Home
Popper
Lone
Baleom
Kedad/
Kerld2
Kodaa
Kodak12
Hone
Poppers
Lone
Balnam
K
Kodak)
Kodak
Gauasiaa noise
Imgulse noise
0.8
0.6
SSIM-GAP
0.6
SSIM-GAP
0.5
0.4
0.4
0.3
0.2
0.2
0.1
0.1
0.0
0.0
Honse
Popper
Bahooa
F16
Kodak/
K
K
Kodal12
House
Poypers
Lone
Hahoom
P16
Keddäi
2
Kedaet)
Kodek12
Shoe
Speokle
maise
Figure 18: Low-level noise detection performance terms of SSTM gaps. For NIMA, we report both technical quality assessment (NIMA-q) and acsthetic asscssment (NIMA-a). Smaller SSIM gaps are better.
28

---

Published in Transactions on Machine Learning Research (12/2023)
0.9
DRISQUE NEMAN
NIQH
NIMA
0.8
0.6
0.6
SSIM-GAP
0.5
SSI44AP
0.5
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.0
0.0
Hoood
Poppons
Lona
Bulsona
F16
Ecdadd
Eosial2
Kedak)
Kodal
Houss
Poppor
Losa
Rthose
Pio
Kndakl
Kaditt
Krudek
Kisdad
Ganseion noise
Imprilse Roise
0.8
0.7
0.7
0.6
0.5
SSIM-GAP
SSI44AAP
0.4
0.4
0.3
0.3
0.2
0.2
0.1
0.1
0.0
0.0
Haue
Poppers
Low
Baborio
Pf6
Kodak/
Kadac
Kidaks
Kudal 22
Hour
Poppers
Low
Fio
Kndak
Kodak?
Krdsk
Kodak
Shoe noise
Speskle make
Figure 19: High-level noise detection performance in terms of SSTM gaps. For NIMA, we report both technical quality asscssment (NIMA-q) and acsthetic asscssment (NIMA-a). Smaller SSIM gaps are better.
can easily exceed 0.1 for most cases. In addition, we provide some visual detection results in Figs. 7 and 16 Our ES significantly outperforms the four baseline methods visually.
A.7.5 Comparison with competing methods
Comparison between ES-WMV with DF-STE for Gaussian and shot noise on the 9 image dataset in terms of Furthermore, we also test our ES- WMV and DF-STE on CBSD68 in [Tb. 7 Our ES- WMV wins in high-level noise cases but lags behind DF-STE in the low-level cases. The gaps between our ES-WMV and DF-STE for all noise levels mostly come from the peak performance between the original DIP and DF-STE modifications in DF-STE have affected peak performance, positively for low-level cases and negatively for high-level cases, not much from our ES method, as evident from the uniformly small detection gaps reported in Tab. 7 Moreover, DF-STE only handle Gaussian and Poisson noise for denoising, and the exact noise level is required hyperparameter for their method to work.
SSIM is reported in Fig. 20
Then we compare our ES- WMV and SV-ES in Fig. 21 The DIP results with ES-WMM versus DOP in impulse noise are shown in Tab. 8 For SB, part of the qualitative detection results on the images are reported in Fig. 22.
For reference, we compare DIP with the recent one-shot methods based diffusion models for solving linear Dr--DDM++ for image denoising, shown in Tab. 9 Like for Tab. 5 we observe that (1) Our ES-WML
is again able to detect near-peak performance for most images: the average PSNR gap is < 1.02 and the average SSIM gap is < 0.08; (2) DDNM+ is sensitive to the noise type and level: from Tab. 9, DDNM+ outperforms DIP and DIP P.P+ES-WWIT WMV when there is Gaussian noise, but this is when the noise level set for pretraining DDNM+ matches the true noise level, i.e, σ, = 0.18, which is unrealistic in practice as the noise level is not known beforehand. When the noise level is not set correctly, e.g., as oy = 0 in the DDNM+ (a,, .00) row of Tab. 9 the performance of DDNM+ is much worse than that of DIP and DIP+ES-WMY Also, for impluse noise denoising, DIP is also clear winner that leads DDNM+
http:://wwee cs. tut. fit.ffi/ffofffe BM3D/indee wtt.idewwt.dddeeeeeeeeeeeee wte.euuee results
29

---

Published in Transactions on Machine Learning Research (12/2023)
DF-STF (le))
DESTE (ligh)
-th -Ous (los)
Ours (ligh)
0.8
SSIM
SSIM
0.6
0.4
0.4
0.2
0.2
Houme
Popper
Lena
Baborin
File
Kadak/
Kada?
Kodak.
Kiadld
Hewer
Popporo
Lowa
Babona
FiG
Kodik
Kedaki
Kolsk
Cauasian noise
Figure 20: Comparison of DF-STE and ES- MV for Gaussian and shot noise in terms of SSIM.
Curs Lime)
SV-FS (low)
4.0
4.0
Ours (hih)
3.5-
3.0
3.0
PSVRGAP
2.5
2.5
2.0
20
1.5
1.5
1.0
0.5
0.5
0.0
Howe
Popyers
Lone
Ralsom
Fi6
Koolal
Kedld?
Radikk2
Home
Popper
Hideas
F16
Kedaki
Kodak2
Kodik33
Rishik12
Gazzesion noise
Impulse noise
40
5.0
3.5
4.5
3.5
2.5
2.0
1.0
1.0
0.5
0.5
0.0
Heusr
Poppers
Babbote
F16
Kiddal
Kodak?
Kodakd
Heue
Poppera
Bahabe
F16
Kedaki
Endal2
Kedak
Kodal
Shoe noise
Speckle noise
Figure 21: Low- and high-level noise detection performance of SV-ES and ours terms of PSNR gaps.
show that DDNM+ may also suffer from the overfitting issue and our ES-WMV help DDNM+ to stop around the performance peak as well.
by large margin; and (3) in Appendix A.8
Table 7: Comparison between ES- CS- and DF-STE for image denoising on the CBSD68 dataset with varying noise level σ: mean and (std). PSNR gaps below 1.0 colored red.
a= 15
50
ES-WMV
28.7(0.2)
27.4(2.0)
24.2(9.3)
DIP (Peak)
29.7(3.0)
28.012.4)
24.9(2.3)
PSNR Gap
1.0(0.7)
0.7(0.5)
0.7(0.5)
DF-STE
31.4(18)
28.4(2.a)
(1.1(2.5)
30

---

Published in Transactions on Machine Learning Research (12/2023)
Table DIP with ES- WMV vs. DOP on impulse noise: mean and (std).
Low Level
High Level
PSNR
SSIM
PSNR
SSIM
DIP-ES
31.64 (5.00)
0.85 (0.18)
24.74 (3.29)
0.67 (0.19)
32.12 (4:32)
(0-07)
27.34 (3.78)
(0.10)
Peppers
F16
Lena
35
35
35
PSNR
20
20
15
S6
15
15
ES
10
01
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Optmizatton Iteration
Optimization Iteration
Optimization Iteration
35
35
30
30
30
20
20
15
15
15
0
2000
4000
6000
8000
10000
%
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Optmization Iteration
Optimization Iteration
Optiizztion Iteration
35
35
30
30
25-
25
15
15
10
10
0
2000
4000
6000
8000
10000
0
2000
4000
8000
8000
10000
0
2000
4000
6000
8000
10000
Optmiization Iteration
Optimization Iteration
Opžimization Iteration
Figure 22:
Comparison between WM- and SB for image denoising (top:
15; middle: o = 25:
bottom: o = 50). The red and blue curves are the PNSR and the ratio metric curves. The orange and green bars indicate the ES points detectod by our ES- and SB, respectively.
Table 9: Comparison of ES- WS- for DIP and DDNM+ Wang et al. (2022) for denoising images with medium-level Gaussian and impulse noise: mean and (std). The highest PSNR and SSIM for each task are in red. In particular, we set the best hyperparameter for DDNM+ (a, = 0.18), which is unfair for the DIP + ES-WMV combination as we its hyperparameter setting.
PSNR
SSIM
Gaussian
Impulse
Gaussian
Impulse
DIP (peak) DIP + EV-WWV
24.03 (2.06) 37.75 (3.32) 0.68 (0.06) 0.96 (0.10)
23.61 (2.07) 36.87 (4.29) 0.60 (0.18)
0.96 (0.10)
DDNM+ (a, .18)
26.93 (2:25)
22.29 (3.00)
0.78 (0.07)
0.62 (0:12)
DDNM+ (a,, .00)
15.06 (0. 39)
15.52 (0.48)
0.25 (0.10) 0.30 (0.10)
31

---

Published in Transactions on Machine Learning Research (12/2023)
A.7.6 ES-WMV as helper
Performance of ES- WMV on DD, GP-DIP, DIP-TV, and SIREN for Gaussian denoising in terms of SSIM gaps (see Fig. 24
1.0
0.8
SSIM
SSIM
0.6
0.4
0.4
(ighh (light
(ligh
(low)
(lon)
0.2
(lo)
0.2
Oua (low)
House
Popres
Lewa
Fi6
Kediad
Kedikl
Kodaki
Home
Prppors
Lanal
Bahont
Krolal
Kisdal
Eodik
Gomasian toise
Impelse
Figure 23: Comparison VAL and ES- WMV for Gaussian and impulse noise in terms SSIM.
0.35
0.35
0.30
0.30
0.25
0.25
SSIM-GAP
0.20
SSIM-OAP
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
x
Rome fleeeitt
Low Romes
om
Rabowrlt)
Rabser (f))
Fiwidy
Koial Muddiid
di
Bosad Kunkal
Fiddal (2
cm
Roow
(BD
Lia Leaiing
Ditove (10)
Rikd Fm6
(10)
Deep Decnder
GP-DI
0.25
SSIM-GAP
0.20
SSMGAP
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
Honu Piprec Repovitt
Low
Lem
lif
ricint
Arslal 1911
Eistild
Austal
Rusatk (15
Ross Home dil Piypon Prypactl
Lew
Fil
Fastit Kealll (10
(i)) Kreikk
DIP.
(d) SIREN
Figure 24: Performance of ES-WMV on DD, GP-DIP, DIP-TV, and SIREN for Gaussian denoising in terms
of SSIM gaps. L: low noise level: H: high noise level
A.7.7 Performance on real-world denoising
We randomly sample 1024 images from the RGB track of the NTIRE 2020 Real Image Denoising Chal- lenge (Abdelhamed ct al.] 2020), and perform DIP-based image denoising. Histograms of PSNR and SSIM gaps are shown in Fig. 25 For DIP with the three different losses, there are only 4.79%, 4.69% and 4.40% images, respectively, whose PSNR gaps are larger than 2dB. As stated from the beginning, ES- WMM is designed with real-world IPs, targeting unknown noise types and levels. Given the encouraging performance above, we test on common real-world denoising dataset- dataset-PolyU Dataset Xu et al. (2018). which contains 100 cropped regions of 512 x 512 from 40 scenes. The results reported in Tab. 10 We do not repeat
32

---

Published in Transactions on Machine Learning Research (12/2023)
300
130
1300
802
200
302
0.50
0.0
6.5
0.0
625
0.30
5585-GAP
Figure 25: Image denoising of DIP ES- on the RGB track of the NTIRE 2020 Real Image Denoising Challenge. Top row: histograms of PSNR gaps for DIP (MSE), DIP (G) and DIP (Huber), respectively; bottom row: histograms of SSIM gaps for DIP (MSE), DIP (61 and DIP (Huber), respectively.
Table 10: DIP with ES- WM- on real image denoising on the PolyU Dataset: mean and (std). (D: Detected)
PSNR(D)
PSNR Gap
SSIM(D)
SSIM Gap
DIP (MSE)
36.83 (3.07)
1.26 (1.28)
0.98 (0.02)
0.01 (0.01)
DIP (E.)
36.20 (2:81)
1.64 (1.58)
0.97 (0.02)
0.01 (0.01)
DIP (Huber) 36.76 (2.00)
1.28 (1.00) 0.98 (0.02)
0.01 (0.01)
the experiments here; the means and standard deviations are obtained over the 100 images of the PolyU dataset. On average, our detection gaps are S 1.64 in PSNR and < 0.01 in SSIM for this dataset across various losses. The absolute PNSR and SSTM detected are surprisingly high.
A.7.8 Image Inpainting
Noisy Image
Our ES-WMV Detecned Image
Peak PSNR Image
Overfitting Image
Figure 26: Visual detection performance of ES-WMV on image inpainting.
33

---

Published in Transactions on Machine Learning Research (12/2023)
In this task, clean image To E 10. is contaminated by additive Gaussian noise E, and then only partially observed to yield the observation y = (æg + +) ④ m, where m E {0, [0,1)8xM is binary mask and @ denotes the Hadamard product. Given y and m, the goal is to reconstruct 2o- We consider the formulation reparametrized by DIP, where Go is a trainable DNN paramotrized by and x is a frozen random seed:
(()) = ([(Go(z) - y) C ml
(43)
Mask m is generated according to i.i.d. Bernoulli model with rate of 50%, i.e., half of pixels not observed in expectation. The noise is set to the medium level, i.e. additive Gaussian with 0 mean and 0.18
variance. We test our ES-WMV for DIP on the inpainting dataset used in the original DIP paper Ulyanov
et al. (2018). PSNR gaps are < 1.00 and SSIM gaps are I 0.05 for most cases (see Tab. 11).
We also
visualize two examples in Fig. 26
Table 11: Detection performance of DIP with ES- WMV for image impainting: mean and (std). PSNR gaps below 1.00 are colored as red; SSTM gaps below 0.05 are colored as blue. (D: Detected)
PS
PSNR Gap
(S)M(D)
SSIM Gap
Barbara
21.59 (0.03)
0.20 (0.03)
0.67 (0.00)
0.00 (0.00)
Boat
21.91 (0.10)
1.16 (0.18)
0.68 (0.00)
0.03 (0.01)
House
27.95 (0.33)
0.48 (0.10)
0.89 (0.01)
0.01 (0.00)
Lena
21.71 (0.30)
0.37 (0.18)
0.80 (0.00)
0.01 (0.00)
Peppers
25.86 (0.22)
0.23 (0.05)
0.84 (0.01)
0.02 (9.00)
C.man
25.26 (0.00)
0.23 (0.14)
0.82 (0.00)
0.01 (0.00)
Couple
21.40 (0.44)
1.21 (0:53)
0.63 (0.01)
0.04 (0.02)
Finger
20.87 (0.04)
0.21 (0.17)
0.77 (0.00)
0.01 (0.01)
Hill
23.54 (0.06)
0.25 (0.11)
0.70 (0.00)
0.00 (0.00)
22.92 (0.25)
0.46 (0.113
0.70 (0.01)
0.01 (0.00)
Montage
26.16 (0.38)
0.38 (0.26)
0.86 (0.01)
0.03 (0.01)
A.7.9 Image Super-resolution
Visual comparisons for 2x image super-rcsolution task with additional low-level Gaussian noise and impulsc noise are shown in
F Figs. 27 and 28 respectively.
A.7.10 RAW images demosaicing and denoising
RAW images demosaicing and demoising are two essential procedures for modern digital cameras to produce high-quality full-color images [Li et al. (2023a)) ). Given a noisy RAW image @lch lcch + n, where H and W are the height and width of the image and n is the noise, the goal is to obtain high quality full-color image Sch Hx33 from it. To achieve this goal, it is obvious that we need to fill in the missing pixels (demosaicing) and remove the noisy components (denoising). In this section, we formulate this problem image-inpainting problem as Li et al. (2023a) and adopt DIP to reconstruct the desired full-color image, In addition, we plug our early stopping method into DIP and explore the effectiveness of our method on this low-level vision task. We conduct experiments on the Kodak dataser and prepare it following the pipeline in Li et al. (2023a). We experiment with Poisson noise (A = 25; a detailed description of the noise intensity could be found in Li el al. (2023a)) ), which is very common noise under low light conditions. We report the experimental results in Tab. 12] It is evident that our method could effectively detect the near-peak points and produce reliable carly stopping signals for DIP.
A.7.11 MRI reconstruction
httpe::/hokk.uu//rrppttt///eta
34

---

Published in Transactions on Machine Learning Research (12/2023)
DIP Peak
DIP+ES-WMV
DIP (Overfitting)
DDNM+ (oy = .12)
DDNM+ (a, = .00)
PUWEI YUll
rowerruml
How Everything
rowerruin
FUwerrumm
Everyihepp
2002
Evicyshang
2002
Evsrytrin
2002
2002
Everyshingg
2002
Figure 27: Visual comparisons for 2x image super-resolution task with additional low-level Gaussian noise.
DIP Peak
DIP+ES-WMV
DIP (Overfitting)
DDNM+ (a, = .12)
DDNM+ (a, = .00)
Figure 28: Visual comparisons for 2x image super-resolution task with additional low-level Impulse noise.
35

---

Published in Transactions on Machine Learning Research (12/2023)
0.90
0.85
0.80
SSIM
0.75
Pank (CI)
0.70
ES (C1)
Peak (C2)
ES (C2)
0.65
Conv Decader
0.60
0
1000
2000
3000
4000
Iteration
Figure 29: Detection on MRI reconstruction
We visualize the performance on two random cases (C1:
Table 12: RAW images demosaicing and denois- ing on the Kodak dataset for 25 cases: mean and (std). (D: Detected)
1001339 and C2: 1000190 sampled from Darestani Heckel (2021) part of the fastMRI datatset (Zbontar
et al. 2018) in Fig. 29 (quality measured in SSTM, con-
sistent with Darestani & Heckel (2021)
PSNR(D) PSNR Gap SSIM(D) SSIM Gap
24.22 (2.49) 0.92 (0.87) 0.58 (0.14) 0.06 (0.06)
A.7.12 Blind image deblurring (BID)
In this section, we systematically test our ES- and VAL on the entire standard Levin dataset for both low-level and high-level cases. We set the maximum number of iterations to 10, 000 to ensure sufficient optimization. The detected images of our ES-WMV are iall better than those of VAL, as shown in Tab. 13
Table 13: BID detection comparison between EM-WMV and VAL the Levin dataset for both low-level and high-level noise: mean and Higher PSNR is in red and higher SSIM is in blue. (D: Detected)
High
PSNR(D)
SSIM(D)
PSNR(D)
SSIM(D)
WMV
28.54(0.01)
0.83(0.0))
(6.41)
0.76çg. 04)
VAL
18.87(1.44)
0.50ro.09)
16.69(1.39)
0.44(o.10)
A.7.13 ES-WMV vs. ES-EMV
We now consider our memory-efficient version (ES-EMV) as described in Algorithm 2 and compare it with ES- AS-,WV, as shown in Fig. 30 Besides the memory benefit, ES-EMV runs around 100 times faster than
ES- WM-, as reported in TTb. 3 aad does seem to provide consistent improvement on the detected PSNRs
for image denoising tasks on NTIRE 2020 Real Image Denoising Challenge (Abdelhamed ct al. 2020 PolyU
dataset Xu el al. (2018) and the classic 9-image dataset (Dabov el al. 22008) (see Tabs. 14[and 15 and Fig. 30)
due the strong smoothing effect (we set a 0.1). In this paper, we prefer to keep it simple and leave systematic evaluations of these variants for future work.
A.7.14 Ablation study
We also notice that smaller learning rates can smooth out the VAR curves and mitigate the multi-valley phenomenon Fig. 33 Therefore, we apply our ES-WMV deep decoder and GP-DIP with smaller learning rates (both 0.001), shown in Fig. 31 Compared to the results of deep decoder and GP-DIP with the default. learning rates in Fig. 10 most of the PSNR gaps decrease.
36

---

Published in Transactions on Machine Learning Research (12/2023)
36
ES-WMV ES-EMV
30
28
22
20
Howse.))
Popper How (f)
Popoon (11
(t)
(t.
Lowe (ff)
Beboon
lnsban (H))
F16d))
Fis(
Kodakl (t.)
Kodill (00)
Kadit
Kndakt Kodal2 (HI)
(t)
Kodaka (P)
(L)
Kobb11
Figure 30: Detected PSNR comparison between DIP with ES- WM- and DIP with ES-EMV on the classic 9-image dataset (Dabov et al. 2008
Table 14: Detection performance comparison between DIP with ES-WMV and DIP with ES-EMV for real image denoising on 1024 images from the RGB track of NTIRE 2020 Real Image Denoising Challenge Ab-
delhamed et al. 2020): mean and (std). Higher PSNR and SSIM are in red. (D: Detected)
PSNR(D)-WMV
PSNR(D)-EMV
SSIM(D)-WMV
SSIM(D)-EMV
DIP (MSE)
34.04 (3.69)
34.96 (3.80)
0.92 (0.07)
0.93 (0.07)
DIP (6)
33.92 (4.34)
34.83 (4.35)
0.93
0.94 (0.08)
DIP (Huber)
33.72 (3.86)
34.72 (4.05)
0.92 (0.05)
0.93 (0.06)
Table 15: Detection performance comparison between DIP with ES-WMV and DIP with ES-EMV for real image denoising on the PolyU dataset Xu et al. (2018): mean and (std). Higher PSNR and SSIM are in red. (D: Detected)
PSNR(D)-WMV
PSNR(D)-EMV SSIM(D)-WMV
SSIM(D)-EMV
DIP (MSE)
36.83 (8.07)
37.32 (3.82)
0.98 (0.02)
0.98 (0.03)
DIP (4)
36.20 (2.81)
36.43 (3.22)
0.97
0.97 (0.02)
DIP (Huber)
36.76 (2.96)
37.21 (3.19)
0.98 (0.02)
0.98 (0.02)
Deep Decoder
GP-DIP
5.0
5.0
4.5
4.5
3.5
3.5
10-
3.0-
2.5
2.5
2.0
1.5
1.5
10-
1.0
C.5
0.0
<.5
00.5
Hosse.)) (L)
Parpers [k)
(H)
Poyper (H)
Lawell (L)
fibbon (L)
(19)
Babeon Fieil)
Kuneel (L)
(1)
Kodals (10
(1f)
Kodill (T0
Houss (L)
Hoost (T))
Popport (U)
Poppers (1f)
Lma (f)
Batwen (II)
Kraikk (I))
(ff)
Kodial
Kedday
Figure 31: Performance of ES- WMV on deep decoder and GP-DIP with smaller learning rates for Gaussian denoising in terms of PSNR gaps. L: low noise level; H: high noise level.
37

---

Published in Transactions on Machine Learning Research (12/2023)
A.8 Potential of ES-WMV for effective ES in zero-shot super-resolution with diffusion models
Peak
ES-WMV
Default
24
24
20
14-
12
4.000
Csechh
20
14
12
4.002
12
Figure 32: Left: PSNR/VAR curves of DDNM+ for 2x noisy image super-resolution (top: with Gaussian noise at 0.12; boltom: with impulse noise). Right: visualization of the super-resolved results by DIP (PSNR peak), DIP with ES-WMV, and the default DDNM+ The DDNM+ paper Wang et al. (2022) fixes the iteration number as 1000 and stops the reverse diffusson process at the very last iteration by default.
Recently, zcro-shot methods based on diffusion models have been proposed to solve linear image restoration tasks, e.g., DDNM+ Wang et al. 2022 However, these methods usually rely on pre-trained models from large external training datasets, while DIP does not need any training data or pre-trained models. In Fig. 32 we show that DDNM+ can also have overfitting issues similar to those in DIP methods, especially when the observation y is noisy but the noise type and/or level are not correctly specified the diffusion models very likely to happen in practice, When DDNM+ is trained assuming knowing the exact measurement noise type/level is often unrealistic. noise, i.c., oy 0.00, but the downsampled image is contaminated by Gaussian noise at a level σy 0.12, or by impluse noise, there is substantial overfiuting to the noise, is evident from both the PSNR plots (left of Fig. 32) and direct visualization of the super-resolved images (right Fig. 32). Moreover, we observe that our ES-MWV mothod can also help DDNM+ detect near- peak performance! We stress that the experiment here exploratory and preliminary and that tackling the overfitting issue in DDNM+ style methods for solving inverse problems out of the scope of this paper. We leave a complete study for future work.
A.9 Limitations and analysis of failure cases
For limitations, our theoretical justification is only partial, sharing the same difficulty of analyzing DNNs in general; our ES method struggles with images with substantial fighffreyeency components; our detection is sometimes off the peak terms of iteration numbers when helping certain DIP variants, e.g. DIP-TV with low-level Gaussian noise Fig. 4h, but the detected PSNR gap is still small.
Our ES method needs three things to succeed: (1) the U-shape of the VAR curve, (2) the VAR valley aligning with the PSNR peak, and (3) the successful numerical detection of the VAR valley. In this section, we discuss
https://github. hwt.://ywwass////////e///aaaaaaa
38

---

Published in Transactions on Machine Learning Research (12/2023)
Deep Decoder
GP-DIP
2.000
3,00
3,007
1000
30.0
4000
5000
5000
1000
2502
3000
7500
10006
1250R
35000
373.0
27.5
Nertam
P3RK
0.003
33.2
15
1000
2000
3000
40.00 3000
6000
1900
3000
2500 3000
1300
20000
32500
13000
17500
20000
Figure 33: Top left: DD with the default learning rate for "Lena(L)"; top right: GP-DIP with the default learning rate for "House(L)") bottom left: DD with learning rate 0.001 for "Lena(L)"; bottom right: GP. DIP with learning rate 0.001 for "House(L)".
0.005
0.004
Virionce
0.002
1000
2000
3000
4000
5000
6000
7000
8000
Iteration
Figure 34: Our ES- DV method on DIP for denoising "Baboon" with low-level Gaussian noise. Left: clean
"Baboon': right: the denoising process.
39

---

Published in Transactions on Machine Learning Research (12/2023)
two major failure modes of our ES method: (I) the VAR valley aligns well with the PSNR peak, but the U-shape assumption is violated. A dominant, pattern is the presence of multiple valleys, see, e.g., the top row of Fig. 33 that shows such examples with DIP variants, DD and GP-DIP (we do not observe the multi-valley phenomenon on DIP itself in Fig. 3). Since our numerical valley detection method aims to locate the first major valley, may not locate the deepest valley among the multiple valleys. Fortunately, for these cases, we observe that using smaller learning rates can help to smooth out their curves and mitigate the multi-valley phenomenon, leading to much smaller detection gaps (see the bottom row of Fig. 33/; (II) the VAR valley does not align well with the PSNR peak, which often happens components, e.g., Fig. 34 We suspect that this is because the initial VAR decrease tends to correlate with the early learning of low-frequency components in DIP. When there are substantial high-frequency components in an image, the PSNR curve takes more time to pick up the high-frequency components, after the VAR curve already reaches the first major valley; hence, the misalignment between the VAR valley and the PSNR peak occurs. In addition, our ES-WMV can fail for images with substantial high-frequency components, e.g. Fig. 34
images with significant high-frequency
40


Reviews:
[{"review_id": "riuURfzAGP", "reviewer": "Reviewer_AVF5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper proposes a simple and effective strategy for how to predict the early stopping point for deep image prior models.\n- The implementation is likely going to be very straightforward, given how simple the idea and the method is. This means great reproducibility.\n- The results at face value, they look promising.\n- The writing is very clear and pleasant. I congratulate the authors.\n\nWeaknesses\n\n- I believe the idea is deceptively simple, and when I read something like that, I automatically become suspicious. At the very least extensive empirical justification is required for such extraordinary claims. I am not sure I would call this a weakness, but I do expect very strong empirical evidence and a very clear section about limitations. This means that for the conclusions to remain, there evidence must be very strong. Please do not take this comment as in 'We did experiments on 7 datasets, what else does he want'. My point is that strong experimental backing is ever more needed given that the paper make a general claim about a general algorithm, so the application domain is very broad.  \n\n- I believe the main weak point for this method is that the devil is hidden in the details. Clearly any such 'change detection' algorithm for the loss curve highly depends on the window with which change is defined, measured, and compared, both temporally, spatially, within batch and between batches. For instance, for a very small-time window quite likely the changes are going to be quite abrupt and thus any simple thresholding test would likely be easily violated.\n\nIn other words, the whole algorithm rests on the assumption that we have a nice clean U curve. However, how often or how easy is this to happen, especially for unknown and hard inverse problems? If anything, since the optimization happens with batch stochastic gradient descent, the fluctuations in the batch alone would be sufficient to make the algorithm very sensitive, especially early on. Of course, I suppose the claim is that with more and more training, the fluctuations would be smaller. However, that is only a conjecture, which is a not a very solid one especially when considering unknown inverse problems.\n\n- This brings me to the next point. It is actually quite easy to make sure variance does not fluctuate: set the weights to a constant (like 0), that is likely to give basically 0 variance. Of course, I suppose the claim is that there is the reconstruction loss and setting the weights to 0 is going to return nonsensical reconstructions. But then there must be one more hyperparameter to tune.\n\n-  More generally, I think it is crucial to have very clearly laid out the experimental details per experiment. What are the different hyperparameters, are they different per experiment? If yes, why and how is it selected?\n\n- Also, what I think it is important to try on harder inverse problems, because -although it is a bit outside my area of expertise- I believe the data and the experiments presented are not that challenging. Example links:\n\n  + https://github.com/topics/inverse-problem\n  + https://github.com/mughanibu/Deep-Learning-for-Inverse-Problems\n\nfocusing on more physical settings (like EEG/MEG, Temporal laser-pulse-shape, Spectral Line Observations) and not as much on denoising/superresolution.\n\n- I do see the point of adding the theoretical analysis and derive the bound. However, in practice the bound looks quite loose in figure 5. Also, even theoretically, if we set eta=1/max(sigma_i) (so an oracle removes the inequality, assuming this would lead to the tighest case), the gap still looks quite large: it should be something like that the first term in equation 7 cancels out and then we are left with a sum over the number of parameters (I think that is what n stands for)  for n*w plus the square norm of the observations. So in fact, the bound becomes even more loser as the inputs and the models become larger and thus the problem gets harder? Is that correct?\n\n- On a similar line, what if the noise is not structure, and not even homogeneous. Then the basic assumptions also would not hold, would they?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on providing a variance-based heuristic for Deep Image Prior for inverse problem.\n\nThe setting: Deep Image Priors do well in solving inverse problems, provided the model is stopped at the right moment during the optimization; if not, the model continues to overfit the noise rather than just reconstructing the true signal.\n\nThe key idea: the paper postulates that tracking the variance over the optimization and checking when it plateaus is a good heuristic for finding out when the right moment for early stopping is.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above. Also, some points to further consider to clarify:\n\n- In figure 13, please add all different hyperapeters is the ablation study, eg, size of mini-batch, learning step eta, model architecture, etc. Importantly, for this paper the ablations might have to be not independently but simultaneously, at least with pairs of important hyperparameters, because the whole thresholding test relies on careful hyperparameter combinations. Unless the authors can show that the algorithm is really robust against any choice. In that case, however, I would be very curious to understand why would that be the case, since one cannot know in advance for an unknown problem what are the right model choices etc. Would that even mean that the hyperparameter is unimportant, if it has no influence on the performance. Further, in figure 3 I would expand the range of values, starting from small patiences, say 1, 10, 50, 100, 250 in a log scale, and same for the window size.\n\n- I think the term 'gap' in page 2 is not clearly explained, although intuitively I understood what is meant (the gap from the best possible peak). Maybe rewrite that bit in the bottom of page 2.\n\n- Generally, there is an abuse of acronyms making it very hard and uncomfortable to follow the text without constantly going back and forth.\n\nGenerally, I think the paper is an honest attempt, and I will not suggest rejection but I will give the benefit of the doubt. However, I do expect overwhelming evidence that the algorithm works well, simply because the algorithm is very simple and it means it must then be broadly applicable.", "broader_impact_concerns": "See above.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9nwfIdMICX", "reviewer": "Reviewer_noei", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n* The method is simple and computationally efficient to incorporate into existing DIP pipelines.\n* The method is general enough to apply to a variety of tasks. \n\nWeakness:\n\n* While the method section is clearly written, the experiment section is less organized. Specifically, the comparison to different ES baselines is broken into individual sections. It would be much better if a summary table/figure was provided for a clean comparison across all ES methods. \n\n* The proposed method generally finds good stopping iteration for DIP but is upper-bounded by the peak performance of DIP itself. In several cases, DIP underperforms task-specific methods. \n\n\nMinor:\n* In section 3.2 and table 5, DDNM+ and DDMN+ seem to be used interchangeably in several places.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a general early stopping (ES) criterion for Deep Image Prior (DIP) methods based on measuring the running variance of past history. The paper demonstrated promising performance across various tasks including denoising, super-resolution, and blind image deblurring.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Please correct typos pointed out in the previous section.\n* Could the authors provide a unified table comparing different methods specifically for the ES baselines.", "broader_impact_concerns": "The reviewer did not find any ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "74HXQMTa34", "reviewer": "Reviewer_NtQj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The target problem of this work is novel, clear and significant.\n    \n- The paper is well-written with nice figures and a clear structure.\n    \n- The motivation of the proposed method is simple and clearly presented.\n    \n\nWeaknesses\n\nThe authors have stated the weaknesses of the paper in Section 4 quite comprehensively. I think the major issue is that the performance is poor for some certain situations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "As a promising and influential method, Deep Image Prior (DIP) does not know when to stop the training process automatically to get near-peak performance. This paper deals with this issue by proposing an early stopping strategy based on the running variance of DIP intermediate reconstruction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe the paper would be better if the authors could give a comparably detailed description of the compared methods, so a wider audience would understand the comparisons better.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a straightforward yet innovative early-stopping heuristic for Deep Image Prior (DIP). The reviewers have been thoroughly persuaded by the revisions, as well as the authors' responses and clarifications. The AE recognized the paper's contribution, ultimately recommending its acceptance.
Decision: Accept (poster)