Early Stopping for Deep Image Prior
Authors: Hengkang Wang
Taihui Li
Zhong Zhuang
Tiancong Chen
Hengyue Liang
Ju Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: early stopping
deep image prior
deep generative models
overparametrization
overfitting

Abstract: Deep image prior (DIP) and its variants have shown remarkable potential for solving inverse problems in computational imaging (CI), needing no separate training data. Practical DIP models are often substantially overparameterized. During the learning process, these models learn the desired visual content first and then pick up the potential modeling and observational noise, i.e., overfitting. Thus, the practicality of DIP hinges on early stopping (ES) that captures the transition period. In this regard, the majority of prior DIP works for CI tasks only demonstrate the potential of the models---reporting the peak performance against the groundtruth but providing no clue about how to operationally obtain near-peak performance without access to the groundtruth. In this paper, we set to break this practicality barrier of DIP, and propose an efficient ES strategy that consistently detects near-peak performance across several CI tasks and DIP variants. Simply based on the running variance of DIP intermediate reconstructions, our ES method not only outpaces the existing ones---which only work in very narrow regimes, but also remains effective when combined with methods that try to mitigate overfitting.

Under review as a conference paper at ICLR 2023
EARLY STOPPING FOR DEEP IMAGE PRIOR
Anonymous authors
Paper under double-blind review
ABSTRACT
Deep image prior (DIP) and its variants have shown remarkable potential for solv- ing inverse problems in computational imaging (CI), needing no separate training data. Practical DIP models are often substantially overparameterized. During the learning process, these models first learn the desired visual content and then pick up the potential modeling and observational noise, i.c., overfitting. Thus, the practicality of DIP hinges early stopping (ES) that can capture the transition period. In this regard, most previous DIP works for CI tasks only demonstrate the the peak performance against the groundtruth
potential of the models but providing no clue about how to operationally obtain near-peak performance without access the groundrruth. In this paper, set break this practical- ity barrier of DIP, and propose an efficient ES strategy that consistently detects near wear-peak performance across several CI tasks and DIP variants. Simply based on the running variance of DIP intermediate reconstructions, our ES method not only outpaces the existing ones-which only work in very narrow regimes, but also remains effective when combined with methods that try to mitigate overfitting.
INTRODUCTION
Inverse problems (IPs) are prevalent in computational imaging (CI), ranging from basic image de- noising, super-resolution, and deblurring, to advanced 3D reconstruction and major tasks scien- tific and medical imaging (Szeliski) (022). Despite the disparate settings, all these problems take the form of recovering a visual object x from = f(x), where ʃ models the forward process to obtain the observation y. Typically, these visual IPs are underdetermined: ær cannot be uniquely determined from y. This is exacerbated by potential modeling (c.g., linear ff to approximate a non- linear process) and observational (e.g., Gaussian or shot) noise, i.c., y == fxx) To overcome the nonuniqueness and improve noise stability, people often encode a variety of problem-specific priors on æ when formulating IPs. Traditionally, IPs are phrased as regularized data-fitting problems:
min E fx)) + XR(x)
E(y. f(x) data-fitting loss, R(x) regularizer
(1)
where À is the regularization parameter. Here, the loss is often chosen according to the noise model, and the regularizer R encodes priors on æ. The advent of deep learning (DL) has revolutionized how IPs are solved: on the radical side, deep neural networks (DNNs) are trained to directly map any given y to an æ; on the mild side, pretrained or trainable DL models are taken to replace certain nonlinear mappings in numerical algorithms for solving Eg. (e.g. plug-and-play, and algorithm unrolling). Recent surveys Ongie et al. (2020); Danai et al. (2020) on these developments trust large training sets {(), ax))} to adequately represent the underlying priors and/or noise distributions. This paper concerns another family of striking ideas that require no separate training data.
Deep image prior (DIP) Ulyanov et al. (2018) proposes parameterizing
Go where
Go is a trainable DNN parametrized by and z is a trainable or frozen random seed. No separate training data other than y used! Putting the reparametrization into Eq. (1) obtain
min E(y, f Go(z)) + XR Go(z). 6
(2)
Go is often "overparamettrizee"" comtrmmmttrrreedddccctiiinnnn "structured" e.g., consisting of convolution networks encode structural priors in natural visual objects. The resulting optimization problem solved via standard first-order methods for mod- DL. (e.g., (adaptive) gradient descent). When has multiple components with different physical
containing substantially more parameters than the size of x, and

---

Under review as a conference paper at ICLR 2023
meanings, can naturally parametrize using multiple DNNs.
This simple idea has led to surpris- ingly competitive results on numerous visual IPs, from low-level image denoising, super-resolution, 2019) and blind deconvolu-
inpainting (Ulyanov et al.] 2018, Heckel & Hand 2019 Liu et al.)
tion (Ren et al. 2020| Wang et al. 2019; Asim et al. 2020 Tran et al. 2021 Zhuang et al. 2022a),
to mid-level image decomposition and fusion (Gandelsman et al. 2019] Ma al. 2022 and to
advanced CI problems (Darestani & Heckel [021] Hand et al. 2018) Williams et al. 2019 Yoo
al. 2021 Baguer al. |l.|| [200 Cascarano et al., 2021 Hashimoto & [ Gong
2022
Veen et al.
2018 Tayal et al. 202 Zhuang et al. 2022b): see the survey Qayyum et al. (2021)
Overfitting issue in DIP A critical detail that we have glossed over is overfitting. Since Go substantially overparameterized, Gø(z) can represent arbitrary elements in the domain.
26
24-
PSNR
20-
Global optimization of (2) would normally lead to y fz)) but (z(z) may not repro- duce æ, e.g., when f is non-injective, or y R f(x) that Gz(z) also accounts for the mod- eling and observational noise. Fortunately, DIP models and first-order optimization methods to- gether offer a blessing: in practice, Gz(z) has bias toward the desired visual content and learns it much faster than learning noise. So the reconstruction quality climbs to a peak before potential degradation due to noise; see Fig. This haanl--eammnnntee--verrrrieii phenomenon has been repeatedly reported in prior works and also backed by theories on simple Go and linear / (Heckel Soltanolkotabi 2020b/a) a). The successes of DIP models claimed above are mostly conditioned that appropriate early stopping (ES) around the performance peaks can be made.
PSNR
16
14
12-
10
0
500
1000
1500
2000 2500 Iteration
Figure 1: The "aartylleminngthhooovvvffffiiii (ELTO) phenomenon in DIP for image denois- ing. The quality of the estimated image climbs to a peak first and then plunges once the noise is picked up by the model Go also.
(ELTO)
Is ES for DIP trivial? Natural ideas trying to perform good ES can fail quickly. (1) Visual in- spection: This subjective approach is fine for small-scale tasks involving few problem instances, but quickly becomes infeasible for many scenarios, such (a) large-scale batch processing, (b) recovery of visual contents tricky to be visualized and/or examined by eyes (c.g., 3D or 4D visual objects), and (c) scientific imaging of unfamiliar objects (c.g., MRI imaging of rare tumors, and mi- croscopic imaging of new virus species); (2) Tracking hall-reference/n--reeeeeee
image quality metrics (QM/N:-IQMs): Without the groundtruth æ, computing any FR-IQM and hence tracking their trajectories (e.g., the PNSR curve in Fig. 1) is out of the question. We consider tracking NR- IQMs as family of baseline methods in Sec. 3.1 the performance much worse than ours; (3) Tuning the iteration number: This ad-hoc solution is taken by most previous works. But since the peak iterations of DIP vary considerably across images and tasks (sec, c.g., Figs. 3 and 23 and Ap- pendices A.7.3 and A.7.5). this might entail numerous trial-and-error steps and lead to suboptimal stopping points; (4) Validation-based ES: ES easily reminds us of validation dalidation-based ES in super- vised learning. The DIP approach to IPs summarized in Eq. (2) is not supervised learning, it only deals with a single instance y. without separate (æ, y) pairs as training data. There are recent ideas (Yaman et al., |021 Ding et al. (2022) that hold part of the observation y out as a validation set to emulate validation based ES in supervised learning, but they quickly become problematic for nonlinear IPs due to the significant violation of the underlying iid assumption; see Sec. 3.3
Prior work addressing the overfitting There are three main approaches to countering overfitting in working with DIP models. (1) Regularization: Heckel & Hand (2019) mitigates overfitting by restricting the size of Go to the underparameterizzed regime. Metzler et al. (018); Shi et al. (2022); Jo et al. 2021): Cheng et al. (2019) control the network capacity by regularizing the norms of layerwise weights the network Jacobian. Liu et al. (2019); Mataev al. (2019); Sun (2020);
Cascarano et al. (2021) use additional regularizer(s) (ZG))))) such as the total variation norm or trained denoisers. However, in general, it is difficult choose the right regularization-lew to preserve the peak performance while avoiding overfitting, and the optimal À likely depends on the noise type and level, as shown in Sec. 3.1 the default X's for selected methods in this category
2

---

Under review as a conference paper at ICLR 2023
still lead to overfitting for high-level noise. (2) Noise modeling: You et al. (020) models sparse additive noise as an explicit term in their optimization objective. Jo et al. (2021 designs regularizers and ES criteria specific to Gaussian and shot noise. Ding et al. (2021) explores subgradient methods with diminishing step-size schedules for impulse noise with the & loss, with preliminary success. These methods do not work beyond the noise types and levels they target, whereas our knowledge about the noise in a given visual IP is typically limited. (3) Early stopping (ES): [Shi et al. (2022) tracks the progress based on a ratio of no-reference blurriness and sharpness, but the criterion only works for their modified DIP models, as acknowledged by the authors. Jo et al. (022)) provides noise-specific regularizer and ES criterion, but is unclear how to extend the methods unknown noise types and levels. Li et al. (2021) proposes monitoring the DIP reconstruction by training coupled autoencoder. Although its performance is similar to ours, the extra autoencoder training slows down the whole process dramatically; see [Sec. 3 Yaman et al. (20TT); Ding et al. (2022) emulate validation- based ES in supervised learning by splitting elements of y into training and validation sets that validation-based ES can be performed. But in IPs, especially nonlinear ones (e.g.. in blind image deblurring- BilllrrrngBBDD,,,,. R & I where * is linear convolution), elements of y can be far from being iid and so validation may not work well. Moreover, holding-out part of the observation in can substantially reduce the peak performance; see Sec. 3.3
Our contribution We advocate the ES approach- the iteration process stops once a good ES point is detected, as (1) the regularization and noise modeling approaches, even if effective, often do not improve the peak performance but push until the last iterations; there could be 2 10x more iterations spent than that of climbing to the peak in the original DIP models; (2) both need deep knowledge about the noise type/level, which is practically unknown for most applications. If their key models and hyperparameters are not set appropriately, overfitting probably remains. Then ES still needed. In this paper, we build a novel ES criterion for various DIP models simply by tracking the trend of the rumning variance of the reconstruction sequence. Our ES method is (1) Effective: The gap between our detected and the peak performance, i.c., detection gap, is typically very small, measured by standard visual quality metrics (PSNR and SSIM); (2) Efficient: Per- iteration overhead is a fraction the standard version in Algorithm 1. or negligible- the variant in [lAloritthm 2] relative to the per-iteration cost of Ef.. (2); (3) General: Our method works well for DIP and its variants, including deep decoder (Heckel & Hand 2019 DD) and sinusoidal representa- tion networks (Sitzmann al. 2020 SIREN), on different noisy types/levels and across 5 visual IPs, spanning both linear and nonlinear. Also, our method can be wrapped around several regularization methods, e.g., Gaussian process-DIP (Cheng et al.] 2019 GP-DIP), DIP with total variation regular- ization (Liu et al. 2019; Cascarano et al.) [021] DIP-TV) to perform reasonable ES when they fail to prevent overfitting; (4) Robust: Our method is relatively insensitive to the two hyperparameters,
i.e., window size and patience number (see [Secs. 2 3 and 3.4 and Appendix A.7.13). By contrast,
the hyperparameters of most methods reviewed above are sensitive to the noise type/level.
26
0.08 0.065
PSNR
OUR EARLY-STOPPING METHOD
24-
MSE
0.07
0.060
22
Variance
0.06 0.055
Intuition for our method We assume: x is the unknown groundtruth visual object of size N, (0') ]01) is the iterate sequence, and (#')221 the reconstruction sequence where x' = (z). Since we do not know x, we cannot access the PNSR or any FR-IQM curve. But we observe that (Fig. 2) generally the MSE (resp. PSNR; recall PSNR(=') =
20-
18-
Detection
0.05
0.04
0.050
0.045
0.040
16-
0.03 0.035
14-
0.02 0.030
12.
0.01
0.025
10
0.00
L0.020
0
500
1000 1500 2000 2500 3000
(Iteration)
Figure 2: Relationship between the PSNR, MSE, and VAR curves. Our method relies on the VAR
10 10g11 /Mll///MM((e curve follows a U (resp. bell) shape: ||a. - ==1 initially drops quickly to a low level, and then climbs back due the noise effect, i.e., the ELTO phenomenon in Sec. we hope to detect the valley of this U-shaped MSE curve. Then how to gauge the MSE curve without knowing x? We consider the running variance (VAR):
curve, whose valley is often well aligned with the MSE valley, to detect the MSE valley-that cor- responds to the PSNR peak.
W-1
Σ W-1
VAR()) = 1/W
|| -
1/W
a
(3)
3

---

Under review as a conference paper at ICLR 2023
Initially, the models quickly learn the desired visual content, resulting in a monotonic, rapidly de- creasing MSE curve (see Fig. 2). So we expect the running variance of {x' (z')121 to also drop quickly, as shown in Fig. 2 When the iteration is near the MSE valley, all the x''s are near but scattered around So
and VtR(t)= xs w Ew=o || x1 p. Afterward, the noise effect kicks in and the MSE curve bounces back, leading to a similar bounce-back in the VAR curve as the a' sequence gradually moves away from
-1 x
i=
Table 1: ES-WMV (our method) on real-world image denoising for 1024 images: mean and (std)
This argument suggests a U-shaped VAR curve, and the curve should follow the trend of the MSE curve, with approximately aligned val- leys, which in turn is aligned with the PSNR peak. To quickly verify this, we randomly sample 1024 images from the RGB track of the NTIRE 2020 Real Image Denoising Chal- lenge (Abdelhamed et al., 2020), and perform DIP-based image denoising (i.e., min E(y, (z)) where y denotes the noisy image). Tab. reports the detected PSNR/SSIM and detection gaps based on our ES method (see Algorithm 1) that tries to detect the valley of the VAR curve. On average, the detection gaps are < 0.95 in PSNR and < 0.02 in SSIM, barely noticeable by eyes! More details are in Fig.
over the images.
(D: detected)
(loss)
PSNR (D)
Cap
SSIM (D)
SSIM Gap
MSE
34,04 (368)
0,92 (0.83)
0.92 (0.07)
0.02 (0.04)
81
33,92 (4.34)
0.92 (0.59)
0.93 (0.05)
0.02 (0.02)
33.72 (3.86)
0.95 (0.73)
(0.06)
(0.03)
and Sec. and Appendix A.7.3
Detecting transition by running variance Our lightweight method only involves comput- ing the VAR curve and numerically detecting its valley-the the iteration stops once the val- ley is detected. To obtain the curve, we set window-size parameter W and compute the windowed moving variance (WMV). To ro- bustly detect the valley, we introduce a patience number P to tolerate up to P consecutive steps of variance stagnation. Obviously, the cost is dominated by the variance calculation per step, which is O(WN) (N is the size of the vi- sual object). In comparison, typical gradi- ent update step for solving Eq. (2)ss at least ( 2(0 N), where [0) is the number of parame-
Algorithm DIP with ES-WMV
Input: random seed z, randomly- initialized d window size W, patience P. empty queue Q iteration counter h = 0, VAR = 00
Output: reconstruction x²' I: while not stopped do
2: update O via Eq. (2) to obtain @++1 and
2+*+1
3:
push *+1 to Q, pop queue if Q W if [ = W then
4:
5:
compute VAR of elements in Q via
Eq.
6:
FAR < VAR then VARmin + VAR, x" < xx*+! k+1
7:
8:
end if
ters in the DNN Gg. Since (0) is typically much
9:
if VARmin stagnates for P iterations
larger than W (default: 100), our running VAR and detection incur very little computational overhead. Our whole algorithmic pipeline is summarized in Algorithm To confirm the ef- fectiveness, we provide sample qualitative re- sults in Figs. 3 and [1] with more quantitative results included in the experiment part (Sec. 3}
then
10:
stop and return x²
11:
end
12:
end if
13:
k = +++
14: end while
see also Tab. T}. Appendix A.7.3 shows on image denoising with different noise types/levels, our ES method can detect near-peak ES points. Similarly, our method remains effective on several popular DIP variants, shown in Fig.
Seemingly similar ideas Our running variance and its U-shaped curve are reminiscent of the classical U-shaped bias-variance tradeoff curve and hence palidation-based ES (Geman et al. 1992.
Yang et al., 2020). But there are crucial differences: (1) our learning setting is not supervised; (2) the variance in supervised learning is with respect to sample distribution, whereas our variance here pertains to the (x* Je21 sequence. As discussed in Sec. we cannot directly apply validation-basse
ES, although it is possible to heuristically emulate by splitting the elements in y (Yaman et al.)
2021 Ding et al. 2022) which might be problematic for nonlinear IPs. Another line of related ideas variance-bar based online stange-point detection in time series analysis (Aminikhanghahi & Cook 2017), where running variance is often used to detect mean-shift assuming the means are piecewise constant. Here, the piecewise constancy assumption does not hold for our )=)))
4

---

Under review as a conference paper at ICLR 2023
Deep Decader
GP-MI
DIP-TV
SIREN
Low- levelevel moise
High-level noise
Figure 3: ES-WMM on DD, GP-DIP, DIP-TV, and SIREN for denoising "F16" with different levels of Gaussian noise (top: low-level noise; bottom: high-level noise). Red curves are PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES points. (We sketch the details of the DIP variants above in Appendixx A.5)
Partial theoretical justification We can make our heuristic argument in Sec. more rigorous by restricting ourselves additive denoising, i.e., y x n. and appealing the popular lineariza- tion strategy (i.e., neural tangent kernel Jacot et al. (2018); Heckel & Soltanolkotabi (20205) in understanding DNNs. The idea is based on the assumption that during DNN training does not move much away from initialization 0", so that the learning dynamic can be approximated by that of linearized model, i.e., suppose that we take the MSE loss
1/y Gz ~ |/ (z) (0((0) (0 )12 F(0),
(4)
where Jo (00) is the Jacobian of G with respect θ 00, and Go (2) + Jo (8%) (0- 00) is the first-order Taylor approximation to Gz(z) around 0" f(0) is simply least-qquares objective. We can directly calculate the running variance based on the linear model, as shown below.
Theorem 2.1. Let o; 's and w; 's be the singular values and left singular vectors of Ja(0"), and suppose we run gradient descent with step size 17 on the linearized objective f(0) to obtain {0'} and {x²} with x. = Ggo (z) + (o" -0)). Then provided that 1) S 1/ (a))
VAR(t) Σ CW... (w,, 8) (1 (1-1700) ?
(5)
where y = 1 GGo and Ow.n.o. 2 0 only depends on W, and for all i.
The proof can be found in Appendix A.2 Theorem 2.1 shows that if the learning rate (LR) n is sufficiently small, the WMV of {2' is monotonically decreasing. We can develop a complementary upper bound for the WMV that does have U shape. To this end, of Heckel & Soltanolkotabi (2020b), which can be summarized (some technical details omitted; precise statement reproduced inlAppendix A.3 as follows: consider the two-layer model Gc(B)= (ULU(BBC)e where C' E "x** models I x 1 trainable convolutions, v E IR 1x1 contains fixed weights, U is an upsampling operation, and B is the fixed random seed. Let J be reference Jacobian matrix solely determined by the upsampling operation U, and o; 's and w, 's the singular values and left singular vectors of J. Assume span (w...... wp Then, when 1 is sufficiently small, with high probability,
make use of Theorem
|| (B) 2112 S (1 noo)) ||ق||| E(n) + E||g|12-
(6)
where E is a small scalar related to the structure of the network and E(n) is the error introduced ( r2)2. So if the gap 0p/0pt1 > 1, [[Go 2112 is dominated by (1- (1-7o²) ||0|| when t is small, and then by E(n) when t is large. But since the former decreases and the latter increases when grows, the upper bound has a shape with respect
by noise: (n) = " ((1 ( -
to f. Based on this result, we have:
Theorem 2.2. Assume the same setting as Theorem 2 Heckel & Soltanolkotabi 2020b). With high probability, our WMV is upper bounded by
(1 - mo??) 2t
12
xx 2
+ 12 Σ +WW-1 -1) -1) (w]n)² + 12e 1lyl2 (7)
(1-
W 1 (1 1 21)
5

---

Under review as a conference paper at ICLR 2023
Exact Calculation Theeram Upper Bound in Theorem 2.2
15-
The exact statement and proof can be found in Appendix A.3 By similar reasoning as above, we can conclude that the up- per bound in Theorem 2.2 also has U shape. To interpret the results, Fig. 4 shows the curves (as functions of t) predicted by Theorems 2. and 2.2 The actual VAR curve should lie between the two curves. These results are primitive and lim- ited, simiar to the situations for many DL theories that provide untight upper and lower bounds; we leave a complete theoret- ical justification as future work.
15-
100
223
300
402
300
Iteration
Figure 4: The exact and upper bounds predicted by Theorems 2.1 and
A memory-efficient variant While Algorithm 1 is already lightweight and effective in practice, we can slightly modify to avoid maintaining Q and hence save memory. The trick is to use exponential moving variance (EMV), together with the expo- nential moving average (EMA), shown in Appendix A.4 The hard window size parameter W is now replaced by the soft forgetting factor the larger the the smaller the impact of the history, and hence a smaller effective window. We compare ES-WMV with ES-EMV in Appendix A.7.11 systematically for image denoising tasks. The latter has slightly better detection due to the strong smoothing effect (a = 0.1). For this paper, we prefer to remain simple and leave systematic evalua- tions of ES-EMV on other IPs future work.
EXPERIMENTS
We test ES-WMV for DIP image denoising, inpainting, super-resolution, MRI reconstruction, and blind image deblurring, spanning both linear and non- linear IPs. For image denoising, we also systematically evaluate ES-WMV on ma- jor variants of DIP, including DD (Heckel
leplk
& Hand 2019). DIP-TV (Cascarano et al.
2021I. GP-DIP (Cheng et al. 22019), and
demonstrate ES-WMV as reliable helper to detect good ES points. Details the DIP variants discussed in Ap- pendix A.5) We also compare ES-WMV with major competing methods, including DF-STE (Jo et al., 2021). SV-ES (Li al.
Figure 5: Baseline ES ES-WMV on denois-
ing with low-level noise. For NIMA, we report both technical quality assessment (NIMA-q) and aesthetic assessment (NIMA-a). Smaller PSNR gaps are better.
2021 DOP You al. 2020). SB (Shi et al. 2022 and VAL Yaman et al, 1222
Ding et al. 2022). Details of major ES-
based methods can be found in Appendix A.6 We use both PSNR and SSIM to access the re- construction quality, and we report PSNR and SSIM gaps (the difference between our detected and peak numbers) as indicators of our detection performance. Common acronyms, pointers to external codes, detailed experiment settings, results on real-world denoising, inpainting, and
super-resolution are in Appendices A.I A.7.1 A.7.2 A.7.7 A.7.9 and A.7.10, respectively.
3.1 IMAGE DENOISING
Prior works dealing with DIP overfitting mostly focus on image denoising, but typically only evalu- ate their methods on one or two kinds of noise with low noise levels, e.g., low-level Gaussian noise. To stretch our evaluation, we consider 4 types of noise: Gaussian, shot, impulse, and speckle. We take the classical 9-image dataset (Dabov et al. 2008), and for each noise type, generate two noise levels, low and high, i.e., level 2 and 4 of Hendrycks, & Dietterich (2019). respectively. See also the performance of our ES- WMV on real-world denoising in Tab. and Appendix A.7.7
Comparison with baseline ES methods is natural to expect that NR-IQMs, such as the classical BRISQUE (Mittal et al. 2012).. NIQE (Mittal et al.) 2013), and modern DNN- NIMA (Esfan- darani & Milanfar 2018) possibly make good ES criteria. We thus up 3 baseline methods

---

Under review as a conference paper at ICLR 2023
using BRISQUE, NIQE, and NIMA, respectively and seek the optimal x' by these metries. presents the comparison (in terms of PSNR gaps) of these 3 methods with our ES-WMT WS- on de- noising with low-level noise; results on high-level noise, and measured by SSIM are included in Appendix A.7.4 While our method enjoys favorable detection gaps (5 2) for most tested noise types/levels (except for Baboon, Kodak1, Kodak2 for certain noise types/levels; DIP itself is sub- optimal in terms of denoising such images with substantial high-frequency components), detection gaps by the baseline methods can get huge (2 10).
Fig.
Competing methods DF-STE (Jo et al. 202T) is specific for Gaussian and Pois- son denoising, and the noise variance is
needed for their tuning parameters. Fig. 6 presents the comparison with DF-STE in terms of PSNR. SSIM results are in Ap- pendix A.7.5) Here, directly report the final PSNRs obtained by both meth- ods. For low-level noise, there is no clear winner. For high-level noise, WS-WMV outperforms DF-STE by considerable margins. Although the right variance level is provided to DF-STE in order to tune their regularization parameters, DF-STE stops after only very few epochs leading to very low performance and almost zero standard deviations they return almost the noisy input. However, we do not perform any parameter tuning for ES-WMV. We further compare the two methods on CBSD68 in Appendix A.7.5
Figure 6: Comparison of DF-STE and ES-WMV for Gaussian and shot noise in terms of PSNR.
We report the results of SV-ES in Ap- pendix A.7.5) since ES-WMV performs largely comparably to SV-ES. However, ES-WMV is much faster in wall-clock time, as reported in Tab. 2) for each epoch, the overhead of our ES- WMV is less than 3/4 of the DIP update itself, while SV-ES is around 25x of that. There is surprise:
Table 2:
Wall-clock time (secs) of DIP and three ES methods per epoch on NVIDIA Tesla K40 GPU: mean and (sad). Total wall-clock time should contain both DIP and certain ES method.
DIP
SV-ES ES WMV ES-EMV
Time 0.448 (0.000)
13:027 (3872)
(0.116) 0.003 (0.003)
while our method only needs to update the running variance of the (~)) SV-ES needs to train a coupled autoencoder which is extremely expensive.
121 each time,
DOP is designed specifically just for impulse noise, so we compare ES- WS-WMV with DOP on impulse noise (see [Appendix A.7.5). The loss is changed to L to account for the sparse noise. In terms of the final PSNRs, DOP outperforms DIP with ES- SMVV by a small gap, but even the peak PSNR of DIP with l lags behind DOP by about 2dB for high noise levels.
Table 3: Comparison between ES-WMV and SB for image denoising on the CBSD68 dataset with varying noise level σ. Higher detected PSNR and earlier detection are better, which are in red: mcan and (sid).
15
25
50
PSNR
Epoch
PSNR
Epoch
PSNR
Epoch
WMV
28.7(3.2)
1962(2506)
27.4(2.6) 3068(2150)
24.2/2.3)
1548
SB
29.003.1) 4908(1757) 27.322.2 5099(1776)
23.00.00
5765(1346)
The ES method in SB is acknowledged to fail for vanilla DIP. Moreover, their modified model still suffers from the overfitting issue beyond the very low noise levels, as shown in Fig. 20 Their ES method fails to stop at appropriate places when the noise level is high. Hence, we test both ES-WMV and SB on their modified DIP model in (Shi et al.) 2022). based on two datasets they test: the classic 9-image dataset (Dabov) eet al., 2008) and CBSD68 dataset (Martin et al.] 2001). Qualitative results the 9 images are shown in Appendix A.7.5) detected PSNR and stopping epochs on the CBSD68 dataset are reported in Tab. 3, For SB, the detection threshold parameter is fixed at 0.01. is evident that both methods have similar detection performance for low noise levels but ES-WMV outperforms SB when the noise level is high. Also, EMV tends to stop much earlier than SB, saving computational cost.
We compare VAL with our ES- WMV on the 9-image dataset with low/high-level Gaussian and impulse noise. Since Ding et al.) (2022) takes 90% pixels to train DIP and that usually de- creases the peak performance, we report the final PSNRs detected by both methods (See Fig. 7). The two ES methods perform very comparably in image denoising, which is probably due to the mild violation of the iid assumption only, and also relatively low-degree information loss due to data splitting. The more complex nonlinear BID in Sec. 3.3) reveals their gap.
7

---

Under review as a conference paper at ICLR 2023
YAL Dec KALDARY
ES-WMV as a helper for DIP variants DD, DIP+TV, GP-DIP represent differ- ent regularization strategies for controlling overfitting. A critical issue, however, is setting the right hyperparameters for them so that overfitting is removed while peak- level performance is preserved. So prac- tically, these methods are not free from overfitting, especially when the noise level is high. Thus, instead of treating them as competitors, we test if ES- WMV can reliably detect good ES points for them. We focus on Gaussian denoising, and report the results in Fig. 8 (a)-(c) and Appendix A.7.6) ES-WMV is able to attain < 1 PNSR gap for most of the cases, with few out- liers. These regularizations typically change the recovery trajectory. We suspect that finetuning of our method may improve on these corner cases.
12
Figure 7: Comparison of VAL and EV-WMV for Gaus- sian and impulse noise in terms of PSNR.
ES-WMV as a helper for implicit neu- ral representations (INRs) INRs, such as Tancik et al. (2020) and Sitzmann et al. 2020). use multilayer perceptrons to rep- resent high-frequency functions in low- dimensional problem domains and have achieved superior results complex 3D visual tasks. We further extend our ES- WMV help the INR family and take SIREN (Sizzmann et al. 2020) as an ex-
2003
20
12
Dry Decake
24:07 DP
5.0
ample. SIREN parameterizes æ as the dis-
cretization of a continuous function: this
function takes into spatial coordinates and returns the corresponding function values. Here, we test SIREN, which is reviewed in in[ppeedixx A.5) as a replacement of DIP models for Gaussian denoising, and sum- marize the results in Fig. 8 and Fig. 21 ES-WMV is again able to detect near- peak performance for most images.
2.0
1.5
1.0
Figure 8: Performance of ES-WMV on DD, GP-DIP, DIP-TV, and SIREN for Gaussian denoising in terms of PSNR gaps. L: low noise level; H: high noise level.
3.2 MRI RECONSTRUCTION
Table 4: ConvDecoder on MRI reconstruc- tion for 30 cases: mean and (std) (D: Detected)
We further test ES- WMV on MRI reconstruction, a classical linear IP with nontrivial forward map- ping: y R F(x), where Fis the subsampled Fourier operator, and we use to indicate that the noise encountered in practical MRI imaging may be hybrid (e.g., additive, shot) and uncertain. Here, we take 8-fold undersampling and parametrize æ using
(S PSNR Gap SSIM(D) SSIM Gap
32.63 (2.36)
0.23 (0.32)
0.81 (0.99)
(0.01)
(Darestani & Heckel 2021), variant of DD. Due to the heavy overparame- terization, overfitting occurs, and ES is needed. Darestani & Heckel (02TT) directly sets the stopping point at the 2500-th epoch, and we run our FS-WMM. We visualize the performance
random cases (C1: 1001339 and C2: 1000190 90 sampled from Darestani & Heckel (202T), part of the fastMRI datatset (Zbontar al., 2018) in Fig. 23 (quality measured in SSIM, consistent with Darestani & Heckel (2021)). It is clear that WS- detects near-peak performance for both cases, and it is adaptive enough to yield comparable or better ES points than heuristically fixed ES points. We fur- ther test our ES- WSS on Conv Decoder for 30 cases from the fastMRI dataset (see Tab. 4). which shows the precise and stable detection of ES-WMV.
3.3 BLIND IMAGE DEBLURRING (BID)
In BID, a blurry and noisy image is given, and the goal is to recover a sharp and clean image. The blur is mostly caused by motion and/or optical nonideality in the camera, and the forward process
8

---

Under review as a conference paper at ICLR 2023
often modeled y k x + nt, where k is the blur kernel, models additive sensory noise, and * is linear convolution to model the spatial uniformity of the blur effect (Szeliski 2022). BID very challenging visual IP due the bilinearity: (k, k Recendly Ren et al. (2020);
Wang et al. (219);; Asim et al. (200); Tran et al. (202)) have tried to use DIP models to solve
BID by modeling & and æ as two separate DNNs, i.e., ming,.0. 1 (ze) * Go. (z. 12 + Vl (z / Vll (2,))l; where the regularizer is promote sparsity the gradient domain for reconstruction of æ, as standard in BID. We follow Ren et al. (2020) and choose multi-laye per- ceptron (MLP) with softmax activation for Go.. and the canonical DIP model (CNN- based encoder- decoder architecture) for Go, (zx). We change their regularizer from the original | (z1)llı to the current, as their original formulation is tested only on very low noise level o = 10 and no overfitting is observed. We set to work with higher noise level a 10-3, and find that their origi- nal formulation does not work. The positive effect of the modified regularizer on BID is discussed in Krishnan al. (201I).
First, we take 4 images and 3 kernels from the standard Levin dataset (Levin et al., (01T). re- sulting in 12 image-kernel combinations. The high noise level leads to substantial overfitting, as shown in Fig. (top left). Nonetheless, ES- WMV can reliably detect good ES points and lead to impressive visual reconstructions (see Fig. 9 (top right)). We systematically compare VAL and our ES-WMV on this difficult nonlin- car IP, as we suspect that nonlinearity can break VAL-dow as discussed in |Sec. 1| and subsam- pling the observation for training-vallddtiion validation splitting may be unwise. Our results (Fig. (bottom left/right) confirm these predictions: the peak performance is much worse after 10% of elements in y are removed for valiation. In contrast, our ES-WMP WMV returns quantitatively near-peak performance, far better than leaving the process overfit. In Appendix A.7.12 both low- and high-level noise on the entire Levin dataset for completeness.
1000
2000
3000
Deridioo
Ove_O
Serrd Conbinaioo
Scrdd Continatio
Figure 9: Top left: ES- WMV on BID; Top right: visual results of ES-WMM Bottom: quantitative results of ES- WM- and VAL, respectively
we test
3.0
3.4 ABLATION STUDY
2.0
1.5
The window size W (default: 100) and pa- tience number P (default: 1000) are the only hyperparameters for ES-WMV. To study their impact on ES detection, we vary them across range and check how the detection gap changes for Gaussian denoising on the classic 9-image dataset (Daboy et al., 2008) with medium-level noise, shown in Fig. 10 for PSNR gaps and Fig. 26 for SSIM gaps. Our method robust against these changes, and it seems larger W and P can bring in marginal improvement.
05
300
1000
2000
5000
160
160
sio
Windea
Figure 10: Effect of W and P
4 DISCUSSION
We have proposed simple yet effective ES detection method IMS. and the ES-EMV variant) that works robustly across multiple visual IPs and DIP variants. In comparison, competing ES methods are noise- or sppemccc--ppccfffcc and only work for limited scenarios; Li et al. (221) has comparable performance but it slows down the running speed too much; validation-basee ES (Ding et al. 2022 works well for the simple denoising task while lags behind our ES method lot in nonlinear IPs, e.g., BID. As for limitations, our theoretical justification is only partial, sharing the same difficulty of analyzing DNNs in general. Our ES method struggles with images with substantial fegh-rrequncy components; DIP needs to run numerous iterative steps for every instance, which is not ideal for time-constrained applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abdelrahman Abdelhamed, Mahmoud Afifi, Radu Timofte, Michael S. Brown, Yue Cao, Zhilu Zhang, Wangmeng Zuo, Xiaoling Zhang. Jiye Liu, Wendong Chen, Changyuan Wen, Meng Liu, Shuailin Lv. Yunchao Zhang, Zhihong Pan. Baopu Li, Teng Xi, Yanwen Fan, Xiyu Yu, Gang Zhang, Jingtuo Liu, Junyu Han, Errui Ding, Songhyun Yu, Bumjun Park, Jechang Jeong, Shuai Liu, Ziyao Zong, Nan Nan, Chenghua Li, Zengli Yang, Long Bao, Shuangquan Wang, Dongwoon Bai, Jungwon Lee, Youngjung Kim. Kyeongha Rho, Changyeop Shin, Sungho Kim, Pengliang Tang, Yiyun Zhao, Yuqian Zhou, Yuchen Fan, Thomas S. Huang, Zhihao Li, Nisarg A. Shah, Wei Liu, Qiong Yan, Yuzhi Zhao, Marcin Mozejko, Tomasz Latkowski, Lukasz Treszczotko, Michal Szafraniuk, Krzysztof Trojanowski, Yanhong Wu, Pablo Navarrete Michelini, Fengshuo Hu, Yunhua Lu, Sujin Kim, Wonjin Kim, Jaayeon Lec, Jang-Hwan Choi, Magauiya Zhussip, Azamat Khassenov, Jong Hyun Kim, Hwechul Cho, Priya Kansal, Sabari Nathan, Zhangyu Ye, Xiwen Lu, Yaqi Wu, Jiangxin Yang, Yanlong Cao, Siliang Tang, Yanpeng Cao, Matteo Maggioni, Ioannis Marras, Thomas Tanay, Gregory G. Slabaugh, Youliang Yan. Myungjoo Kang. Han-Soo Choi, Kyungmin Song, Shusong Xu, Xiaomu Lu, Tingniao Wang, Chunxia Lei, Bin Liu, Rajat Gupta, and Vineet Kumar. NTIRE 2020 challenge on real image denoising: Dataset, methods and results. In 2020 IEEF/CVF Conference on Computer Vision and Putlern Recognition, CVPR Workshops 2020, Seattle, WA, USA, June 14-19, 2020, pp. 2077-2088. Computer Vision Founda- tion / IEEE, 2020. doi: 10.1109/ 2000000000..00002555..
Samanch Aminikhanghahi and Diane J. Cook. A survey of methods for time series change point detection. Knowl. Inf. Syst, (1)))333 33733999777111 2017. doi: 10.1077/100115--16-09977
Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Blind image deconvolution using deep gener- ative priors. IEEE Trans. Computational Imaging, 6:1493-1506, 2020. doi: 10.1 109//T/11.2. 3032671.
Daniel Otero Baguer, Johannes Leuschner, and Maximilian Schmidt. Computed tomography recon- struction using deep image prior and learned reconstruction methods. CoRR, abs/2003.04989, 2020.
Khosro Bahrami and A. C. Kot. A fast approach for no-reference image sharpness assessment based on maximum local variation. IEEE Signal Process. Lett., 21(6):751-755, 2014. doi: 10.1 10.1 LSP.2014.2314497.
Pasquale Cascarano, Andrea Sebastiani, Maria Colomba Comes, Giorgia Franchini, and Federica Porta. Combining weighted total variation and deep image prior for natural and medical image restoration via ADMM. In 2021 21st International Conference on Computational Science and Its Applications (ICCSA). Cagliari, Italy, September 13-16, 2021 Workshops, pp. 39 49-46. IEEE, 2021. doi: 10. N0.1009/CCSA44449..2222222
Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective on the deep image prior. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 5443-5451. Computer Vision Foundation / IEEE, 2019, doi: 10. 10.11000 1WW.C//V..201..005..5
Frederique Crete, Thierry Dolmiere, Patricia Ladret, and Marina Nicolas. The blur effect: percep- tion and estimation with a new no-reference perceptual blur metric. In Bernice E. Rogowitz, Thrasyvoulos N. Pappas, and Scott J. Daly (eds.), Human Vision and Electronic Imaging XII, San Jose, CA, USA, January 29 February 1, 2007, volume 6492 of SPIE Proceedings, pp. 64920I. SPIE, 2007. doi: 10.1117/12.702790.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen O. Egiazarian. Image restoration by sparse 3d transform-domain collaborative filtering. Jaakko Astola, Karen O. Egiazarian, and Edward R. Dougherty (eds.), Image Processing: Algorithms and Systems VI, San Jose, California, USA, January 28-29, 2008, volume 6812 of SPIE Proceedings, pp. 681207. SPIE, 2008. doi: 10.1117/12..666355
Mohammad Zalbagi Darestani and Reinhard Heckel. Accelcrated MRI with un- un-trained neural net- works. IEEE Trans. Computational Imaging, 7:724-733, 2021. doi: 10. 10.1 109/TC1.2021.3000 2091.3097.96.
10

---

Under review as a conference paper at ICLR 2023
Lijun Ding, Liwei Jiang, Yudong Chen, Qing Qu, and Zhihui Zhu. Rank overspecified robust matrix recovery: Subgradient method and exact recovery. In Mare' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 26767-26778, 2021.
Lijun Ding, Zhen Qin, Liwei Jiang. Jinxin Zhou, and Zhihui Zhu. A validation approach over- parameterized matrix and image recovery. CoRR, abs/2209.10675, 2022. doi: 10.48550/arXiv. 20675.
Hossein Talebi Esfandarani and Peyman Milanfar. NIMA: neural image assessment. IEEE Trans. Image Process., 37(8):39984401 2018. doi: 10. 101//
Yossi Gandelsman, Assaf Shocher, and Michal Irani. "double-dip": Unsupervised image decom- position via coupled deep-image-priors. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 11026-11035. Computer Vision Foundation / IEEE, 2019. doi: 10. 1 10R.11/9......9..11..1
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural Comput., 4(1):1-58, 1992. doi: 10. 166/neeccco 1t1f2/neco.199..111.. 1992.4.1.1.
Kuang Gong, Ciprian Catana, Jinyi Qi, and Quanzheng Li. Direct reconstruction of linear parametric images from dynamic PET using nonlocal deep image prior. JEEE Trans. Medical Imaging, 41 (3):680-689, 2022. doi: 10. 20000
Paul Hand, Oscar Leong, and Vladislav Voroninski. Phase retrieval under a generative prior In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò desa-Bianchi, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréul, Canada, pp. 9154-9164, 2018
Fumio Hashimoto and Kibo Ote. Direct PET image reconstruction incorporating deep image prior and a forward projection model. CoRR, abs/2109.00766 2021.
Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non-convolutional networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Reinhard Heckel and Mahdi Soltanolkotabi. Compressive sensing with un-trained neural networks: Gradient descent finds a smooth approximation. In Proceedings of the 37th International Confer- ence on Machine Learning, ICML 2020, 13-18 July 2020. Virtual Event, volume 119 of Proceed- ings of Machine Learning Research, pp. 4149-4158. PMLR, 2020a.
Reinhard Heckel and Mahdi Soltanolkotabi. Denoising and regularization via exploiting the struc- tural bias of convolutional generators. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. 2020b.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Arthur Jacot, Clément Hongler. and Franck Gabriel. Neural tangent kernel: Convergence and gen- eralization in neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris- ten Grauman, Nicolò Cesa-Bianchi, and Roman Garett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8580-8589, 2018.
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehi- cles: Problems, datasets and state of the art. Found. Trends Comput. Graph. Vis., 12(1-3) 2020. doi: 00.150//000000009.
Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, ppppppp5555...00 5067-5076. IEFE, 2021. doi: 10.1 1091CCC48992.2221.00555
11

---

Under review as a conference paper at ICLR 2023
Dilip Krishnan, Terence Tay, and Rob Fergus. Blind deconvolution using normalized sparsity measure. In The 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011, Colorado Springs, CO, USA, 20-25 June 2011, pp. 233-240. IEEE Computer Society, 2011. doi: 10. 10.1109/CVPR.201 1.1.1..011 59995221.
Anat Levin, Yair Weiss, Frédo Durand, and William T. Freeman. Understanding blind deconvolution algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 33(12):2354-233 ---277554--677 2011. doi: 10.1109
TPAML2011.148,
Taihui Li, Zhong Zhuang, Hengyue Liang, Le Peng, Hengkang Wang, and Ju Sun. Self-validation: Early stopping for single- ingtancn deep generative priors. In 32nd British Machine Vision Con- ference 2021, BMVC 2021, Online, November 22-25, 2021, pp. 108. BMVA Press, 2021.
Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S. Kamilov. Image restoration using total variation regularized deep image prior. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019. Brighton, United Kingdom, May 12-17, 2019, pp. 7715-7719. IEEE, 2019. doi: 10.1109/CCASPP2000 268888.........88866
Xudong Ma, Alin Achim, and Paul R. Hill. Unsupervised image fusion using deep image priors. CoRR, abs/2l 10.09490, 2021.
David R. Martin, Charless C. Fowlkes, Doron Tal, and Jitendra Malik. database of human seg- mented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the Eighth International Conference On Computer Vision (ICCV-01), Vancouver, British Columbia, Canada, July 7-14, 2001 Volume 2, pp. 416-425. IEEE Computer Society, 2001. doi: 10. 10.1109//CC222001...333
Gary Mataev, Peyman Milanfar, and Michael Elad. Deepred: Deep image prior powered by red. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Christopher A. Metzler, Ali Mousavi, Reinhard Heckel, and Richard G. Baraniuk. Unsupervised learning with stein's 's unbiased risk estimator. CoRR, abs/1803 0111885.11555 10531, 2018.
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. No-reference image quality as- sessment in the spatial domain. IEEE Trans. Image Process., 21(12):4695-4708 2012. doi: 10.1 10.1109/1122201222244455
Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. Making a "completely blind" image quality analyzer. IEEE Signal Process. Lett., 20(3):209-212, 2013, doi: 10. 10.1109/LSP220122222722
Gregory Ongie, Ajil Jalal, Christopher A. Metzler, Richard G. Baraniuk, Alexandros G. Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems in imaging. IEEE J. Sel. Areas Inf. Theory, 1(1):39-56, 2020. doi: 10. 109/jsait. 2020.2991.66666
Adnan Qayyum, Inaam Ilahi, Fahad Shamshad, Farid Boussaid, Mohammed Bennamoun, and Ju- naid Qadir. Untrained neural network priors for inverse imaging problems: survey. TechRxiv, mar 2021. doi: 10.36227/techxxiv. 12082155.v.
Dongwei Ren, Kai Zhang, Qilong Wang. Qinghua Hu, and Wangmeng Zuo. Neural blind decon- volution using deep priors. 2020 IEEÉ/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2020, Seattle WA. USA, June 13-19, 2020, pp. 3338 3347. Computer Vision Foundation/ IEEE, 2020. doi: 10. 1099CVPR42260222000004444
Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G. M. Snoek. On measuring and controlling the spectral bias of the deep image prior. Int. Compur. Vis., 130(4):885 2022. doi: 10. 1007/s 1007/s11263-001 1221--21--1172---
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell. Maria-Florine Balcan, and Hsuan-Tien Lin aedss),.vvane in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
12

---

Under review as a conference paper at ICLR 2023
Zhaodong Sun. Solving inverse problems with hybrid deep image priors: the challenge of preventing overfitting. CoRR, abs/2011.01144 2020.
Richard Szeliski. Computer Vision Algorithms and Applications, Second Edition. Texts in Com- puter Science. Springer, 2022. ISBN 978-3-030-34371-2. doi: 10. 00.1007/978-3-03--333222
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich- Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florins Balcan, and Hsuan-'Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Hugo Larochelle,
Kshitij Tayal, Raunak Manekar, Zhong Zhuang, David Yang, Vipin Kumar, Felix Hofmann, and Ju Sun. Phase retrieval using single-instance deep generative prior. CoRR, abs/2106.0433 2021.
Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 11956-11965. Computer Vision Foundation / IEEE, 2021. doi: N0.11099CVPR44437..00 202.6433.7.221111177
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. Deep image prior. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. pp. 9446-9454. Computer Vision Foundation IEEE Computer Society, 2018.
doi: 10.11099CVRR.2088.0000
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Ahh Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- lems 32: Annual Conference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 2968-2979, 2019.
David Van Veen, Ajil Jalal, Eric Price, Sriram Vishwanath, and Alexandros G. Dimakis. Compressed 2018.
sensing with deep image prior and learned regularization. CoRR,
Zhunxuan Wang, Zipei Wang. Qiqi Li, and Hakan Bilen. Image deconvolution with deep image and kernel priors. In 2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019. pp. 980-989, IEEE, 2019.
Francis Williams, Tesco Schneider, Cláudio T. Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. Deep geometric prior for surface reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019. Long Beach, CA, USA, June 16-20, 2019, pp. 10130-10139. Computer Vision Foundation IEEE, 2019. doi: 10. 10990CVV...2119.0003....
Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising:
A new benchmark. CoRR, abs/1804.02661 2018
Burhaneddin Yaman, Seyed Amir Hossein Hosseini, and Mehmet Akcakaya. Zero-shot physics- guided deep learning for subject-specific MRI reconstruction. In NeurlPS 2021 Workshop Deep Learning and Inverse Problems, 2021.
Zitong Yang. Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade- off for generalization of neural networks. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 10767-10777. PMLR, 2020.
Jaejun Yoo, Kyong Hwan Jin, Harshit Gupta, Jérôme Yerly, Matthias Stuber, and Michael Unser. Time-dependent deep image prior for dynamic MRI. IEEE Trans. Medical Imaging. 40(12); 3337-3348, 2021. doi: 10.1 109/0MM.2221 1,004288..
Chong You, Zhihui Zhu, Qing Qu and Yi Ma. Robust recovery via implicit bias of discrepant learning rates for double wwerpparametriizatonn. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 121 2020, virtual, 2020.
13

---

Under review as a conference paper at ICLR 2023
Jure Zbontar, Florian Knoll, Anuroop Sriram, Matthew J. Muckley, Mary Bruno, Aaron Defazio, Mare Parente, Krzysztof J. Geras, Joe Katsnelson, Hersh Chandarana, Zizhao Zhang, Michal Drozdzal. Adriana Romero, Michael G. Rabbat, Pascal Vincent, James Pinkerton, Duo Wang, Nafissa Yakubova, Erich Owens, C. Lawrence Zitnick, Michael P. Recht, Daniel K. Sodickson, and Yvonne W. Lui. fastmri: An open dataset and benchmarks for accelerated MRI. CoRR, abs/1811.08833 2018.
Zhong Zhuang, Taihui Li, Hengkang Wang, and Ju Sun. Blind image deblurring with unknown kernel size and substantial noise. CoRR, has/2208.0948833 2022a. doi: 10.48550/arX1v.2200. 09483.
Zhong Zhuang, David Yang, Felix Hofmann, David Barmherzig, and Ju Sun. Practical phase trieval using double deep image priors. arXiv preprint arXiv:2211.0079, 2022b.
A APPENDIX
A.I ACRONYMS
List of Common Acronyms (in alphabetic order)
CI
computational imaging
CNN
convolutional neural network
DD
deep decoder
DIP
deep image prior
DIP-TV
DIP with total variation regularization
DL
deep learning
DNN
deep neural network
ELTO
early-learning- ttllaarmnnnntee----veitttiig
ES
early stopping
EMA
exponential moving average exponential moving variance
EMV
FR-IQM
full-reference image quality metric
GP-DIP
Gaussian process DIP implicit neural representations
INR
IP
inverse problem
MSE
mean squared error no-reference image quality metric
NR-IQM
PSNR
peak signal-to to-iiee ratio
SIREN
sinusoidal representation networks
SOTA
state-of-the-art
VAR
variance windowed moving variance
WMV
A.2 PROOF OF
Proof. To simplify the notation, write ύ = y least-squares objective in Eq. (4) equivalent to
Goo (2), Jc(( and = 0 00.00 So the
- Jcll2
(8)
and the gradient update reads
c' - c'- n (JC -))
(9)
where α and x. = Jc Go The residual at time t can be computed
r¹ = ÿ - Jc'
(10)
= - (c--1 - (JO 1))
(11)
(I -JJ (ū Jc--1)
(12)
14

---

Under review as a conference paper at ICLR 2023
(I JJJ (ÿ - Jc
(13)
= [I JJJJ (g- - Jc -Jce
(using e = 0)
(14)
= [I JT) ÿ.
(15)
Assume the SVD of J as J = WEVT. Then
r¹ [I -1y/W/''')'s و
_(1 (1- no?)²² w/ iw::
(16)
and so
Jc = y r² Σ 211 (1- w yw,
(17)
Consider set of W vectors V = (01,... Uw) }. We have that the empirical variance
2
W
W
VAR(V) Σ
U Σ
vj
Van
1
1 W
V
(18)
W
2
2
So the variance of the set {x², x=+1
x
+ }.
same as the variance of the set
Jct..), can be calculated as
1 W- Σ Dw (1- (1 mo?))
W 1
2
Σ 10 0
Σ
(w! 1)
Σ w==
1- (1 - no? c+w
W
W
(19)
W
W-1
27
(mm) Ø)
W Σ
noo?) (
Σ
- (1 - (ooo 2+:0
(20)
W
===
m==
(w01 8) 2 ( ( + W (1 - onno?)? 22
? 22
2W (1
n1,,)²
(1 (?o w)
W/2
1 - (1 ???
no?
A. (oor (1-
+ (1 (--uul)) (1=17?) 2t
W2 - 2W (1 (1-no?) nao)((1 (1 - ww
?on
n/o!
(21)
W Σ we y) (1 - no?
1 - (1 - ?
(1- (1 - noo
W
(22)
no?
2 no?
?o?
So the constants Ow.n.rr 's are defined as
1 (1 - no? 2
(1 - (1 - n)? W2
CWin.
W
(23)
W2no?
2 no?
110
To see they are nonnegative, it is sufficient to show that W - (1 - ?
(1 - (1- (1-no?) no?) 2 0
2 ?o?
no?
no? (1 (1- (1-(1-107)25) n2 (1- (1-107) 20. 0. (24)
Now consider the function
b(E, W) = <W (1-
(2 {1 (1 W)² & 10,1],
(25)
First, one can casily check that hig W) 0 for all W 2 1 and all E [0, 1], i.c., A(C, W)
is monotonically increasing with respect to W. Thus, in order to prove Cw.m.; 2 0, it suffices to show that h(g, 1) 2 0. Now
h(E,1)= = E(1 (1 {) - (2 -)²- = 0,
(1)
(26)
completing the proof.
15

---

Under review as a conference paper at ICLR 2023
A.3 PROOF OF 2.2
We first restate Theorem 2 in Heckel & Soltanolkotabi (2020b).
Theorem A.1 (Heckel & Soltanolkotabi (2200)). Let a E R be a signal in the span of the first p trigonometric basis functions, and consider a noisy observation = a + n, where the noise n. ~ N (o, - /n 1). To denoise this signal, we fit a two-layer generator network (B) Rc),(UBC),, where U an upsampling operator that implements circular convolution with a given kernel u. Denote a ||u|||| Fglu @ w/ where g(I) = (1 - cos (1)/=x) and @ denotes the circular convolution. Fix any E E (0,0p//r) and suppose k z Cun/s", where Cu > a constant only depending on u. Consider gradient descent with step size 1
and B wid (0, 1), and [I....,1, .....,,,1.,, -1).....11/.....................
S |F =2 is the Fourier trans- Then, for all iterates t obeying
form of u ) starting from Co mid N (0,w?), entries, w x lilla
vT
t S noz.
the reconstruction error obeys
a
[B) - -xl12 S (1- -)oo)) ||0|| +
Σ no 1) (2((wn? n)² + +
with probability at least 1 exp (-x²) n
Note that since B and (0, 1) and hence is full-rank with probability one, the original Theorem 1 & 2 of Heckel & Soltanolkotabi (2020b) rename BC into C' and state the result directly on C', i.e., assume the model is (eLU(UU"))r It is easy see the original theorems imply the version stated here.
With this, we can obtain our Theorem 2.2, stated in full technical form here:
Theorem A.2. Let E R." be a signal in the span of the first p trigonometric basis func- tions, and consider noisy observation y = x. + n, where the noise n N (0. /2/n. T). To denoise this signal, we fit two-layer generator network Gc
RcLU(UBB)e, where -1/ and misd N(0, 1), and U is an upsampling operator that im- plements circular convolution with a given kernel u. Denote a = F (/ 1/2 where gt)) = (1 (1-cos -ocs ((t)/(()) and @ denotes the circular convolution. Fix any E E (0, (0,0p/0 01). and sup- pose k 2 Cun/E5, where Cn > O is a constant only depending on u. Consider gradient descent with step size 17 S ||Fu| 1 is the Fourier transform of ) starting from Co rid N (0,w²). entries, w x /n
v = -,,
1ulle Then, for all iterates t obeying & S
100 our WMV obeys
x (1 - (1-)0?? 2t
12 W
12 Σ (1 -W i=1
2 (w/n)² + 12e 1212 (27)
WMV <
(1 1)
with probability at least 1 - exp (- -
Proof. We make use of the basic inequality: l|a - b112 < 2||a|1 + +21bbl2 for any two vectors a, b of compatible dimension. We have
W-1
W
1
|G (B)
G
(28)
W
W=D
j=0
W-I
W-1
| (BC -x x + x
Gous (B)
(29)
W
W
w=0
1=0
W
W Σ
2
<
|Goo+ (B) x012
+ +2
G (B)
(30)
W
W
w=0
1=0
2
| (B) 2B)
2
Σ
G (B) =
(31)
W
0=
16

---

Under review as a conference paper at ICLR 2023
+ 2
convex and Jensen's 's inequality)
W-1
4 W
G (B) x
(32)
In view of Theorem A.I
|G (B) x=2 3 3(1 no?)) 20 | 1122+ + Σ ((1 not) no1- -1) (no n 35 11
"
(33)
Thus,
| (B) - 2x
0==
3/ 1
(1 110 21+2w
Σ
n1- -1) (w?n)² +3W="llyls (34)
10=0
nop
- (1 2
+3W Σ n1= 4 -1) (w)"n²² 3W 3
S <3
- (1 n17?)2
(35)
(1 mo??) 2r
S 331121 1- (1 moo)
+ 3W Σ (1-n0?? -
-t - (w!n)² + <3We?lull..
(36)
completing the proof.
A.4 ES-EMV ALGORITHM
The exponential moving variance version of our method summarized in Algorithm
Algorithm DIP with ES- EMV
Input: random seed randomly inittaliiie Gg, forgetting factor (0, 1), patience number P. iteration counter A: = 0, EMA O, EMV 0, EMV min
Output: reconstruction x²
1: while not stopped do
2: update 9 via Eq. (2) to obtain g and R+1
3: FMA k+1 = (1 a:) FMMA + *
4:
EMVA-1 (1 a EV + a(1 - [a] a0
EMM* $
5:
if FEMV++1 < EMV min then
6:
EMV min < EMV x f- x k+1 end
7:
8:
if EMV min stagnates for P iterations then
9:
stop and return x²* end if
10:
11:
== k +1
12:
end while
A.5
MORE DETAILS ON MAJOR DIP VARIANTS
Deep Decoder (DD) Heckel Hand 2019 differs from DIP mainly terms the network architecture: is typically under-parne ander-parrmettrriedd network consisting of mainly 1 x convolutions, upsampling. ReLU and channel-wise normalization layers, while DIP uses an pver-parameterized, U-net like convolutional network.
17

---

Under review as a conference paper at ICLR 2023
GP-DIP CChng et al. 2019) uses the original DIP U et al. 2018 network and formula-
tion, but replaces the stochastic gradient descent (SGD) by stochastic gradient Langevin dynamics
(SGLD) in the gradient update step. i.e., for generic gradient step for optimizing Eq. (2) reads:
0 0
V ((Go(z))) + XR((G(z))) +
(37)
where 1) is zero-mean Gaussian with an isotropic variance level t.
DIP-TV CCascarano al. 2021 uses the original DIP Ulyanov et al. 2018 network, with
Total Variation (TV) regularizer added. Then, the proposed objective is solved with Alternating Direction Method of Multipliers (ADMM) framework.
(Sitzmann et al. 2020) treats the object directly as continuous function on R2 or R (or higher-dimensional spaces depending on the application) and hence parameterizes it as a multi-layer perceptron (MLP): 1) the input to SIREN is the 2D/3D coordinate of each pixel instead of random values, and 2) the network uses a sinusoidal activation function instead of the commonly used ReLU. When substituting the DIP network with SIREN and solve Dqllee...cc (2) problems, similar overfitting issue still observed.
SIREN
A.6 MORE DETAILS ON MAJOR ES METHODS
Here, provide more details major competing methods, all of them ES-based except for You
et al. 2020).
Spectral Bias (SB) Shi et al. (2022) operates on DD models, and proposes two modifications to change the spectral bias: (1) controlling the operator norm of the weight w for each convolutional layer by the normalization
w
(38)
max (1, |1ooloop///
ensuring that ||w0llo, S X, which turn controls the Fourier spectrum of the underlying function represented by the layer; (2) performing Gaussian upsampling instead of the typical bilinear up- sampling suppress the smoothness effect of the latter. These two modifications with appropriate parameter setting (A, and a in Gaussian filtering) can improve the learning of the high- high-frequency components by DI,, and allow the Saarinness-verrshaapnssss stopping criterion
Ar(x')
Σ r(x
Σ r(x (ww--WWW)
(39)
where r(w')= = B(x') B(a')/S(x"). and B-)) and S(.) are the blurriness and sharpness metrics in Crete et al. (2007) and Bahrami & Kot (014)) respectively. In other words, the criterion in Eq. (39) measures the change of average blurrinsss-verr-haagg
ratios over consecutive windows of size W. and small changes indicate good ES points. But, said, this criterion only works for the modified DD models and not other DIP variants, as acknowledged by the authors in Shi et al. (2022)
and confirmed in our experiment (see Sec. 3.1).
DF-STE Jo et al. 2021) targets Gaussian denoising with known noise levels (i.e., y x + n. where is iid Gaussian noise), and considers the objective
1 min n lly Goly))l + n² JG.
(40)
where tr JG. (y) is the trace of the network Jacobian with respect to the input, i.c., the divergence term Jo al. (2021). The divergence term a proxy for controlling the capacity of the network. The paper then proposes heuristic zero-crossing stopping criterion that stops the iteration when the loss starts to cross zero into negative values. Although the idea works reasonably well on Gaus- sian denoising with low and known noise level (the variance level o² is explicitly needed in the regularization parameter ahead the divergence term), starts to break down when the noise level increases even if the right noise level is provided; see Sec. 3.1] Also, although the paper has tended the formulation to handle Poisson noise, it is unclear how to generalize the idea for handling other types of noise, well how to move beyond simple additive denoising problems.
18

---

Under review as a conference paper at ICLR 2023
SV-ES (x') h21
Li al. (2021) proposes training autoencoder online using the reconstruction sequence
min
LASE Dw E. (xx))- 10.17
(41)
Any x' is passed through the current autoencoder, and the reconstruction error AE is recorded. We observe that the error curve typically follows U-shape, and the valley of the curve is approx- imately aligned with the peak of the PNSR curve. We hence design an ES method by detecting the valley of the error curve. This method works reasonably well across different IPs and different DIP variants. major drawback is the efficiency: the overhead caused by online training of the autoencoder is meder-of-magnitueee larger than the cost of DIP update itself, shown in Tab. 2
DOP You et al. 2020) considers additive sparse (e.g., salt-and salt-and-pepper pepper noise) noise only and proposes modeling the clean image and noise explicitly in the objective:
min y - Go(z) (g og h l) g.g.h
(42)
where the overparametrized term g g h h (o denotes the Hadamard product) is meant to capture the sparse noise, where similar idea has proved effective for sparse recovery in Vaskevicius et al. 2019).. Different properly-tune -uned learning rates for the clean image and sparse noise terms necessary for success. The downside includes the prolonged running time as it pushes the peak reconstruction to the very last iteration, and the difficulty to extend the idea to other types of noise.
A.7 ADDITIONAL EXPERIMENTAL DETAILS RESULTS
A.7.1
EXTERNAL CODES
DIP:
DVP-TV: http:://gghhaacccoooooccca////a2
GP-DIP: http:///wooooocca......aaaaaaaar
DF-STE: httpp://stttt..cooo.sssssaa/ssss
SV-ES: http:://....aaiaccocoaammaaaaaaa
DOP: h00.l//ss/.cchaococ//////////010
CBSD68:
A.7.2 EXPERIMENT SETTINGS
Our default setup for all experiments is as follows. Our DIP model is the original one from Ulyanov al. (2018;; the optimizer ADAM with learning rate 0.01. For all other models, we use their default architectures, optimizers, and hyperparameters. For ES- ES-WMV, the default window size W = 100, and patience number P 1000. We use both PSNR and SSIM access the reconstruction quality, and we report PSNR and SSIM gaps (the difference between our detected and peak numbers) as an indicator of our detection performance. For most experiments, we repeat the experiments 3 times to report the mean and standard deviation; when not, we explain why.
Noise generation Following the noise generation rules of Hendrycks & Dietterich (2019 ! simulate four kinds noise and three intensity levels for each noise type. The detailed information is follows.
Gaussian noise: 0 mean additive Gaussian noise with variance 0.12, 0.18, and 0.26 for low, medium, and high noise levels, respectively;
19

---

Under review as a conference paper at ICLR 2023
Impulse noise: also known salt-and- noise, replacing each pixel with probability p E [0, 1] into white or black pixel with half chance each. Low, medium, and high noise levels correspond
0.3, 0.5, 0.7, respectively;
Speckle noise: for each pixel x [0, 1], the noisy pixel is 2(1 + E), where is O-mean Gaussian with variance level 0.20, 0.35, 0.45 for low, medium, and high noise levels, respectively;
Shot noise: also known as Poisson noise. For each pixel, x E [0, 1], the noisy pixel is Poisson distributed with rate λx, where À is 25, 12, for low, medium, and high noise levels, respectively.
A.7.3 DENOISING EXAMPLES
On image denoising with different types and levels noise, our ES method can help DIP to detect near-peak ES points, shown in Fig. 11 We also explore the possibility of using the loss for ES here, but we fail to find correlations between the trend of the loss and that of the PSNR curve.
Gaussian
Impulse
Shot
Speckle
Low-lovel noise
High moise
Figure 11: Our ES-WMV method on DIP for denoising "F16" with different noise types and levels (top: low-level noise; bottom:
high-level noise). Red curves are PSNR curves, and blue curves are VAR curves. The green bars indicate the detected ES point.
Gaussian
Impulse
Shot
Speckle
Low-level noise
20
820
High-level noise
Figure 12: Our ES-WMV method on DIP for denoising "F16" with different noise types and levels (top: low-level noise; bottom: high-level noise). Red curves are PSNR curves, and brown curves are loss curves.
A.7.4 COMPARISON WITH BASELINE METHODS
To further compare with baseline methods, we report the PSNR gaps of high-level noise cases and SSIM gaps of low- and high-level noise cases in Fig. 15 Fig. 16 and Fig. 17 respectively, which show similar trend the results of PSNR gaps. The detection gaps of method very marginal (< 0.02) for most noise types and levels (except for Baboon and Kodakl for certain noise types/levels), while the baseline methods well exceed (1. for most cases. In addition, we provide
20

---

Under review as a conference paper at ICLR 2023
BRISQUE
NIQE
N
NIMA-a
ES WMV
Peuk
Figure 13: Visual comparisons of NR-IQMs and ES. WMV. From top to bottom: Gaussian noise (low), Gaussian noise (high). impulse noise (low), impulse noise (high).
BRISQUE
NIQE
NIMA-q
NIMA-
ES-WMV
Peak
Figure 14: Visual comparisons of NR-IQMs and ES-WMV. From top to bottom: shot noise (low), shot noise (high), speckle noise (low), speckle noise (high).
21

---

Under review as a conference paper at ICLR 2023
30
hIDE ND00 Car
00-
L200
Kodad
Kodaic
Ilom
Roder
Autid
KodiJ
Grdal
Eutar
Fopes
Kadal
Koai
Sket mobe
Speide
Figure 15: High-level noise detection performance in terms of PSNR gaps. For NIMA, we report both technical quality assessment (NIMA-q) and aesthetic assessment (NIMA-a). Smaller PSNR gaps are better.
HEIQQUE
0.-
0.7-
03
6.1
Kndad
Endatt
Audel
Grosit
Inpe txbe
0.5
0.-
Lew
Kait
Poym
Lew
Kodar
Kndad
Sar Adie
Speake bxic
Figure 16: Low-level noise detection performance in terms of SSIM gaps. For NIMA, we report both technical quality assessment (NIMA-q) and aesthetic assessment (NIMA-a). Smaller SSIM gaps are better.
some visual detection results in Figs. 13 and 14 four baseline methods visually.
Our ES- WS significantly outperforms than the
A.7.5 COMPARISON WITH COMPETING METHODS
Comparison between ES- WMV with DF-STE for Gaussian and shot noise on the 9-image dataset in terms of SSIM is reported in Fig. 18 Furthermore, we also test our ES-WMV and DF-STE on CBSD68 in Tab. 5 Our ES- WMV wins in high-level noise cases, but lags behind DF-STE in the low-level cases. The gaps between from the peak-perffrmance between the original DIP and DF-STE- modifications in DF-STE have affected the peak performance, positively for lovel cases and negatively for high-level cases,
ES-WMV and DF-STE for all noise levels mostly come
22

---

Under review as a conference paper at ICLR 2023
0.8
NIOE NIVA.
0.7
0.6
0.7-
0.1
0.0
Bepebte
0.5
077
06.
022
02-
Kutik Kndatt
Figure 17: High-level noise detection performance in terms of SSIM gaps. For NIMA, we report both technical quality assessment (NIMA-q) and aesthetic assessment (NIMA-a). Smaller SSIM gaps are better.
100
DI-STIIOO Dat
08-
0.8-
0.6
02-
02
Knddal
Nover
Entas
Gaseba
Shatrake
Figure 18: Comparison of DF-STE and ES-WMV for Gaussian and shot noise in terms of SSIM.
much from our ES method as evident from the uniformly small detection gaps reported in Tab. 5 Moreover, DF-STE can only handle Gaussian and Poisson noise for denoising, and the exact noise level a required hyperparameter for their method to work.
Then we compare our ES-WMV and SV-ES in Fig. 19 The DIP results with ES-WMMV vs. DOP on impulse noise are shown in Tab. For SB, part of the qualitative detection results on the images reported in Fig. 20
Table 5: Comparison between ES- WMV and DF-STE for image denoising the CBSD68 dataset with varying noise level a: mean and (sad) PSNR gaps below 1.0 are colored as red.
25
50
ES ES-WMV
28.7(32)
27.4/2.)
24.2/2.3)
DIP (Peak)
29.70300
28.0(2.4)
24.9(2.3)
Gap
1.0g0.7)
0.7(0.5)
0.700.5)
DF DT-STE
31.4(1.)
28.4(2.2)
21.1(2.5)
A.7.6 ES-WMV AS HELPER
Performance of ES WMV DD, GP-DIP, DIP-TV, and SIREN for Gaussian denoising in terms of SSIM gaps (see Fig. 21
http:/ wwwp://wwww///..// tut. fw/ foi /GCF www.H///0./neeee #ref_resu t.
23

---

Under review as a conference paper at ICLR 2023
3.0
Oor Piga
15-
33.
135
00.
Kodad
3.5
10
1.5
00
Citaa
Spekte
Figure 19: Low- and high-level noise detection performance of SV-ES and ours in terms of PSNR gaps.
Table 6: DIP with ES-WMV vs. DOP on impulse noise: mean and (sdd).
Low Level
High Level
PSNR
SSIM
PSNR
SSIM
DIP-ES
31.64 (5.69)
0.85 (0.18)
24.74 (3.23)
0.67 (0.19)
DOP
32.12 (4.52)
0.92 (0.07)
27.34 (378))
0.86 (0.10)
A.7.7 PERFORMANCE REAL-WORLD DENOISING
Table 7: DIP with ES-WMV on real image denoising on the PolyU Dataset: mean and (sid). (D: Detected)
PSNR(D)
PSNR Gap
SSIM(D
SSIM Gap
(1.22)
(0.02)
(000))
DIP (f1)
36.20 (2.81)
1.64 (1.58)
0.97 (0.02)
0.01 (0.0))
DIP (Huber)
36.76 (2.96)
1.28 (1.09)
0.98 (0.03)
0.01 (0.0)
As stated from the beginning, ES-WMV is designed with real-world IPs, targeting unknown noise types and levels. Given the encouraging performance above, we test on a common real-world denoising datasct-Polyl3 Dataset Xu et al. (2018), which contains 100 cropped regions of 512 x12 from 40 scenes. The results are reported in Tab. 7 We do not repeat the experiments here; the means and standard deviations detection gaps are < 1.64 in PSNR and < 0.01 in SSIM for this dataset across various losses. The absolute PNSR and SSIM detected are surprisingly high.
obtained over the 100 images of the PolyU dataset. On average,
A.7.8 RESULTS FOR MRI RECONSTRUCTION
The detection performance of ES-WMV for MRI reconstruction is shown in Fig. 23 in terms of SSIM.
A.7.9 IMAGE INPAINTING
In this task, a clean image Ty E [0, HxW contaminated by additive Gaussian noise s, and then only partially observed yield the observation y (xo + e) @ m, where {0, ">xw is binary mask and O denotes the Hadamard product. Given y and m, the goal is to reconstruct æn- We consider the formulation reparametrized by DIP, where Gg is trainable DNN parametrized by
24

---

Under review as a conference paper at ICLR 2023
Peppers
F16
Lena
15
PSNR
20
Tetio
20
15
WWW
15
15
10
10
10
5
2000
6000
6000
10000
2000
4000
0000
8000
10000
4000
6000
10000
25
25
20
20
20
15
15
15
Opemination Raration
Optimization Ecratio
Optimization Etration
30
30
25
23-
00
2000
4000
6000
8000
10000
2000 4000
6000
8000 10000
0g
2000 4000
6000
8000
10000
0.00 mizittinonn Iteraiion
Ostmizztion Reration
Optmization iteration
Figure 20: Comparison between ES-WMV and SB for image denoising (top:
15; middle:
25; bottom: 50). The red and blue curves are the PNSR and the ratio metric curves. The orange and green bars indicate the ES points detected by our ES- WSV and SB, respectively.
0.35
0.35
1125
0.20
0.10
1.10
0.05
(if)
(krp
(b) dl---
0.35
0.35
0.20
0.30
0.15
005
0.0
1/10
Low 511
DIP-T
Figure 21: Performance ES-WMV on DD, GP-DIP, DIP-TV, and SIREN for Gaussian denoising in terms of SSIM gaps. L: low noise level; H: high noise level
25

---

Under review as a conference paper at ICLR 2023
10-
1.0
03
0.4
sight
84
pug Val (ik) Dan (800
VAl (he)
02
PP
Kudad
Avidd
Ove
Koddf?
82
Preper
Kosta
Figure 22: Comparison of VAL and ES-WMV for Gaussian and impulse noise in terms of SSIM.
0.85
0.80
0.75
Peak (CI)
0.70
ES
Peak (C2)
0.65
Con-Deaskr
2000
4003
Figure 23: Detection on MRI reconstruction
and is a frozen random seed:
(0) = ([Ga(z) y) ml|
(43)
The mask is generated according to iid Bernoulli model with rate of 50%, i..., half of pixels not observed in expectation. The noise is set to the medium level, i.e., additive Gaussian with mean and 0.18 variance. We test our ES- WMV for DIP the inpainting dataset used in the original DIP paper Ulyanov et al. (2018). The PSNR gaps are < 1.00 and the SSIM gaps are < 0.05 for most cases (see Tab. We also visualize examples in Fig. 24
Table 8: Detection performance of DIP with ES- WMV for image inpainting: mean and (st). PSNR gaps below 1.00 are colored as red; SSIM gaps below 0.05 are colored blue. (D: Detected)
PSNR(D)
PSNR Gap
SSIM(D)
SSIM Gap
21.59 (0.03)
(0.03)
(0.0)
(0.00)
Boat
21.91
1.16
0.68
0,03
House
27.95 (0.33)
0.48 (0.10)
0.89 (001)
0.01 (0.00)
24.71 (0.30)
(0.18)
(0.00)
0.01 (0.00)
Peppers
25.86 (0.22)
0.23 (0.05)
0.84 (0.01)
0.02 (0.00)
C.man
25.26 (0.09)
0.23 (0.14)
0.82 (0.00)
0.01 (0.00)
Couple
21.40 (0.44)
(0.53)
(001)
0.04 (0.02)
Finger
20.87
0.24 (0.17)
0.77
0.01
Hill
23.54 (0.08)
0.25 (0.11)
0.70 (0.00)
0.00 (0.00)
22.92 (0.25)
(0.11)
(001)
0.01 (0.00)
Montage
26.16
0.38
0.86
0.03
26

---

Under review as a conference paper at ICLR 2023
Naiey Image
ES-WMY Datetied Image
Perk PSNR Image
Overiiting Image
Figure 24: Visual detection performance of ES-WMV on image inpainting.
A.7.10 IMAGE RESOLUTINO
In this task, degraded observation y is obtained as the downsampled version of a noisy image: i.c., (Zgg + E). where D.(.): : [0, 1]3xtH xtW + [0,
is a downsampling operator that resizes an image by the factor t. Then given y and t, the goal is reconstruct Zo. We consider the formulation reparametrized by DIP, where Go is trainable DNN parametrized by 0 and z is frozen random seed:
( = IP) - yll
(44)
The noise is again set to the medium level, i.c., additive Gaussian with 0 mean and 0.18 variance. We our ES- WMS for DIP on the super-resolution dataset used in the original DIP paper Ulyanov et al. 2018). The PSNR gaps are < 1.00 and the SSIM gaps are S 0.05 for most cases (see Tab. 9). Our ES WMV is again able to detect near-peak performance for most images.
Table 9: Detection performance of DIP with ES-WMV for 4x image super-resolution: mean and (sid). PSNR gaps below 1.00 colored as red; SSIM gaps below 0.05 are colored as blue. (D: Detected)
PSNR(D)
PSNR Gap
SSIM(D)
SSIM Gap
Bahoon
17.82 (0.00)
0.10 (0.04)
0.38 (0.00)
0.01 (0.01)
19.93 (0.05)
(0.01)
(.59)
0.01 (0.00)
Bridge
18.04 (0.04)
0.33 (0.09)
0.43 (0.00)
0.00 (0.00)
Coastguard
20.76 (0.05)
0.17 (0.13)
0.33 (0.01)
0.02 (0.01)
Comic
16.70 (0.07)
(0.06)
(0:01)
(0.00)
Face
21.67
0.63 (0.12)
0.56 (0.01)
0.06 (0.01)
Flowers
18.96 (0.08)
0.12 (0.03)
(o.01)
0.02 (0.00)
Foreman
20.62 (0.04)
(0.0)
(0.00)
(0.00)
Lena
22.40 (0.07)
0.30 (0.08)
0.70.00.0)
0.04 (0.00)
Man
19.94 (0.07)
0.22 (0.05)
0.52 (0.00)
0.02 (0.01)
Monarch
19.68 (0:00
1.40 (0.90)
(0.00)
0.03 (0.00)
Pepper
21.20 (0.14)
0.14 (0.04)
0.67 (0.01)
0.04 (0.01)
Ppt3
17.55 (0.10)
0.19 (0.10)
0.71 (0.01)
0.01 (0.00)
19.09 (0.08)
(0.05)
0.56 (0.01)
0.01 (0.01)
27

---

Under review as a conference paper at ICLR 2023
A.7.11 ES-WMV vs. ES-EMV
We now consider our memory-efficient version (ES-EMV) as described in Algorithm 2, and compare with ES-WMV, as shown in Fig. 25 Besides the memory benefit, ES-EMV runs around 100 times faster than ES- WMV, as reported in Tab. 2 and does seem to provide a consistent improvement on the detected PSNRs for image denoising tasks on NTIRE 2020 Real Image Denoising Challenge (Ab- delhamed et al. 2020), PolyU dataset Xu et al. (2018) and the classic 9-image dataset (Dabov et al., 2008) (see [Tass. an TT and 25). due the strong smoothing effect (we set a 0.1). this paper, we prefer to stay simple and leave systematic evaluations of these variants for more tasks as future work.
Table 10: Detection performance comparison between DIP with ES-WMV and DIP with ES-EMV for real image denoising on 1024 images from RGB track NTIRE 2020 Real Image Denois-
ing Challenge (Abdelhamed al. 2020): mcan and (sid). Higher PSNR and SSIM are in red. (D:
Detected)
PSNR(D)-WMV
PSNR(D)EMV
SSIM
SSIMIDJEMV
DIP (MSE)
34.04 (3:68)
34.96 (3.80)
0.92 (0.07)
0.93 (0,07)
DIP (ε1)
33.92 (4:34)
34.83 (4:35)
0.93 (005)
0.94 (0.05)
(Huber)
(3.86)
34.72 (4.04)
(006)
(0.06)
Detection performance comparison between DIP with ES- WS-WWV and DIP with ES-EMV for real image denoising on the PolyU dataset Xu et al. (018): mean and (std). Higher PSNR and SSIM are in red. (D: Detected)
Table 11:
PSNR(D)-WN)
SSIM(D)-WMV PSNR(D)-MV
SSIM(D-EMV
DIP (MSF)
36.83
37.32
0.98
0.98 (0.05)
DIP (6,)
36,20 (2.11)
36.43 (3:22)
0.97 (0.02)
0.97 (0.02)
DIP (Hober)
(2.96)
37.21 (3.19)
(0.02)
0.98 (0.02)
WS-WMV ES-EMV
34
32
28
26
20
Houw
Hosse (E
Poppors (L)
Peppors (1))
Lewa (h.
Lesa (H)
Daboaa
Bahsan (H))
(L6
Fi6 (1f)
Kodbkl (L)
K (ff)
Kodak? Kodbl2 (L)
Kodaki Kodald (L)
di
Kotal15 (t.
(H)
Kobk (80)
Figure 25: Detected PSNR comparison between DIP with ES-WMV and DIP with ES-EMV on the
classic 9-image dataset (Dabov al. 2008).
A.7.12 BLIND IMAGE DEBLURRING (BID)
In this section, we systematically test our ES WMV and VAL on the entire standard Levin dataset for both low-level and high-level cases. We set the maximum number of iterations as 10, 000 to ensure that we perform sufficient optimization. The detected images of our ES- WVV are substantially better than those of VAL., shown in Tab. 12
28

---

Under review as a conference paper at ICLR 2023
Table 12: BID detection comparison between ES- ES-WMV and VAL on the Levin dataset for both low- level and high-level noise: mean and (sad), Higher PSNR is red and higher SSIM in bluc. (D: Detected)
Low Level
High Level
PS
SSIM(D)
PSNR(D)
SSIM(D
WMV
28.54)).61)
0.83(0.04)
26.41(0.67)
0.76/0.04)
18.87(1.4)
16.69(1.39)
0.44/0.10)
A.7.13 ABLATION STUDY
We vary the window size W (default 100) and patience number P (default: 1000) across range and check how the detection gap changes for Gaussian denoising with medium-level noise on the classic 9-image dataset (see Fig. 26
0.35
0.35
0.25
0.25
SSIM-GAP
0.20
0.20
5SIM-GAI
0.15
0.15
0.10
0.05
0.05
0.00
0.00
-0.05
-0.05
-0.10
-0.10
1000
2000
5000
100
300
500
Putience number
Window size
Figure 26: Effect patience number and window size on detection in terms of SSIM gaps
A.8 ANALYSIS FAILURE CASES IN FIG. 8
We note that there are some occasional failures cases when applying our ES on some DIP variants Fig. 8 In this section, we provide VAR curves of these cases. For the failure of GP-DIP on the "House (L)" image in Fig. 8 GP-DIP has weird multi-valley, gradual descending pattern in the VAR curve, corresponding to a multi-peak, gradual ascending pattern the PSNR curve. The first major valley in the VAR curve is roughly aligned with the first major peak, the final best peak, in the PSNR curve. So although our valley-detection method successfully detects the first major valley, the PSNR gap is relatively large. Overall, although our ES method works well with GP-DIP for most of the test cases, we would not recommend GP-DIP for practical use. The concern is the speed: as method trying to mitigate the overfitting, the best reconstruction of GP-DIP tends to be around the very last iterates. The failure on the "Lena(L)" image is due to similar multivalley pattern in the VAR curve.
For both cases, we observe that using smaller learning rates for GP-DIP and DD helps to smooth their curves and mitigate the multi-valley phenomenon which likely will lead much smaller detection gaps. We hesitate to refine in this direction, as our focus of this paper is on the ES method itself.
29

---

Under review as a conference paper at ICLR 2023
Deep Decoder
GP-DIP
3000
3003
2.000
2000
5600
1500
20000
12504
15009
37500
2000
Earatio
Figure 27: VAR curves failure cases. Left: DD for "Lena(L)"; Right: GP-DIP for "House(L)" in
Fig. 8
30


Reviews:
[{"review_id": "s0lUpWNRuNY", "reviewer": "Reviewer_hubG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper solves the early-stopping problem for deep image prior by tracking the variance of reconstructed images within a moving window during training. It shows that early stopping when the variance reaches the minimum results in the best deep image prior. ", "summary_of_the_review": "The paper explores an interesting early-learning phenomenon in DIP and proposes a novel early stopping method to optimize DIP. However, the experiments are not well-organized to support the claims. The organization and writing of the paper need to be improved.", "strength_and_weaknesses": "Strength: \n- The paper demonstrates the early-learning pheromone of the deep image prior both empirically and theoretically. \n- A broad range of experiments are conducted, including different applications (image denoising, deblurring, MRI reconstruction)\n\nWeakness:\n- A sensitivity analysis on hyperparameters (patience number and window size) is conducted, but it's unclear which dataset it that. Figure 3  shows the pattern of training curves looks really different across different noise levels or different DIP methods/variants. Are they using the same hyperparameter?  Will the performance still be stable in that case?\n- The experiments are not systematically conducted. The paper lists a lot of related works, but fails to organize the experiments in a logical way. Some of them are benchmarked on one dataset, while others are on another. What DF-STE is? Why has it only experimented on the nine-image dataset, not CBSD68? \n- The improvement compared to SB is marginal. The gaps are all within one standard deviation.\n- Lack of other denoising baselines, like noise2self, self2self and etc.\n- How does the training loss look compared to the other training curves? Does the low variance part look flat? Can you tell anything about early-learning from the loss curve?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "l2Kosx4aNq", "reviewer": "Reviewer_kTJ2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors propose a technique to detect peak performance without a reference for Deep Image Prior (DIP) based computational imaging algorithms.  The proposed strategy utilizes the running variance of past reconstruction iterates to determine when to terminate training. Authors provide theoretical results to justify the proposed early stopping criterion. Through numerical experiments, they demonstrate that the proposed method can detect peak performance reasonably well across a wide range of inverse problems and types of measurement noise.", "summary_of_the_review": "Overall, in its current form I would recommend borderline rejecting the paper. The proposed method shows promise and works reasonably well across different types of additive noise. However, in my opinion the justification for the particular early stopping metric is somewhat lacking. The provided theory doesn't provide a satisfying answer either. Could the authors provide more explanation and further backing why running variance would show the U-shape phenomenon close to the ground truth signal? Furthermore,  the experimental results seem to show that the proposed technique is mostly reliable in high-noise scenario and a detailed investigation with respect to noise level limitations is missing. Could the authors provide more discussion on the aforementioned limitations?", "strength_and_weaknesses": "Strengths: \n\n-  The paper is very well-written and organized. It is easy to follow and the structure is logical. \n\n- The problem the paper is investigating is a very relevant one and therefore this paper is well-motivated.  \n\n- The proposed technique can be combined with any DIP-based method, is simple to implement and the overhead in terms of computation is minimal.  Therefore, it has potential in practice.\n\nWeaknesses:\n\n- I am not completely satisfied with the justification of tracking running variance to detect peak performance. The theoretical justification doesn't seem to be too convincing, as the provided bounds are very loose, and having a U-shaped upper bound does not necessarily mean that the running variance will also be U-shaped. Even though the experiments show that the metric works in some cases, it is not clear to me why variance should be minimized close to the solution and increase during overfitting. \n\n- Based on the experimental results, the proposed technique is somewhat inaccurate in low-level noise scenarios (Figure 3 top row) and often surpassed by other methods (Table 3, Figure 6, Figure 7). A more in-depth study on the noise regime where the proposed method is reliable would be very useful.\n\n- The proposed method has a significant memory overhead. Storing W=100 iterates in some applications may be prohibitive. The proposed EMV direction is interesting and can potentially address this issue, however currently there are not enough experiments on this variant to fully support its merit. \n\n- Most experiments are on very small datasets (1-9 images). Evaluation on larger datasets would help filtering out statistical variations in the presented results. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HEj_Ni7zcJ", "reviewer": "Reviewer_sEH9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an Early Stopping (ES) method for Deep Image Prior (DIP) based on the running variance of the intermediate outputs. For the task of denoising, a broad set of experiments is provided to study it under different noise types and intensities and against other approaches for early stopping. Besides, some experiments showing how it can be utilized in blind image deblurring and MRI reconstruction are provided.", "summary_of_the_review": "The empirical results seems to be well-aligned with the justification of the proposed method. The performance is generally improved on denoising compared to previous ES for DIP methods and the experiments on this task are extensive. There are some situations such as too much high-frequency components or too much smoothness in the optimization curve that could affect the result negatively. While the idea of tracking the running variance of the outputs seems to be simple, I have not seen it used for ES in DIP. The significance cannot be properly measured when only looking at family of DIP methods and not including other SOTA methods for denoising. The situation is worse on the other tasks as the comparisons are far more limited and does not even include latest methods based on DIP, for example for deblurring, not to mention the rest of SOTA methods.", "strength_and_weaknesses": "- S1: The motivation and justification of using the proposed metric for ES is discussed in details.\n    \n- S2: Providing extensive experiments on denoising and showing robustness to different noise types and higher intensities of noise.\n    \n- S3: Outperforming non-reference based image quality metrics and other methods of ES for denoising specifically designed for DIP either in terms of average quality or wall-clock time per iteration\n    \n- S4: Proposing a lighter variant of the algorithm to save on computations per iteration\n    \n- W1: The method does not perform well when an image consists of too much high-ferequency components.\n    \n- W2: Based on Figures 3, 8a-b, and 17, it seems like in practice the method is prone to higher error or failure when the changes in the PSNR curve are relatively smooth or when the noise level is low.\n    \n- W3: The experiments are imbalanced and mainly focused on denoising. For example, for blind image deblurring a limited subset of the dataset and/or kernels are considered and the the results in case of the default noise levels used in the original Ren et al. (2020) solution are not provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6XAbEsI9Lfj", "reviewer": "Reviewer_cx5R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an early stopping strategy for deep image prior.  This is achieved by using an efficient ES strategy that consistently detects near-peak performance across several CI tasks and DIP variants. The paper provides high level intuition, theoretical proof and multiple use cases to demonstrate the proposed method. Also, the limitation is well discussed. ", "summary_of_the_review": "Overall, the paper is good for me but I just want to know the answer to the points listed in weakness.  Other than that, the paper is a good contribution to improving the DIP if it is competitive with the SOTA inverse problem solutions ", "strength_and_weaknesses": "- Strength\n1. The paper is well-written with enough details \n2. High-level intuition is great and easy to follow \n3. Theoretical analysis is provided\n4. Multiple use cases in DIP are discussed\n5. The limitation is shown in different scenarios. \n\n- Weakness \n1. The theoretical analysis is not strongly related to the strategy, can you explain why the theoretical helps the early stopping criteria? \n2. The experiments focus on image denoising, MRI reconstruction, and blind deblurring, why choose these three? is that possible to solve other general inverse problems, e.g., superresolution, implanting? \n3. Is that early stopping DIP competitive with the SOTA approaches, e.g., diffusion-based models in these tasks? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5Wr0sqzRzBQ", "reviewer": "Reviewer_9Jvw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an early stopping mechanism for DIP based on the windowed moving variance of the output images. An early stop is detected when the variance does not decrease after a certain number of training steps. The proposed method is observed to have small PSNR and SSIM gaps compared to other early stopping methods and is applicable to different DIP variants.", "summary_of_the_review": "I hold an overall positive view on this work for its technical novelty and empirical contribution.\n\n===================================\n\nUpdate after the rebuttal discussion:\n\nAfter reading comments from other reviewers and the feedback from the authors, I can relate to the concerns of other reviewers that the ad-hoc manner of the proposed method, especially on the hyperparameter configuration of the patience time, creates another \"early-stopping\" problem. This issue is more severe considering the lack of theoretical analysis and essentially undermines the practicality of the proposed method. Co-considering this and the underwhelming performances from case to case, I decided to adjust my evaluation negatively towards this work.", "strength_and_weaknesses": "Strength: The proposed method is simple enough and to some extent aligned with the reconstruction quality. Extensive experiments do show the superiority of the the method. And the proposed method is generally applicable to various types of noises.\n\nWeaknesses:\n\n1. In some cases, the PSNR gaps are still very high, up to 5dB. And according to the plots of variance curves, the proposed metho still detects early stopping time that is way off the correct time (with highest PSNR).\n\n2. I strongly recommend the authors to provide some output images to be included in the manuscript, since PSNR is not perfectly aligned with visual quality some times (usually have some smoothening effect). Moreover, the NF-IMQ reference baselines are naturally biased towards better \"quality\" in their metrics instead of PSNR. Therefore, solely comparing PSNR-gaps might not be quite fair.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject