Image Reconstruction via Deep Image Prior Subspaces
Authors: Riccardo Barbano
Javier Antoran
Johannes Leuschner
José Miguel Hernández-Lobato
Bangti Jin
Zeljko Kereta
Venue: Accepted by TMLR ()

Abstract: Deep learning has been widely used for solving image reconstruction tasks but its deployability has been held back due to the shortage of high-quality paired training data. Unsupervised learning methods, e.g., deep image prior (DIP), naturally fill this gap, but bring a host of new issues: the susceptibility to overfitting due to a lack of robust early stopping strategies and unstable convergence.
We present a novel approach to tackle these issues by restricting DIP optimisation to a sparse linear subspace of its parameters, employing a synergy of dimensionality reduction techniques and second order optimisation methods. The low-dimensionality of the subspace reduces DIP's tendency to fit noise and allows the use of stable second order optimisation methods, e.g., natural gradient descent or L-BFGS.
Experiments across both image restoration and tomographic tasks of different geometry and ill-posedness show that second order optimisation within a low-dimensional subspace is favourable in terms of optimisation stability to reconstruction fidelity trade-off.

Published in Transactions on Machine Learning Research (02/2024)
Image Reconstruction via Deep Image Prior Subspaces
Riccardo Barbano
riccardo. viccardo.barbane barbano.....................bb.b 1 19 uk
Deportment of Computer Science
University College London
Javier Antorán
ju666@com.acc.u
Department of Engineering University of Cambridge
Johannes Leuschner University of Bremen
Jeusckn@@uni-bremen..d
José Miguel Hernández-Lobato
jmb233@cam.ac.uk
Department of Engineering University of Cambridge
Bangti Jin
b.jin edu.hk
The Chinese University of Hong Kong
Zeljko Kereta
z.kertta@ucll. uk
Department of Computer Science University College London
Reviewed on OpenReview: https://opeereeeee net/ forum? id= torWsBui9N
Abstract
Deep learning has been widely used for solving image reconstruction tasks but its deployability has been held back due to the shortage of high-quality paired training data. Unsupervised learning methods, e.g., deep image prior (DIP), naturally fill this gap, but bring host of new issues: the susceptibility to overfitting due to lack of robust early stopping strategies and unstable convergence. We present novel approach to tackle these issues by restricting DIP optimisation to a sparse linear subspace of its parameters, employing synergy of dimensionality reduction techniques and second order optimisation methods. The low- dimensionality of the subspace reduces DIP's tendency to fit noise and allows the use of stable second order optimisation methods, e.g., natural gradient descent or I-BFGS. Experiments across both image restoration and tomographic tasks of different geometry and ill- pol-posedness show that second order optimisation within low-dimensional subspace is favourable in terms of optimisation stability to reconstruction fidelity trade-off.
1 Introduction
Deep learning (DL.) approaches have shown impressive results wide variety of linear inverse problems imaging, e.g., denoising (Tian al. 2020), super-resolution (Ledig et al. (2017 Ulyanov et al. 22220), magnetic resonance imaging (Zeng ct al. 2021 and tomographic reconstruction [Wang et al. 2020). Mathematically, linear inverse problem of recovering unknown image dd from measurements y E Bdy is formulated
y = Ax +
(1)
for (ill-conditioned) matrix A Xd, and exogenous noise

---

Published in Transactions on Machine Learning Research (02/2024)
However, conventional supervised DL approaches not ideally suited for practical inverse problems. Large quantities of clean paired data, typically needed for training, are not available in many problem domains, e-g., tomographic reconstruction. Moreover, the ill-posedness (due to the forward operator A and noise e) and high-dimensionality the images z pose significant challenges, and can be computationally very demanding. Whereas standard imaging lasks, e.g,, denoising and deblurring, use high dimensional observations (dg R dy), tomographic imaging often requires reconstructing an image from observations that are of a much lower dimensionality compared to the sought-after images. For example, reconstructing CT of walnut may require reconstructing from observations that are only 3% of the size of the original image.
Unsupervised DL approaches do not require paired training data, and have received significant attention in the imaging community. Among these approaches, DIP (Ulyanov et al., 2018) has garnered the most traction. DIP parametrises the reconstructed image as the output of a convolutional neural network (CNN) with a fixed input. The reconstruction process learns low-frequency components before high-frequency ones Shi et al., 2022), which can act as form of regularisation.
(Chakrabarty & Maji 2019:
Alas, the practicality of DIP is hindered by two key issues. Firstly, each DIP reconstruction requires training the entire network anew. This can take from several minutes up to couple of hours for high-resolution images (Barbano et al. 2022b). Secondly, DIP requires careful early stopping (Wang et al. 2021 to avoid overfit overfitting, which is often based on case-by-case heuristics. However, validation-based stopping criteria are
often not viable in the unsupervised setting as they violate ii.d. assumptions, see Wang et al. (0021).
This paper aims to address both of the existing issues inherent to DIP, and to illustrate the approach on image restoration and reconstruction. Building upon recent body of evidence which shows that neural network
(NN) training often takes place in low-dimensional subspaces (Li ct al.) 2018 Frankle & Carbin 2019 Li
et al. 2023), we restrict DIP's optimisation to sparse linear subspace of its parameters. This has a two-fold beneficial effect. First, subspace optimisation trades off some flexibility in capturing fine image structure details for robustness to overfitting. This extraordinarily well suited in imaging problems belonging to inherently lower dimensional structures, but it is also shown to be competitive in restoring natural images. Moreover, allows using stopping criteria based on the training loss, without sacrificing performance. Second, the low dimensionality induced by the subspace allows using second order optimisation methods (Amari 2013 Martens & Grosse 2015a).. This greatly stabilises the reconstruction process, facilitating the use of simple loss-based stopping criterion, and reduces the nunber of iteralions to convergence.
Our contributions can be summarised as:
We extract a principal subspace from DIP's parameter trajectory during a synthetic pre-training stage. To reduce the memory footprint of working with non-axis-aligned subspace, we sparsify the extracted basis vectors using top-k leverage scoring.
We use second order methods: natural gradient descent (NGD) and limited-memory Broy- den Fletcher Goldfarb Shanno (L-BFGS), to optimise DIP's parameters in a low-dimensional subspace.
We provide thorough experimental results across several linear inverse problems, comprising image restoration and tomographic tasks of different geometry and degree of ill-posedsess, showing that subspace methods are favourable in terms of optimisation stability reconstruction fidelity trade-off.
2 Related Work
The present work lies at the intersection of overfitting in unsupervised methods, subspace learning and second order optimisation. Below we discuss recent advances the related fields.
Since the introduction of DIP Ulyanov et al. 2018 2020 stopping optimisation before overfitting to noise has been necessity. The analysis in Chakrabarty & Majil (2019)
Avoiding overfitting in DIP:
and (hh al.] (2022) elucidates that U-Net is biased towards learning low-frequency components before high-frequency ones. The authors suggest sharpness- stopping criterion, which however requires
2

---

Published in Transactions on Machine Learning Research (02/2024)
modified architecture. Jo et al. (0211) propose criterion based the Stein's unbiased risk estimate (Eldar
2008) for denoising, which however performs poorly for ill-posed settings (Metzler et al. 2018 Wang el al.
(2021) propose a running image-variance estimate as a proxy for the reconstruction error. Our experiments find this method somewhat unreliable for sparse CT reconstruction. Ding et al. (2022) and Yaman ct al. (2021 propose to split the observation vector into training and validation sub-vectors, and use the loss lunctional on the latter as a stopping criteria. Unfortunately, this violates the i.i.d. data assumption that underpins validation-based early stopping (Shalev-Shwartz & Ben-David 2014 Theorem 11.2). Independently, [Lu et al. (2019) and Baguer et al. (2020) add TV regulariser to the DIP objective 2 This only partially alleviates the need for early stopping and has seen widespread adoption. To the best of our knowledge, the present work is the first to successfully avoid overfitting without significant performance degradation.
Linear subspace estimation: Literature on low-rank matrix approximation is rich, with randomised SVD approaches being the most common (Halko et al. 2011 Martinsson & Tropp, 2020). However, in high- dimensions, even working with a small set of dense basis vectors can itself be prohibitively expensive. Matrix sketching methods (Drineas et al. 2012 Liberty 2013) alleviate this through axis-aligned subspaces. To the best of our knowledge, our work is the first to combine these two method classes, producing non-axis-aligned but sparse approximations.
Optimising neural networks in subspaces: Closely related to the present work is that of Li et al. (0018) and Wortsman et al. (2021), who find that networks can be trained in low-dimensional subspaces of the parameters without loss of performance, and that more complex tasks need larger subspaces. Similarly to our methodology, Li et al. (2023) identify subspaces from training trajectories and observe the resulting robustness to label noise. Results reported in Frankle & Carbin (2019) show that pruning a very large number of parameters in fully-connected and convolutional feed-forward networks yields trainable sub-networks that can achieve performance comparable to that of the original network. This principle has yielded speedups in work evaluation (Wen et al. 2016 Daxberger al., 2021). [hwarrzzZiv et al. (2022) use a low-rank estimate of the curvature around an optimum of a pre-training task to regularise subsequent supervised learning.
Second order optimisation for neural networks: Despite their adoption in traditional optimisation (Liu & Nocedal 1989), second order methods are rarely used with neural networks due to the high cost of dealing with curvature matrices for high-dimensional functions. Martens & Sutskever 2012 use truncated conjugale-gradient to approximately solve against a network's Hessian. However, a limilation of the Hessian is that it is not guaranteed to be positive semi-definite (PSD). This is one motivation for NGD (Foresee &
Hagan 1997 Amari 2013: Martens 2020). that uses the FIM (guaranteed PSD). Commonly, the KFAC approximation (Martens & Grosse 2015a) is used to reduce the costs of FIM storage and inversion. Also, common deep-learning optimisers, e-g., Adam (Kingma & Ba 2015 or RMSprop (Hinton et al. 2014] may be interpreted as computing online diagonal approximations to the Hessian.
3 Deep Image Prior
DIP expresses the reconstructed image x f(xo 9) in terms of the parameters 0 Rd of a CNN f dd x de + Rd+ and fixed input xo Id= The parameters 0 are learnt by minimising the loss
(()) = [[ 0) v12 + (TV 0)),
(2)
composed of a data fidelity and total variation (TV), weighed by A > 0. TV is the most popular regulariser for image reconstruction (Rudin et al.
1992: Chambolle et al., 2010). Its anisotropic version is given by
TV(x) = Σ -Kkg-Xi1hh + Σ X - X1, f.
1
where X E hx is a vector x E Rdd reshaped into an h x w image, and d,=h- - w. In this work, f is a fully convolutional U-Net, see C.3 for more details. This implicitly regularises the reconstruction by preventing overfitting to noise, long the optimisation is stopped early enough.
3

---

Published in Transactions on Machine Learning Research (02/2024)
DIP optimisation costs can be reduced by pre-training synthetic data. E-DIP (Barbano et al 2022b first generates samples from a training data distribution P of random ellipses, and then applies the forward The network input is set as the filtered back-projes pack-projection (FBP) 20 = xx = A,y, where At denotes the (approximate) pscudo-inverse of A. The pre-training loss given by
model A and adds white noise, following ①
( = Ez.g-=lff(cc -
x1:2:
(4)
where P denotes the (empirical) joint distribution between the ground truth x and the corresponding noisy data. The pre-trained network can then be fine-tuned on any new observation y by optimising the objective with FBP as the input. E-DIP decreases the DIP's training time, but can increase susceptibility to overfitting, making early stopping even more critical, cf. discussion in Section
4
Methodology
We now describe our procedure for optimising DIP parameters in subspace. We first describe how the F-DIP pre-training trajectory is used to extract a sparse basis for low-dimensional subspace of the parameters. The objective is then reparametrised in terms of sparse basis coefficients. Finally, we describe how L-BFGS and NGD are used to update the sparsified subspace coefficients.
Step - Identifying the sparse subspace: First, we find a subspace of parameters that is low- dimensional and easy lo work with, but contains a rich enough set parameters to fit to the observation y. We leverage E-DIP pre-training trajectory to acquire basis vectors by stacking dpre parameter vectors, sampled uniformly spaced chockpoints on the pro-training trajectory, into matrix @pre E de We then compute top-dsub SVD of Ore x USVT, and keep the left singular vectors U Rd where dsub S dpre is the dimensionality of the chosen subspace. We then sparsify the orthonormal basis U by computing leverage scores (Drineas et al. 2012), associated with each DIP parameter as
den's
Σ UR.
i= 1= do.
We keep only the basis vector entries corresponding to diev < da largest leverage scores. This can be achieved by applying a diagonal mask M E (0,1]dexds satisfying [M] = 1 E arg top-dhev f), where (l., ld The sparse basis MU contains at most diev dsub non-zero entries.
Pre-training and sparse subspace selection are only performed once, and can be amortised across different reconstructions. We choose dpre x 103, resulting in large memory footprint of the matrix @rr,, though this is stored in cpu memory. Alternatively, incremental SVD) algorithms (Brand 2002) can be used to further reduce the memory requirements (sce Appendix A). Training DIP in the sparse subspace requires storing only the sparse basis vectors MU in accelerator (gpu / (pu) memory. Thus, sparsification allows training in relatively large subspaces daub > 10 of large networks dg 107.
Step 2 Network reparametrisation:
We write the objective L()) terms of sparse basis coefficients
Rd as
(c) = (((()), with 7()) = grre + MUc.
(5)
This restricts the DIP parameters gore (from pre-training) to change only along the sparse subspace MU. The coelficient vector e is initialised as sample from uniform distribution on the unit sphere.
Step 3 - Second order optimisation: The subspace reparametrised objective L. in (5) differs from traditional DL loss functions in that the melhod-llpenndent local curvature mairix of C.y can be computed and stored accurately and efficiently, without resorting to restrictive approximation its structure, such as the often used diagonal or KFAC approximations (Martens & Grosse 2015b). These facts open the door second order optimisation of C,, which may converge faster than first order methods. However, repeatedly evaluating second order derivatives of neural networks has prohibitive cost, further compounded by the rapid
4

---

Published in Transactions on Machine Learning Research (02/2024)
change of the local curvature of the non-quadratic loss when traversing the parameter space. We mitigate this by performing online low-rank updates of a curvature estimate while only accessing loss Jacobians. In particular, we use L-BFGS (Liu & Nocedal 1989) and stochastic NGD (Amari, 1998 Martens 2020) in the experimental evaluation. The former estimates second dircctional derivatives by solving the socant equation. The latter keeps an exponentially moving average of stochastic estimates of the Fisher information matrix (FIM).
NGD for DIP in subspace:
The exact FIM is given
(6) E Rxdes the Jacobian of the subspace loss C. the current coefficients (with Ta, RR.. being the identity matrix), and Jj == Vof(at, (e)x=)()) E TR Xdo is the neural network Jacobian at 0= (c)). Note that TV's contribution is omitted since the FIM is defined only for terms that depend on the observations and not for the regulariser (Ly et al. 2017). At step we estimate the FIM by the Monte-Carlo method as
E E (l))) Vec,,.
where VcC,(c) (Af
F.
E AJ) 2f AJ,MU, with 2, (a,,
(7)
The moving average of the FIM is updated as
F+ BF, (1-+ with B E (0,
(8)
Our implementation of NGD follows the approach of Martens & Grosse 2015a)), with adaptive damping, and step size and momentum paramcters chosen by locally minimising quadratic model. See Appendix B for additional discussion.
Note that he methodology presented in this section can be extended beyond the DIP framework, in the spirit of lest-time fine-tuning strategies of learned reconstruction methods (Darestani el al. 2022]. Here we offer an alternative for adapting reconstruction algorithms at test-time, by introducing a robust optimisation framework.
5
Experiments
Our experiments cover a wide range of image restoration and tomographic reconstruction tasks. In Section 5.1 we conduct an ablation study on CartoonSet (Royer et al. (017). examining the impact of subspace dimension deub, subspace sparsity level diev, degree of problem ill-posedness, and choice of the optimiser on reconstruction speed and fidelity. The acquired insights are then applied to challenging tomography tasks and image restoration. We compare the reconstruction fidelity, propensity to overfitting, and convergence stability relative to vanilla DIP and E-DIP on a sea-ures reslutoson CT scan of walmut, in Section 5.2 and on medically realistic high-resolution abdomen scans from the Mayo Clinic, in Section 5.3
We simulate observations using T with dynamically scaled Gaussian noise given by
C~N(0, o²Ta,) with p/dg
(9)
with the noise scaling parameter set to p 0.05, unless noted otherwise. We conclude with denoising and deblurring on a standard RGB natural image datasct (Set5) in Scction 5.4 For studics in Sections 5.2 5.3 and
5.4 we use a standard fully convolutional U-Net architecture with either ~ 3M (for CT reconstruction lasks)
or ~ 1.9M (for natural images) parameters, see architectures in Appendix C.3. For the ablative analysis in Scction 5.1 we use a shallower architecture (~ .5M) with only 64 channels and four scales, koeping the skip connections in lower layers. Following the literature, we train the vanilla DIP (Ulyanov et al, 2018) (labelled
DIP) and E-DIP (Barbano et al. 2022b) with ADAM. We train subspace coefficients with Adam (Sub-DIP

---

Published in Transactions on Machine Learning Research (02/2024)
Ground truth
DIP max
E-DIP max
E-DIP last
Sub-DIIP NGD last
PSNR: 34.35 dB
PSNR: 33.99 dB
PSNR: 32.55
PSNR: 33.76 dB
Figure 1: Reconstruction comparison for an example CartoonSet. image from 45 angles. "max" indicates oracle (highest possible) PSNR, which we note is only available a ground truth image exists. "last" denotes the final reconstruction.
non-sparse subspace size comparison
sparsity level comparison
test-time vs conv. PSNR
33.0
33
32.5
31.5
PSNR (cBB
32
32.0
31.5
51.0
3000
30.5
29.5
29
3000
500
10:0
1500
10
10-2
10-1
subspace dimension duub
proportion of weights kept dus / dg
time to conv. PSNR (s)
Parelo fronlier
Sub-DIPP Adam Sub DIP L-BPGS
DEP conw.
Sub DIP max Sub-DIP come.
Sub(1800) DIP max
Sub/1200) DIP max
DEP
E-DIP max
Sub((800)-DDP comy.
Sub(12000-DDP conv.
E-DIP
DIP NGD
The influence of subspace dimension daub (left), sparsity level dw//do (middle) on PSNR, and the PSNR vs time Parcto-frontier (right) for CartoonSet. The dashed line defines a Parcto-frontier, "max" refers the oracle stopping PSNR, while "conv." refers the PSNR the stopping point found by applying Algorithm 1 cf. Appendix A, to (2), with 8=0.995 and patience of p=100. Left and middle plots use NGD. Results show mean and standard deviation over 25 reconstructed images from 95 angles. Runs are performed on A100 GPU. Note that the PSNR line for DIP conv., in the leftmost panel, is almost completely obscured by F-DIP conv. PSNR, since they have an almost identical PSNR value.
Figure 2:
Adam), which serves a baseline, L-BFGS (Sub-DIP L-BFGS) and NGD (Sub-DIP NGD). Image quality is assessed through peak signal-to-noise ratio (PSNR).
For tomographic tasks we use the same pre-training runs for E-DIP and all subspace mothods: minimising 44 over 32k images of ellipses with random shape, location and intensity. Pre-training inputs are const from an FBP obtained with the same tomographic projection geometry as the dataset under consideration. Analogously, for image restoration tasks, we pre-train on ImageNet (Deng al. 2009).
The method is built top of the E-DIP library ggithub. com/educattiing-ddppp and data are available at gggttuu....m. com/subspaceedd1
The full implementation
5.1 Ablative analysis on CartoonSet (Royer et al.) 2017)
We investigate Sub-DIP's sensitivity to subspace dimension d sparsity level diev, and ill-posedness on 25 images of size (128 px)² from CartoonSet (Royer et al.) 2017 google.github...
github. ho//aartonset)) Example reconstructions are shown in Fig. [] We simulate a parallel-bean geometry with 183 detectors and 45, 95 or 285 angles, corresponding to, respectively, a very sparse-view (d,=5,35), moderately sparso-view (d,_=1385), and a fully sampled setting 5d,555)155. We construct subspaces by sampling d2 parameters

---

Published in Transactions on Machine Learning Research (02/2024)
at uniformly spaced checkpoints during 100 pre-training epochs ellipses in Barbano et al. 2022b). We
measure the degree of overfitting by comparing the highest PSNR obtained throughout optimisation (max)
with that given by Algorithm | applied to the training loss (2) with 6=0.995 and p-100 steps (conv.).
In Appendix we report results of additional ab- lation experiments on the extracted subspaces, ex- amining their adaptability changes in the forward operator, and comparing them to random subspace bases.
45 angles
95 angles
285 angles
30.5
33.5
37.5
30.0
33.0
29.5
32.5
PSNR (cB)
32.0
36.5
29.0
31.5
36.0
Subspace dimension: Fig. (left)) explores the trade-off between subspace dimension daub and re- construction quality. We use 95 angles, subspace methods with no sparsity (de and the NGD optimiser. Both standard DIP and E-DIP overfit, showing a R 3 dB gap between max and conv. PSNR values, while subspace methods exhibit only a = 0.5 dB gap. Both max and conv. PSNR present mono- tonically increasing trend with subspace dimension, while the gap at dsub > 14 stays roughly constant at. ~022 dB, Thus, these subspace dimensions are too small for significant overfitting to occur. In spite of this, 1sub=100 is enough to obtain better conv. PSNR than DIP and E-DIP.
28.5
31.0-
28.0
30.5
35.5
conv
3000
35.0
X
max
29.5
DIP
D-DI
Sub-DIP Adam
Sub-Dif L-BEGS
Sub-DIP NGD
Figure 3: PSNR mean and standard deviation over 50 CartoonSet images from 45, 95, and 285 angles.
Subspace sparsity level: Fig. 2 (middle) shows that for diw/dd > 0.01, the reconstruction fidelity is largely insensitive to the sparsity level. This is true for both daub=1200 and 1100= This effect is also somewhat independent of the subspace dimensionality dsub. Hence, sparse subspaces should be constructed by choosing a large daub and then sparsifying its basis vectors ensure computational tractability.
Problem ill-posedness: We report the perfor- mance with varying numbers of view angles in Fig. with non-sparse subspaces of dimension 1800= The smaller is the number of view angles, the more ill-posed the reconstruction problem becomes. Sub-
45 angles
angles
285 angles
space methods present a smaller gap between max and conv. PSNR (~ 0.25 dB) than DIP and E-DIP (~ 2.5 dB). Subspace methods also present better fidelity at convergence for all the studied number of view angles. Notably, the improvement is larger for more ill-posed settings (~ 2.5 dB at 45 and 95 R 1 dB at 285 angles), despite the worsening per- formances for all methods in sparser settings. This is expected as there is a reduced risk of overfitting in data-rich regimes and more flexible models can do better. E-DIP's max reconstruction fidelity is consistently above that of other methods by least
PSNR (dB)
30
29
27
28
30
27
26
0
20g
20g
200
time (s)
time (s)
time (s)
DU
Sub-DIF L-BIGS
E-DIP
Sub- NGD
Sub DIP Adam
Figure 4: Average PSNR trajectories for 45, 95, and 0.5 dB. This may be attributed to full-parameter 285 angles over 50 reconstructions.
flexibility with benign inductive biases from pre-training. However, obtaining max PSNR performance requires oracle stopping and is not achievable in practice. In Fig. we show PSNR trajectories of the studied moth (DIP, E-DIP, Sub-DIP Adam, Sub-DIP LBFGS, and Sub-DIP NGD) for 45, 95, and 285 angles, averaged over 50 reconstructed images. This figure provides additional evidence supporting previous obsorvations. While DIP and E-DIP tend overfit to noise once they reach their maximum PSNR values, subspace methods consistently maintain stable reconstruction performance with no noticeable performance degradation, due to the inherent regularising effect obtained by restricting to subspace.
7

---

Published in Transactions on Machine Learning Research (02/2024)
First second order optimisation: We compare optimisers' conv. PSNR their time to convergence. Fig. (right) shows that Sub-DIP T-BFGS and NGD converge in less than 50 seconds. These methods are optimal along the known Pareto-frontier, reaching ~ 1.5 dB higher reconstruction fidelity than DIP and E-DIP. LBFGS converges in only ~0 seconds. Sub-DIP Adam retains protection against overfitting but converges a rate similar to non-subspace first order methods (in ~ 180 seconds). These trends hold across studied degrees of ill- ill-posodnes;; see Appendix A.1.
Ground truth
DIP max
DIP last
E-DII max
Sub-DIP NGD last
PCT Walmut
PSNR 29.93 dB
PSNR: 26.81 dB
PSNR 29.49
PSNK: 29. dB
Figure 5: Reconstructions of the Walmut using 60 angles. Sub-DIP reconstructions do not capture any noise, but present slightly increased ringing around very thin structures. "max" " indicates oracle (highest possible) PSNR. "last" denotes the final reconstruction.
DIP
E-DIP
moving Var(x
PSNR (dB)
28-
10
10
Algorithm Early stop criterion
Inputs: metric g(-), patience P, de- crease proportion 8
10
Sub-DIP L-BFGS
Sub-DIP NGD
Imin < 00, 5 0, ámin +- 00
30
while < imin + do if g()) Jmin then
PSNR (dB)
Loss C,
28
PSNK
Imin + g()) and imin end
Var(s) or Lues
stupping point
10
6 ++1
end
4000
8000
2000
4000
6000
8000
Output: émin
optimisation time (s)
optimisation time (s)
Figure The evolution of PSNR and stopping metrics (variance- based for DIP and E-DIP, and loss-based for Sub-DIP NGD and L-BFGS) optimisation time Walnut with one seed.
5.2 PCT Walnut (Der Sarkissian et al. 2019
In this section we study the methods on a real-measured high-resolution PCT problem. We reconstruct a (501 2x)² slice from a very sparse, real-measured cone-beam scan of walnut, using 60 angles and 128 detector pixels (dg = 7680). We compare reconstructions against ground-truth (Der Sarkissian et al. 2019 which uses classical reconstruction from full data acquired al 3 source positions with 1200 angles and 768 detectors. This task involves fitting both broad-stroke and fine- -re-ggdieed image details (see Fig, 5), making it good proxy for PCT of industrial context. We pre-train the 3M parameter U-Net for 20 epochs and save dpre=ūk parameter checkpoints. Following Section 5. we construct a dgub=4k dimensional subspace and sparsify it down to 0.5 of the parameters.
Fig. (left) shows the optimisation curves for all optimisers, averaged across sceds. Qualitatively, the results are similar the cartoon data. Second order subspace methods converge their highest reconstruction fidelity within the first 500 seconds and do not overfit. Vanilla DIP and E-DIP also converge quickly but
8

---

Published in Transactions on Machine Learning Research (02/2024)
suffer from overfitting leading to ~3 dB and ~ 1.8 cB of performance degradation, respectively. Sub-DIP Adam takes over 3000 seconds to converge and does not overfit.
Stopping criterion challenges: Capturing the oracle performance of DIP and E-DIP would require robust stopping criterion. Note that we cannot base a slopping criterion on 2 We instead turn to the method from Wang et al. (2021). which minimises rolling estimate of the reconstruction variance across optimisation steps, a proxy for the squared error. Following Wang ct al. (2021) we compute variances with a 100 slep window and apply Algorithm [ with a patience of p=1000 steps and 8=1. Figure f shows this metric to be very noisy when applied to DIP and F-DIP. This breaks the smoothness assumption implicit in the stopping criterion (Algorithm D leading to stopping more than dB before/after reaching the max PSNR. This phenomenon is attributed to the severe ill- ill-posedness of the tasks causing variance across large subspace of reconstructions that fit our observations well. For tomographic reconstruction, the variance curve becomes more non-convex, and its minimum does not correspond to the optimal PSNR. Since subspace methods do not overfit and the loss is smooth, we can use it as our stopping metric (6=0.995, p = 100).
NCT Walnut
Mayo (100 angles)
Mayo (300 angles)
30
36
40.0
34
37.5
PSNN (dB)
32
35.0
32.5
24
28
26
24
25.0
22.5
101
102
30
104
10
102
10a
10+
10
102
10
10
optimisation time (s)
optimisation time (s)
optimisation time (s)
E-MPP
Sub-TMP
S.b-DIPL-BPCS
Sob-1MP NGD
Figure 7: Optimisation curves for the μCT Walnut data (left), and Mayo using 100 angles (middle) and Mayo using 300 angles (right) averaged over 10 images.
5.3 Mayo Clinic dataset (Moen et al., 2021
To investigate a medical setting, we use 10 clinical CT images of the human abdomen released by Mayo Clinic (Moen et al.] 2021) and study the behaviour of subspace optimisation. We reconstruct from a simulated lan-beam projection, with 300 angles and white noise with the noise scaling parameter p = 0.025, cf. (9. As a more ill-posed reference setting, we use 100 angles (comparable sparsity to the Walnut setting) and Gaussian noise with p=0.05. For both tasks, we use the 3M parameter U-Net, as in Section 5.2 pre-trained on 32k ellipse samples. For the sparse setting (100 angles), we use dgub = 4k dimensional subspace constructed from 55 checkpoints, but with sparsity ratio 66m/4400.23 For the more data-rich setting (300 angles), we use dgub= = 8k, sampled from dpce=10k checkpoints, and similarly, we sparsify it down to de.//4.0..25.
In the 300 angle setting, Sub-DIP NGD reaches 37 dB within 200 seconds. PSNR the contimutes to increas, albeit slowly, without performance saturation, cf. Fig. 7 (right). In contrast, for the sparser Walnut and Mayo data, Sub-DIP NGD maintains a steep PSNR increase until reaching max PSNR. Interestingly, L-BFGS does not perform well in the 300 angle setting, obtaining < tendency to stop the iterations too early in high dimensions.
36 dB PSNR. This might be due L-BFGS's
Note that the obscrved time efficiency of the Sub-DIP reported in Fig. 2 is not preserved in high-dimensional CT tasks; see Fig. 7 This behaviour is expected as the efficiency of second order optimisation methods is affected both by the high- -igediimensionality of the measurements -eeauuttntt--ruuuttingngg in costlier forward operator ovaluations and more importantly, by the depth of the U-Net-rcsulting in costlier voctor-Jacobian and facobian-vector products. As discussed, second order methods converge in fewer iterations (see e.g., Fig. 19 in Appendix A) but require higher per-iteration cost.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Ground
max
PSNR: 34.40
PSNR 24.36 dB
PSNR: 34.24 JB
FSNK: 33.83 dB
Mayn anglee
PSN 38.3883333
PSNK dB
PSNN 38 dB
PSNK
Mayo (300 angles)
Figure 8: Example reconstructions on the Mayo dataset 100 (top) and 300 (bottom) angle settings.
However, is worth noting that when working with a limited runtime budget, Sub-DIP (L-BFGS and NGD) methods may take more time to reach their maximum PSNR, but they consistently deliver the highest image quality within a 100-second runtime, as illustrated Fig. 7 At the same time, Sub-DIP methods do not suffer performance degradation due to overfitting to noise. comparison in terms of the number of iterations, instead of optimisation time, be found in the rightmost panel in Figs. 19, 20 and 21 in the Appendix.
Fig. g shows example reconstructions using 100 and 300 angles. If stopped within narrow max PSNR window, DIP and E-DIP can deliver reconstructions that better capture high-frequency details than Sub-DIP methods as expected. However, while DIP and E-DIP reconstructions become noisy once they start to overfit, Sub-DIP methods do not exhibit any noise. We deem the increased robustness es reduced flexibility tradc-off provided by the Sub-DIP to be favourable, even in the well-determined setting.
In Appendix A.3 investigate the effect of the dataset used extract the subspace, in the 300 angle case. Namely, compare the task-agnostic dataset of random ellipses with task-specifie Mayo dataset. As expected, the results show that using a task-specific dataset, matching the target image manifold, improves reconstruction performance, However, the generality of the approach is compromised, since suitable dataset of images specific enough for a task and modality at hand might not always be available and the method starts to resemble supervised learning. In the data-poor settings, using a synthetic dataset provides a good performance compromise.
5.4 Image restoration Set5
We conduct denoising and deblurring on five widely used RGB natural images ("baboon", "jet F16", "house", "Lena", "peppers") of size px)2. The pre- pre-training is done on ImageNet (Deng et al. 2009), a dataset of natural images, which we use extract the basis for the subspace methods.
For denoising we consider four noise settings with the noise scaling parameter p E (0.10, 0.15 0.25, 0.5), cf. (9), We extract single subspace for all the noise levels. To this end, during the pre-training stage, we construct dataset by adding noise to each training datum, with randomly selected noise scaling parameter Uni(0.05, 0.5). Then deub 8k subspace with sparsity level diev/do = 0.25 is extracted from dpre = 10k samples. In the high noise case (p = 0.5), we sub-extract to a smaller danb = 1 subspace. This sub-exlraction is necessary to minimise overfitting high noise level scenarios. Further comments on this are at the bottom of the page. For deblurring we consider two scttings, using a Gaussian kernel with std of (0.8, 1.6) pixels and p 0.05 Gaussian noise. We then follow an analogous procedure; adding p = 0.05 Gaussian noise and applying s Uni(0.4,2) 2) blur to each training datum, and then constructing a single subspace.
10

---

Published in Transactions on Machine Learning Research (02/2024)
Denoising
Deblurring
30
20
PSNR (dB)
10
30
20
P
0.15
p-0.5
0.8
DIP
200
400
600
200
400
600
Sub-DIP NGD
NGD
optimisation time (s)
Denoising
Deblurring
p 0.15
p 0.5
K = 0.8
1.6
30
24
30
PSNR (dB)
27
21
27
18
24
X
max
15
21
21
12
DIP
Sub-DIP Adam
Sub-DIP NGD
Figure 9: Denoising (p = (0.15, 0.5}) and deblurring (x {0.8, 1.6))) on the "jet F16" and "house" images. On the top we show PSNR trajectories for each of the tasks, comparing DIP and Sub-DIP NGD, and on the bottom we show mean and std of max and conv. PSNR over the studied 5 images. Results for the other two noise levels are reported in Appendix A.
Ground truth
DIP max
DIP last
Sub-DIP NGD last
PANK
db
Denoising =p= 0.15)
Figure 10: Restoration of the noisy (with noise scaling parameter p = 0.15) "baboon" image.
As common practice when deploying DIP on restoration tasks, we do not include the TV (x= 0) 2 for neither denoising nor deblurring. Instead, the regularising property of the reconstruction stems exclusively from restricting the DIP optimisation to low-dtmensional subspace of its parameters.
The top row in Fig. g shows PSNR trajectories for denoising and deblurring on "jet F16" and "house" images. These images contain large regions of continuous colour intensity with clearly defined edges. Hence, we expect subspace methods to perform well. This is confirmed by the results: Sub-DIP NGD and DIP have a comparable max PSNR. The bottom row Fig. g shows the mean and std of max and conv. PSNR over studied natural images for low and high noise and blur. The conv. reconstructions are computed Algorithm ] applied to the training loss with 8=0.995 and p=100 steps. We notice that DIP mildly
11

---

Published in Transactions on Machine Learning Research (02/2024)
= 0.1
= 0.25
= 0.5
25.0
22
20
22.5
20
20.0
18
30
22
25
25
20
20
0
500
1000
500
1000
0
500
1000
optimisation steps
sub
500 d sub 1000
sub
3000
sub
8000
Figure 11: Investigation of the regularising effect of the dimensionality of the chosen subspace on Set5. We report PSNR trajectories for two specific images, the "baboon" and the "peppers" using Sub-DIP NGD. Our analysis encompasses three distinct noise levels and four dimensions of the subspace.
Ground truth
drub = 300
drub 1000
desb
3000
d
8000
Restoration of the noisy (with noise scaling parameter = 0.1) "house" image in Sct5 using different subspace dimensions.
Figure 12:
oulperforms Sub-DIP in lerms of time to max PSNR. However, DIP is also significantly more prone to overfit noise and exhibits a sharp decline image quality after reaching max PSNR. Thus, selecting a good stopping criterion is not only critical but also challenging.
Fig. 10 shows the denoising results for the "baboon" image with moderate noise level (p 0.15). The "baboon" image contains fine-grained details and sharp transitions throughout the image. Henoe, expect a somewhat comparatively worse performance of subspace methods on this task, as higher frequency information can be lost when using low-dimensional approaches. The results confirm this intuition: the max PSNR for the "baboon" with E-DIP is least dB higher than that for subspace methods. Moreover, while standard DIP experiences sharp decline after achicving max PSNR, subspace methods retain the performance.
To gain deeper insight into the interplay between the dimension of the subspace and the noise level corrupting the data, sweep over daub {500, 1000, 3000, 8000) and {0.1, 0.15, 0.25, 0.5]. While we maintain the ratio dior/ds at for subspaces up to 3k, for computational fcasibility we reduce it to 0.25 for the 81-dimensional case. As shown in Fig. 11 and in Fig. 23 and Fig. 22 in Appendix A.4, in high-noise scenarios (iee., p 0.5), using a larger subspace leads to overfitting. Conversely, in low-noise settings, lower- diver-dimesioonll subspace limits the network's capacity to fit higher frequencies and adapt, resulting in overly smooth roconstructions, cf. Fig. [2] We can thus conclude that noisier problems require more regulorisation, thus domer-dimensioonl subspace may be beneficial.
12

---

Published in Transactions on Machine Learning Research (02/2024)
6 Conclusion
In this work, we develop novel approach that constrains DIP optimisation to sparse principal low- dimensional subspace, extracted from pre-training trajectories. This greally miligates, if not completely eliminates, overfitting. Our approach may be understood from the perspective of the bias-variance trade-off. At initialisation, the vanilla DIP presents a useful bias towards learning low-frequency image components; but the over- parmmaaamtttiiiionnnn of the neural network leads to overfitting. Pre-training only partially removes DIP's low pass bias, allowing the F-DIP often fit. images quickly. This comes at the cost. of increased variance. Our approach seeks efficiently navigate the bias-variance trade-off. Constraining the optimisation to a low-dimensional subspace greatly reduces the variance. By extracting the subspace from principal directions of pre-training trajectories, and through the use of leverage scoring, we limit the bias introduced into the model. Furthermore, optimising in lower dimensional subspaces allows using fast and stable second order optimisation methods. In future work, alternative approaches for the identification of the subspace and their implicit regularising properties will be investigated.
In our experiments on several image restoration and tomographic tasks, subspace DIP methods deliver reconstructions on par with DIP's max performance, and result in better reconstruction quality than the overfit. DIP reconstructions.
Acknowledgements
R.B. acknowledges the support from the i4health PhD studentship (UK EPSRC EP EP/S021930/1), and from The Alan Turing Institute (UK EPSRC EP EP/N511 APP/550019/// 11).) J.A. acknowledges the support from Microsoft Research, through its PhD Scholarship Programme, and from the EPSRC. J.L. was funded by the German Research Foundation (DFG; GRK 2224/1) and by the Federal Ministry of Education and Rescarch via the DELETO project (BMBF, project number 05M20LBB). J.M.H.-L. acknowledges support from Turing AI Fellowship EP/V023756/1 and an EPSRC Prosperity Partnership GP/T005356/1L The work of B.J. is partially supported by UK EPSRC grant EP HP/VO26259/1, Hong Kong RGC General Research Fund (Project 14306423), and a start-up fund from The Chinese University of Hong Kong. Z.K. acknowledges the support from the UK EPSRC grant EPP CP/X010740/1. This work has been performed using resources provided by the Cambridge Tier-2 system operated by the University of Cambridge Research Computing Service (http://wwrr http://www.hpp... wwtp://wwp.cc..a..... funded by EPSRC Tier-2 capital grant EP/T022159/1
References
Jonas Adler, Holger Kohr, and Ozan Öktem. Operator discretization library (ODL). Software available from https: // github. com/ odwgroup/ odl 2017.
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural compulation, 10(2):251 276, 1998.
Shun-ichi Amari. Information geometry and its applications: Survey. In Frank Nielsen and Frédéric Barbaresco (eds.), Geometric Science of Information First International Conference, GST 2013, Paris, France, August 28-30, 2013. Proceedings, volume 8085 of Lecture Notes in Computer Science, pp. 3. Springer, 2013. doi: 10.1007
1. URL
Javier Antorán. Understanding uncertainty in bayesian neural networks. Mphil in Machine Learning and Machine Intelligence Thesis, University of Cambridge, 2019.
Javier Antorán, Riccardo Barbano, Johannes Louschner, Josó Miguel Hornández-Lobato, and Bangti Jin. A probabilistic deep image prior for computational lomography. Preprint, arXiv:2203.00477 2022.
Javier Antorán, David Janz, James Urquhart Allingham, Erik A. Daxberger, Riccardo Barbano, Eric T. Nalisnick, and José Miguel HemándezzLooatoo Adapting the lincarised Laplace model evidence for modern deep learning. CoRR, abbb/220..... 089000, 2022. doi: 10.48550 aaxii 2206008900. URL https
//doi.org/10.. 1855010.48555aaam aa1550 220...2266600 08900
13

---

Published in Transactions on Machine Learning Research (02/2024)
Javoe; Antorán and Antonio Miguel. Disentangling and learning robust representations with natural clustering. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 694-699, 2019.
Daniel Otero Baguer, Johannes Leuschner, and Maximilian Schmidt. Computed tomography reconstruction using deep image prior and learned reconstruction methods. Inverse Problems, 36(9):094004, 2020.
Riccardo Barbano, Johannes Leuschner, Javier Antorán, Bangti Jin, and José Miguel Hernández-Lobate Bayesian experimental design for computed tomography with the lincarised deep image prior. CoRR, abs obss2200.055144 2022a. doi: 200/55000/XXxv222...0060 URL 05714
/1w5///.....///22111.... 2207
Riccardo Barbano, Johannes Leuschner, Maximilian Schmidt, Alexander Denker, Andreas Hauptmann, Peter Maaß, and Bangti Jin. An educated warm start for deep image prior-based micro CT reconstruction. JEEE Transactions on Computational Imaging, 8:1210-1222 2022b. doi: 10.1109 TT/9//CT...22223/31133
Matthew Brand. Incremental singular value decomposition of uncertain data with missing values. In Computer Easion-ECCV 2002: 7th European Conference on Computer Vision Copenhagen, Denmark, May 28-31, 2002 Proceedings, Part 1 7, pp. 707-720. Springer, 2002.
Prithvijit Chakrabarty and Subhransu Maji. The spectral bias of the deep image prior. CoRR, abs/ /te//9922.00000 2019. URL httt:://axxivorrg/bbbb..
08905
Antonin Chambolle, Vicent Caselles, Daniel Cremers, Matteo Novaga, and Thomas Pock. An introduction to total variation for image analysis. In Theoretical foundations and numerical methods for sparse recovery. pp. 148-340.600 de Gruyler, 2010.
Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In International Conference on Machine Learning, pp. 4754-4776. PMLR, 2022.
Erik A. Daxberger, Eric T. Nalisnick, James Urquhart Allingham, Javier Antorán, and José Miguel Hernández- Lobato. Bayesian deep learning via subnetwork inference. In Marina Meila and Tong Zhang (ods.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 2510-2521 PMLR, 2021. URL http::/ attp://proceeddings. mlr http://www.m.....ee.eeeeeeeeeeee v139/ httpa/////119/d/dxd/ee dwxbergereaa..a.../// html htmml
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEFE conference on computer vision and pottern recognition, pp. 248-255. Icco, 2009.
Henri Der Sarkissian, Felix Lucka, Maureen van Eijnatten, Giulia Colacicco, Sophia Bethany Coban, and K. Joost Batenburg. Cone-Beam X-Ray CT Data Collection Designed for Machine Learning: Samples 1-8, 2019. URL https::///o/l http://oo..org////// 10. www.//2e0ee...... 2686726 Zenodo.
Lijun Ding, Zhen Qin, Liwei Jiang, Jinxin Zhou, and Zhihui Zhu. A validation approach to over-parameterized matrix and image recovery. CoRR, abs/ 10677 2022. doi: 10.48550/ar 2009. URL https:
//doi.org/10. www.o////......///////////// 48550/ arXiv. www.aa.www............. 22..
Petros Drineas, Malik Magdon-Ismail, Michael W. Mahoney, and David P. Woodruff. Fast approximation of matrix coherence and statistical leverage. J. Mach. Learn Res., 13:3475-3500 2012. doi: 40.5555/2503308
2503352. URL https:://at acm. ort//oo/////ooooo do1/10. 03555/250330... 2503352
Yonina Eldar. Generalized sure for exponential families: Applications to regularization. IEFF Transactions on Signal Processing, (2):4441 481, 2008.
F. Dan Foresce and Martin T. Hagan. Gauss-Newton approximation to Bayesian lcarning. In Proceedings of International Conference on Neural Networks (ICNN'97), Houston, TX, USA, June 9-12, 1997, pp. 1930- 1935. TEFE, 1997. doi: 10.1109/ICNN.1997.614177 URL https://doi. fecboi.oorg//011 1109 ICNN. 1997. 614194
14

---

Published in Transactions on Machine Learning Research (02/2024)
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Nathan Halko, Por-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomnoss: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev., 53(2):217 288, 2011. doi: 00.1137/090777 URL
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky, Neural networks for machine learning course slides, available at https://www www.stooootooooooooooo http://www..aaaeee//c/////a//a/a
ses___sdd__sse_ee __sss__e___________ee______eee pdf 2014.
Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising, In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 5067-500 5076. TEFF, 2021. doi: 0TAp///TTCCC.009222000000000.000 URL https://doo.... / N0.1109/ICV448922 0022100504.
Diederik P. Kingma and Jimmy Ba. Adam: method for stochastic optimization. Proceedings of the 3rd International Conference for Learning Representations, San Diego, 2015. Available at arxiv:1412.6980,, 2015. URL http::/arxiv.. orrg abs//412.
Frederik Kunstner, Philipp Hennig, and Lukas Balles. Limitations of the empirical fisher approximation for natural gradient descent. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' AlAchéé Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4158 4169, 2019. URL httpps://proceedings. neurips ceurrps.cccpappe//0000 cc/paper/2019/ Abstraacttttttttttt........
Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 105-114. IEEE Computer Society, 2017. doi: (41109/CVPR.0177.11 URL lhttps:///dot /ttt:::://oi.oor/10... 1109////..//// 2017 19
Johannes Leuschner, Maximilian Schmidt, Daniel Otero Baguer, and Peter Maass. Lodopab-ct, a benchmark dataset for low-dose computed lomography reconstruction. Scientific Data 8(1):1 12, 2021.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https://openreviiet
ryup8--WWi
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. JEFE Transactions on Pattern Analysis and Machine Intelligence, 45(:333111 3420, 2023. doi: 10.1109/TPA..1.22222 RMA.MM..022..1771100
Edo Liberty. Simple and deterministic matrix sketching. In Inderjit Dhillon, Ychuda Koren, Rayid Ghani, Ted E. Senator, Paul Bradley, Rajesh Parekh, Jingrui He, Robert L. Grossman, and Ramasamy Uthurusamy (eds.), The 19th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD 2013, Chicago, IL, USA, August 11-14, 2013, pp. 581 588. ACM, 2013. doi: 10.1145/2487555.447700 URL https: //doi org/10 1145/2487574 2487623
Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Math. Program., 45(1-3):503 528, 1989. doi: 10.1007/BF0158900 URL https://dbb..org100..00
BF01589116
Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S Kamilov. Image restoration using total variation regularizod deep image prior. In ICASSP 2019, 2019. doi: 10. T0.1109/CCASSP:2001
15

---

Published in Transactions on Machine Learning Research (02/2024)
Alexander Ly, Maarten Marsman, Josine Verhagen, Raoul Grasman, and Eric-Jan Wagenmakers. A tutorial on fisher information, 2017. URT.
James Martens. New insights and perspectives on the natural gradient method. J. Mach. Learn. Res., 21: 146:1 146:76, 2020. URL http:///wa...ooopppppp//////1110
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, JCML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 2408 2417. JMLR.org, 2015a. URL hatt:://proceedinngs html
httpeedddnnn......resr/sseeee/ee .ml.n.ss..........m............. mlr. ppress/v3//maree
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015b.
James Martens and Iyy Sutskever. Training deep and recurrent networks with hessian- free optimization. In Grégoire Montavon, Genevieve B. Orr, and Klaus-Rober Müller (eds.), Neural Nelworks: Tricks of the Trade Second Edition, volume 7700 of Lecture Notes in Computer Science, pp. 479 535. Springer, 2012. doi: 00.1007978--3-4222
8 _2. URL http::::///oi..o/1....oo.. wttp///......./....00//0///0---- 3-4---288-----88-8-88 __
Per-Gumnar Martinsson and Joel A. Tropp. Randomized numerical linear algebra: foundations and algorithms. Acta Numer, 29:403 572, 2020. ISSN 0962-4929. doi: 10.10177:40962249992000000 URL httpps://db
Christopher A. Motzler, Ali Mousavi, Reinhard Hockel, and Richard G. Baraniuk. Unsupervised learning with stein's unbiased risk estimator. arXiv preprint arxiv:1805. 10531,
105.1011,, 2018.
Taylor R. Moen, Baiyu Chen, David R. Holmes III, Xinhui Duan, Zhicong Yu, Lifeng Yu, Shuai Leng, Joel G. Fletcher, and Cynthia H. McCollough. Low-dose et image and projection datasct. Medical Physics, 48(2): 902 911, 2021. doi: https:/ /to..oooooooo/// 1wtpg//1.1002/ppp2/ 145994 URL httph:tt///// //wapm.aa..m/ onl www.ineebbrrry.wlle........
doi/abs/00. 14594
Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar Mosseri, Forrester Cole, and
Kevin Murphy. Xgan: Unsupervised image-to-image translation for many-to-many mappings, 2017. URL
05139
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259 268, 1992.
Silvia Scarpetta, Magnus Rattray, and David Saad. Matrix momentum for practical natural gradient learning. Journal of Physics A: Mathematical and General, 32(22):4047, 1999.
Nicol N Schraudolph. Fast curvature matrix-vecte vector products for second-order gradient descent. Neural computation, 14(7):1723 1738, 2002.
Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning From Theory to Algo-
rilhma. Cambridge Universily Press, 2014. ISBN 978-1-10-705713-5. URL h/tp: cambridge
machh seaachinn--earning
Zenglin Shi, Pascal Mettes, Subhransu Maji, and Cees G. M. Snoek. On measuring and controlling the spectral bias of the deep image prior. International Journal of Computer Vision, 130:885-908, 2022.
Ravid Shwartz-Zii Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and An- drew Gordon Wilson. Pro-train your loss: Easy bayesian transfer learning with informative priors. CoRR, abs/2225510070 2022. doi: 10.48550 40i50//ariii....../ 2wt.Xi...2551029790 10279
URL
2rg//1.4855000 48550/455533 atto///axii......vv 2205
16

---

Published in Transactions on Machine Learning Research (02/2024)
Chunwei Tian, Lunke Fei, Wenxian Zheng, Yong Xu, Wangmeng Zuo, and Chia-Weo Lin. Deep learning image denoising: An overview. Newral Nehworks, 131:251-22 12:251 2020. doi: 10.1016/j.neumet.222)) 2020.07..022200......
URL https://doi.org//n 1016/j01ejjj wwwnne.eeeeee..........e neunet www.ee.eee......... 2020.07. 0220.07.020 .25
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognátion (CVPR), pp. 9446-9454, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. Int. J. Comput. Vis., 128(7):
1867-1888, 2020. ISSN 1405- doi:
Wim van Aarle, Willem Jan Palenstijn, Jan De Beenhouwer, Thomas Altantzis, Sara Bals, K. Joost Batenburg, and Jan Sijbers. The ASTRA Toolbox: A platform for advanced algorithm development in electron 10.10166/..ltrammaiit
tomography. Uttramicroscopy. 157:35-47 2015. doi:
Ge Wang, Jong Chul Ye, and Bruno De Man. Deep learning for tomographic image reconstruction. Nature Machine Intelligence, 2(12):737-743) 2020.
Hengkang Wang, Taihui Li, Zhong Zhuang, Tiancong Chen, Hengyue Liang, and Ju Sun. Early stopping for abt/1122.b21...../// 2211..00710004
deep image prior. CoRR, abs/2112.06074 2021. URL
Wei Wen, Chumpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett. (eds.), Advances in Neurol Information Processing Systems 29: Annual Conference on Neurol Information Processing Systems 2016, December 5- 10, 2016, Barcelona, Spain, pp. 2074 2082, 2016. URL https //proceedings.nenee httpp/pcp...pppppp//////////// /ttt/////////1100000000000000000 wtw.:///e....0aa-arraart.... html
Mitchell Wortsman, Maxwell Horton, Carlos Guestrin, Ali Farhadi, and Mohammad Rastegari. Learning neural network subspaces. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, TCML, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11217 11227. PMLR, 2021. URL http://proceeddng.. mlr. mlr.press/
#139/wortsmanl html
Burhancddin Yaman, Seyod Amir Hossein Hosscini, and Mchmet Akcakaya. Zero-shot physics-guided deep learning for subject-specifie MRI reconstruction. In NewrIPS 2021 Workshop Deep Learning and Inverse Problems, 2021. URL htttps://openreviex nott//ormmmidddee22
Gushan Zeng, Yi Guo, Jiaying Zhan, Zi Wang, Zongying Lai, Xiaofeng Du, Xiaobo Qu, and Di Guo. A review on deep learning MRI reconstruction without fully sampled k-space. BMC Medical Imaging, 21(1): 195, 2021. doi: 10.1186/1122800-01--000
22000///000111111120000000-0-000 URL /11
17


Reviews:
[{"review_id": "1TS2drdcyt", "reviewer": "Reviewer_FJzL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe method is well-motivated, and innovative in integrating techniques from different domains that include subspace estimation, optimization and DIP. \n\nThe experiment design, from ablation study to comparison with existing methods, is robust and comprehensive, with good experimental details. \n\nResults show strong performance of the proposed method across multiple datasets and tasks.\n\n\n**Weaknesses**\n\nSome aspects of method design need more justification, e.g. the rationale behind idetifying the subspace from pretraining checkpoints.\n\nSome important experimental details are unclear, especially the stopping criterion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method to mitigate overfitting in DIP. It identifies a sparse and low-dimensional subspace for the network parameters and optimizes within this subspace. The sparse subspace is computed from pretraining trajectory, and second order methods are used for optimization thanks to the reduced dimensionality. Experiments show that the Sub-DIP method is sigfinicantly less prone to overfitting, compared to DIP and E-DIP.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The authors may want to justify the choice of using pretraining checkpoints to identify the subspace. Why is optimizing within this subspace a good choice? It seems to the reviewer that, a desirable subspace should capture \"good\" network parameters, for example, the parameters that achieve good reconstruction performance. However, pretraining checkpoints may not provide this.\nFor example, early checkpoints capture under-trained networks. On the other hand, later checkpoints may not be diverse enough, as the training may have converged. The reviewer is curious about the rationale behind this choice.\n\n2. What is the stopping criterion for DIP, E-DIP, sub-DIP for each dataset? In muCT-walnut experiment, the authors claimed that DIP and E-DIP cannot use the same criterion as sub-DIP. Why is this the case? More information should be provided since \"conv.\" PSNR is largely dependent on the choice of stopping criterion. How did authors select the criterion for each method? Additionally, how is lambda in (2) chosen?\n\n3. Can the authors comment on why using time, instead of iterations, as x-axis? Time is less reliable since it can be affected by many factors such as CPU/GPU load. How would the results look if plotted against iterations? \n\n4. The authors can comment on the requirement of a pretraining dataset for sub-DIP (and E-DIP), and contrast it with the original DIP which does not require pretraining. Furthermore, given access to data samples, other alternatives, such as Plug-and-Play and diffusion models, can be applied. The authors may want to address the pros and cons of these methods, to place their a work in a larger scope. \n\n5. The L_gamma in (6) is different from the one defined in (5) (the former does not include TV). The authors may want to clarify this before deriving $\\nabla_c L_\\gamma (c)$ to avoid confusion.\n\n6. The reviewer is curious how pretraining can be amortised across different reconstructions, as stated in Step 1.\n\n7. In Figure 1, is \"last\" the same as \"conv.\"? \n\n8. Figure 2: Is the DIP conv. result missing?\n\n9. \"Fig. 2 (right) shows that Sub-DIP LBFGS and NGD converge in less than 50 seconds. These methods are optimal along the known Paretofrontier, with the former reaching ∼2 dB higher reconstruction fidelity and the latter converging faster (in ∼20 seconds).\"\nIt seems LBFGS converges faster and the gap is not as large as 2dB.\n\n10. Consider moving Algorithm 1 to the main body.\n\n\nOverall this paper presents an interesting novel methodology with promising results. The authors can further improve their work by addressing these points.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tu7j2vlSym", "reviewer": "Reviewer_aq9Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The approach is interesting and novel. It is definitely conceivable that reducing the dimensionality of the network parameters in some way will provide additional regularization.\n2. The ablation studies presented are fairly comprehensive. \n\nWeaknesses:\n1. I think the paper does not provide enough intuition as to why the particular scheme of reducing the dimensionality of the network parameter vector works. Thus, the choices feel a bit arbitrary. \n2. I think the evaluation methodology used is fairly limited and has several drawbacks. For example, in the case of medical imaging, improved performance of relevant downstream tasks (eg. detection tasks relevant for diagnosis) are far more important than PSNR. For applications where perceptual quality is the most important (eg. upsampling in computational photography), PSNR is still not always a good metric since it favors images with oversmoothed features. Thus, the authors should consider using a a better variety of metrics that are more suited to various applications considered in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Deep image prior (DIP) for image reconstruction/restoration is often doomed to produce poor results due to its tendency to eventually overfit the measurements when run till convergence. This work proposes a technique to regularize DIP by representing the parameters of the network in a lower dimensional space. The work claims to achieve improved empirical performance in terms of traditional measures of image quality, such as PSNR, as compared to conventional DIP on. variety of datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think it is important that the authors add further intuition for why the specific scheme of dimensionality reduction is good. Additionally, I think it would be important to present this paper as a preliminary investigation of a new regularization scheme for DIP, and tone down statements that indicate that this paper brings DIP closer to real-life applications, since based on the results presented, it is hard to conclude this. It is also essential to mention the limitations of the evaluation methods used in the context of real-life applications.\n\nUsing a wider variety of evaluation metrics as well as task-based evaluation metrics will definitely strengthen the paper. Also,\nit seems that the basis vectors used for the dimensionality reduction were extracted from a pretraining scheme based on an ellipse dataset. What sort of biases does this dataset introduce? Does this favor better reconstruction of piecewise smooth images rather than images with fine details? A discussion of this will also strengthen the paper.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qTpr1kIHnD", "reviewer": "Reviewer_HM4E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\n- The method is intuitive and well-motivated. To the reviewer’s knowledge, while pre-training the DIP has been proposed in the past, re-parameterizing optimization over low-dimensional subspaces and allowing for more efficient second-order optimization is new.\n- There are clear speedups in runtime and overfitting is reduced.\n- The experiments are quite extensive, and the authors do a good job of ablating various aspects of their method.\n- The results would be of interest to the DL + inverse problems community.\n\n## Weaknesses:\n- While early stopping is mitigated, new hyper parameters are introduced that have a big influence on performance (e.g., size of the subspace and its sparsity pattern). Authors do, however, prescribe recommendations for how to set these parameters, but tuning is still required.\n- The approach is only demonstrated for linear inverse problems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work considers solving imaging inverse problems in low-data regimes. The Deep Image Prior (DIP) has shown to be a powerful implicit prior without requiring paired training data, but suffers from long runtimes and the need for early stopping to avoid overfitting. This work develops a method to mitigate both issues by performing inversion over an extracted low-dimensional principal subspace of parameters (found by pertaining on synthetic data). The low-dimensionality and sparsity of the subspace aid in reducing overfitting. Once this subspace is constructed, one can then leverage second-order optimization methods such as L-BFGS or natural gradient descent to improve the DIP’s runtime. The method is demonstrated on several datasets and inverse problems, including denoting, deblurring, and tomographic reconstruction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Answers to the following questions and comments would, in my opinion, strengthen the work.\n- Did the authors analyze the relationship between the amount of noise/undersampling was present in the measurements versus the size and sparsity of the chosen subspace? If not, do the authors have a sense of what would be ideal choices for these parameters as a function of the amount of noise is present? I would suspect that for noisier problems, more regularization may be necessary, hence choosing a lower-dimensional subspace with a sparser basis may be beneficial.\n- What would be some of the challenges in bringing this approach to nonlinear settings? In principle, I believe this could also work for problems such as phase retrieval or one-bit compressed sensing, but perhaps a discussion of these challenges or a simple example demonstrating the approach in a nonlinear setting could be interesting to include.\n- How sensitive is the approach to distribution shifts in the forward model and images to be recovered, e.g., if one pre-trained with fewer/more angles than one sees at inversion time or one used synthetic images to recover natural images?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The ideas proposed in this paper are novel and intuitive. While there is no theoretical analysis that supports the methodology, extensive empirical results support the authors claims and provide an advancement with respect to state of the art in image restoration with Deep Image Prior. All reviewers support the acceptance of this paper, and I'm recommending this accordingly. 

In preparing the camera ready version of this paper, I encourage the authors to address the reviewers comments that haven't been addressed already, if any, as well as revise for typos and inaccurate statements (e.g., the expression $d_\text{pre} = \mathcal O (10^3)$ does not make sense from an O-notation perspective, since $10^3$ is a constant; I presume the authors simply mean "in the order of", or $d_\text{pre} \approx 10^3$ ).
Decision: Accept (poster)