Image Reconstruction Via Autoencoding Sequential Deep Image Prior
Authors: Ismail Alkhouri
Shijun Liang
Evan Bell
Qing Qu
Rongrong Wang
Saiprasad Ravishankar
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Reconstruction
Deep Image Prior
Generative Models
Primary area: generative_models

Abstract: Recently, Deep Image Prior (DIP) has emerged as an effective unsupervised one-shot learner, delivering competitive results across various image recovery problems. This method only requires the noisy measurements and a forward operator, relying solely on deep networks initialized with random noise to learn and restore the structure of the data. However, DIP is notorious for its vulnerability to overfitting due to the overparameterization of the network. Building upon insights into the impact of the DIP input and drawing inspiration from the gradual denoising process in cutting-edge diffusion models, we introduce Autoencoding Sequential DIP (aSeqDIP) for image reconstruction. This method progressively denoises and reconstructs the image through a sequential optimization of network weights. This is achieved using an input-adaptive DIP objective, combined with an autoencoding regularization term. Compared to diffusion models, our method does not require training data and outperforms other DIP-based methods in mitigating noise overfitting while maintaining a similar number of parameter updates as Vanilla DIP. Through extensive experiments, we validate the effectiveness of our method in various image reconstruction tasks, such as MRI and CT reconstruction, as well as in image restoration tasks like image denoising, inpainting, and non-linear deblurring.

Image Reconstruction Via Autoencoding Sequential Deep Image Prior
Ismail R. Alkhouri 1,2, Shijun Liang*3,
Evan Bell", Qing Qu?, Rongrong Wang Saiprasad Ravishankar
Department of Computational Mathematics, Science, & Engineering, Michigan State University
2 Department of Electrical Engineering & Computer Science, University of Michigan Ann Arbor 'Department of Biomedical Engineering, Michigan State University
'Department of Mathematics, Michigan State University falkhour3, 1iangs16, belleval, wangront ravisha3) Smsaa edu (ismailal, qingqu) @umi ch. edu
Abstract
Recently, Deep Image Prior (DIP) has emerged an effective unsupervised one- shot learner, delivering competitive results across various image recovery problems. This method only requires the noisy measurements and forward operator, relying solely on deep networks initialized with random noise to learn and restore the structure of the data. However, DIP is notorious for its vulnerability to overfit- the network. Building upon insights into
ting due to the overparamettrrizatoo the impact of the DIP input and drawing inspiration from the gradual denoising process in cutting-edge diffusion models, we introduce Autoencoding Sequential DIP (ISeqDIP) for image reconstruction. This method progressively denoises and reconstructs the image through a sequential optimization of network weights. This is achieved using an anput-adaptive DIP objective, combined with autoen- coding regularization term. Compared to diffusion models, our method does not require training data and outperforms other DIP-based methods in mitigating noise overfitting while maintaining a similar number of parameter updates as Vanilla DIP. Through extensive experiments. in various image reconstruction tasks, such as MRI and CT reconstruction, as well in image restoration tasks like image denoising, inpainting, and non-linear deblurring. Our code is available at the GitHub repository aSeqDIP.
validate the effectiveness of method
Introduction
Inverse imaging problems arise across various real-world applications [1]. These include tasks such image denoising [2], image deblurring [3], restoring missing portions of images [4] (in-painting), super-resolution [5], magnetic resonance imaging (MRJ) [6], and reconstructing X-ray computed tomography (CT) scans [7]. The common challenge in these scenarios is the reconstruction of images from limited and/or corrupted measurements.
Recently, plethora of Deep Neural Network (DNN) techniques have emerged to solve inverse imaging problems, including supervised models that are trained to process measurements and generate estimates approximating the true image [8, 9, 10, 1, 12). Generative model, such Diffusion models (DMs) [13], have shown state-of-the-an (SOTA) performance various imaging inverse problems such the works in [14, 15, 16]. However, training diffusion models is not only computationally expensive but also demands a significant amount of clean (or fully- sally-s data points [17], limiting
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

their application to data-limited or data data-corrune corrupted tasks such as in medical imaging (MRI and CT). To address this challenge, previous 3MM-ased MRI and CT methods typically fine-tune pre-trained DMs (trained natural images) using limited fully-sampled data points, which are not as readily available as natural images [14, 15). As such, there is pressing need to develop methods that can alleviate the reliance on large, clean datasets and/or pre-trained models.
One well-known method is Deep Image Prior (DIP) [18], which does not require pre-trained models and relies on the parameters of an architecture (e.g.. U-Net [19]). DIP finds a solution without needing any data other than the limited/corrupted task task- specific measurements and the forward operator. DIP has shown competitive reconstructionrrettaatiin results in inverse imaging problems such as MRI [20] and image denoising [18]. However, DIP (and most of its variants) suffer from the problem of overfitting.
In this paper, introduce Autoencoding Sequential DIP (aSeqDIP) to mitigate noise overfitting artifacts. Our approach comprises a sequential update of network weights. A significant advantage of our approach, in contrast to DM-based methods, is its ability to operate without the need for training data or pre-trained models, thus fitting into the framework of dataless training [21]. Notably, the total number of parameter updates in aSeqDIP remains equivalent to that of Vanilla DIP [18]. Furthermore, unlike a recent work [20], DIP eliminates the need for gradient-base input updates.
Contributions:
Building upon an insight about the impact of the DIP network input, we introduce Autoencoding Sequential DIP (IP) which incorporates a U-Net architecture whose weights are updated sequentially. These updates are based objective functions that consist of (i) input-adaptive data consistency term and (ii) an autoencoding regularization term used for noise overfitting mitigation.
Our extensive experimental evaluations, in terms of standard image reconstruction metries and required run-time, highlight the superior (or competitive) performance of aSeqDIP compared to DIP-based and leading DM-based methods for the tasks of MRI and CT reconstruction, denoising, in-painting, and non-linear deblurring.
Organization: In Section 2, we provide an overview of the considered tasks, and review recent advancements both DIP-based and DM-based methods. Section introduces our approach, aSeqDIP. Our experimental results are presented in Section 4, followed by discussions on conclusions and future directions in Section 5.
2
Preliminaries & Related Work
2.1 The Image Reconstruction Problem
Image reconstruction tasks can be defined recovering image x" E R" from measurements y E R", where m I n, governed by the forward operator A. For multi-coil MRI, the forward operator is A MFS, where M denotes coil-wise undersampling, F is the coil-by-coil Fourier transform, and represents the sensitivity encoding with multiple coils?. For CT, a simplified forward operator will be used to study the sparse-views views setting: A CR, where C is undersampling operator that selects specific projection views or angles, and R. denotes the radon transform [22] (corresponding to parallel-beam geometry in CT). For both MRI and CT with limited acquired data (done to accelerate scan or reduce X-ray dose, etc.), the condition m < applies.
For the denoising task, the forward operator is the identity matrix, whereas for in painting, it is equivalent to applying a binary mask element-wise that masks corrupted or missing pixels. For deblurring, the non-linear forward operator is the neural network approximated kernel in [3]. Next, we review prior arts within the context of Dip-based and DM-based methods.
2.2 Related Work
Deep Image Prior (DIP) was first introduced by [18). The authors demon- strated that the architecture of generator network alone is capable of capturing significant amount
DIP-based Methods:
IIn practice, the entries of x' and y in MRI are complex numbers. However, to generalize the problem's detinition, use real numbers.

---

of low-level image statistics even before any learning takes place. Specifically, the DIP image reconstruction is obtained through the minimization of the following objective:
i argmin [ - y X fz(z),
(I)
0
where & is the reconstructed image, and 0 corresponds to the parameters of network f: IR" R", which is typically implemented using U-Net architecture [19]. The input to the network, z E R", is randomly chosen and remains fixed throughout the optimization process. While standard DIP was shown to perform well in many tasks, selecting the number of iterations to optimize objective (1) poses a challenge as the network would eventually fit the noise present in y or could to undesired images based on the null space of A.
To mitigate the problem of noise overfitting, previous studies considered different approaches such as regularization, early stopping (ES), and network pruning [23]. For regularization-be -aaedd methods, the work in [24] enhanced the standard DIP by introducing total variation (TV) regularization term for denoising and deblurring tasks, whereas the study in [25] proposed combining DIP with stochastic gradient Langevin dynamies (SGLD) [26]. The authors in [27] use running variance as the criterion for ES, whereas the authors of [28] propose combining self- validation and training to apply ES.
The input to the standard DIP (or Vanilla DIP) network is random noise vector that, in most works. remains fixed during the optimization. Nevertheless, other works, such as those in [29] and [30], have explored cases where the input contains some structure of the ground truth. The approach employed in cesrrence-guidede DIP (Ref-Guided DIP) [29] follows the same objective as standard DIP in (1). However, instead of using fixed random noise vector as input, it utilizes reference image closely resembling the one undergoing reconstruction. This method was applied to the task of MRI. This methodology proves particularly effective when datasets comprising structurally similar data points are available. The reference required here makes this method data-dependent approach.
Inspired by the departure from using a random fixed input, the authors in [20] recently introduced Self-Guided DIP. Unlike Ref-Guided DIP [29], a prior image that closely resembles the unknown (to be estimated) image is not needed, and the optimization occurs simultaneously with respect both the input and the parameters of the network. Specifically, Self-Guided DIP employs the following objective:
0,2 2 = argmin [AE,,,o(zz +)] - + a E, + 7) -z1
(2)
0,2
where 17 is random noise, and a is a regularization parameter. The first (resp. second) term is used for data consistency (resp. denoising regularization) and final reconstruction is obtained as * = [f + v))]. aSeqDIP is different from Self-Guided DIP as our method does not require gradient-based updates for the input, making it computationally less expensive. Self-Guided DIP has demonstrated superior performance compared to Vanilla DIP, TV-DIP, and SGLD-DIP, thus serving as primary baseline for comparison.
DM-based Methods: recent years, there has been an abundance of DM-based methods proposed
to address inverse imaging problems [14, 15, 16, 31, 32, 33]. A well-known method for natural images is Diffusion Posterior sampling (DPS) [16]. DPS incorporates a gradient step into the reverse sampling process of pre-trained DMs, ensuring data consistency and enabling sampling from the conditional distribution. In the context of image reconstruction and restoration tasks, numerous iiffusion-basee approaches have emerged, as evidenced by works such as [34, 35, 36]. Notably, the authors in [14] and [15] introduced a SOTA DM-based approach for addressing the MRI and CT reconstruction inverse problems, respectively. They propose incorporating the predictor-corrector sampling algorithm [37] for data consistency, akin to DPS, thereby facilitating the sampling from a conditional distribution.
One clear distinction between aSeqDIP and DM-based methods that our approach does not necessitate pre-trained models. For our experiments in MRI, CT, and denoising (as well as in-painting and deblurring) lasks, we will utilize Score-MRI [14], Manifold Constrained Gradient (MCG) [15], and DPS [16], respectively, as DM-based baselines.
3

---

3 Autoencoding Sequential DIP
In this section, we begin by investigating the impact of the input on DIP. Then, we introduce our method, aSeqDIP. We note that while we consider linear and non-linear inverse problems, in our formulations, we use a linear forward model to simplify notation.
3.1 Motivation of aSeqDIP: The Impact of the Network Input in Vanilla DIP
Here, we aim address the question: How does employing noisy version of the ground truth image, which retains some structure of the ground truth, as the fixed input to the Vanilla DIP objective in (1), affect performance? To investigate,
conduct the following experiment.
Consider the MRI task defined as y Axx.
input to standard DIP objective (1) be
denoted as = x + 8, where ô ~ (O, o²I). Here, controls the magnitude of the perturbations added the ground truth image, indicating that larger a results greater deviation between and x". We optimize (1) for various values of recording the best possible PSNR compared to the ground truth, i... prior to the start the noise overfitting decay.
Figure 1 displays the average results for 8 images. Notably, for all images, a closer similarity of the DIP network input x", as indicated by a, corresponds higher reconstruction quality, measured by PSNR. Larger variance in the standard Gaussian distribution corresponds to larger additive perturbations even for the case of x (the red curve). We conjecture that this still leads to larger distances from the ground truth and hence worse performance.
Based this discussion, notable insight emerges:
Best Possi ble PSNR (dB)
DIP with input
The proximity of the DIP network input the ground truth correlates with the quality of the re- construction. This promotes the question: Can we develop an input-adaptive DIP method that mitigates noise overfitting?
DIP input
31
We proceed address this question by proposing our method, which we refer to as Autoencoding Se- quential DIP (ISeqDIP). In Appendix A, provide a case study and theory on the impact of the DIP network input through the lens of the Neural Tan- gent Kernel in residual networks. The onset of severe noise overfitting therein is delayed for better inputs (Appendix A.2).
Input Noise Level
Figure 1: Average best possible PSNR values (in dB) obtained from standard DIP in (1) for 8 MRT (with 4x acceleration factor) scans (y-axis), where network input is either a perturbed version of the ground truth or pure noise. The noise is a zero-mean additive Gaussian noise with strength determined by o (x-axis).
3.2 The Proposed DIP Algorithm
Consider that we have a U-Net architecture defined by : RR whose weights are given by ÓR- where 1: E [K], and [K] := (1,.... K}. Each set of parameters in fe, takes an input 2₁ and outputs for (z+). Based on the insight from the previous subsection, we initially set to (resp. A Ayy) for denoising, in- un-painting, and deblurring (resp. MRI and CT). The initialization of Φi follows the same initialization as any other DIP-based method. The parameters in før. and the input, are then updated sequentially through
Φk argmin |A (zL-1) y112 All (Zk-1) 21-1
(3)
Za S (Z
where À E R4 is regularization parameter, and the initialization of de the optimized dx-1 in (3). The final reconstruction is given as:
x = ZK = (ZK-
(5)
The proposed procedure outlined (3) and (4) consists two key components. First, the optimization of each set of weights in for using objective that consists of the data consistency term and the

---

Algorithm/ Autoencoding Sequential Deep Image Prior (aSeqDIP).
Input: Measurements y. forward operator A, number of input updates K, number of gradient updates N per input update, regularization parameter A, and learning rate 3.
Output: Reconstructed image
Initialization: 20 = A " 60 ~ I(O,I).
For each & [K]
2: Initialize 0k c Pi-1 for it E (2,.... : }), and 0² + 2000) for
3: For each i [ (Network parameters update)
4:
PR VV [Afo - yllă Allfor (Zn-1) zk-1 111
16=00 17"
5: Obtain Zy: = S sg(N (2x-1). (Network input update)
6: Reconstructed image: & zK (2K-1)
()
f) Initialize 4x 4x-1
Initialize randomly Optimize for N steps
Initialize d + Optimize for N steps
y
2, (26)
Ko fo. (z,)
Zx-1 lov.. (2x-22
Optimize for steps
(4, sox (Rx-11)
Figure 2: Illustrative block diagram of the proposed aSeqDIP procedure. Each trapezoid corresponds to the
updates of for that takes 2 as input and is initialized with the optimized parameters Φ for & E {2, K} randomly for ls 1. The optimization for each set of weights takes place based on (3) and is run for N steps. The final reconstruction is fox (ZK-1).
second autoencoding term that aims to alleviate noise overfitting. Second, the update of the input, Zs. after optimizing each of weights fox, so that our method inpul-adaptive.
Algorithm presents the procedure of our proposed approach. As imputs, the algorithm takes y. A. K, N, A, and the learning rate 3. Apart from the measurements and the forward operator, the remaining parameters are considered hyper-pant parameeraaeeerss typical in most DIP-based methods. The parameters in fe,, set Φk-1 (step 2) and subsequently optimized for N iterations using gradient- madient-based optimizer, such as gradient descent (as depicted in Algorithm 1) or Adam [38]. A block diagram of proposed aSeqDIP method is presented in Figure 2.
In the following remarks, we provide insights into our proposed aSeqDIP method.
Remark 3.1 (Differences from Vanilla DIP [18]). Assume that the iterates of (3) and (4) converge, i.e. as k + 0o, + z" and d2 + o- Then, according to (4), for continuous mapping f. we have z fo- (z*). Substituting this into (3) in the limit, get o² (argming |Afa(z')
y112 fo(2²) =z"), )²), which corresponds the minimizer of
min A
y
z = fz(
(6)
The limit points of aIPqDIP correspond to the solution of a constrained version of the Vanilla DIP objective in (I). The constraint enforces additional prior that could alleviate overfitting. While its not straightforward to use a gradient-based algorithm for (6) given the hard constraint, the aSeqDIP scheme's limit points nevertheless minimize (6). Furthermore, aSeqDIP automatically estimates the network input by sequential feed forward process without needing expensive updates. The main point is show that aSeqDIP is solving the optimization problem in (6), which different than Vanilla DIP in (1).
Remark 3.2 (Differences from Self-Guided DIP [20]). While both aSeqDIP and Self-Guided DIP [20] update the input and network parameters simultaneously, there exist fundamental differences. Firstly, Self-Guided DIP solves the optimization problem in (2) which does not strictly enforce the auto-eneoder constraint z (z) as in (6). Secondly, aSeqDIP only requires a network forward pass to update resulting in significantly fewer computations as will further be demonstrated in our experimental results. Thirdly, the second term the aSP objective does not require computing the expectation, as it is an auto-encoder rather than denoiser that results in higher resistance to noise overfitting. Lastly, our method does not require initializing randomly and generating random vectors (n7 in (2)). The selection of 7 introduces additional
that we avoid, focusing solely
5

---

DM-based Approaches
Pre-trained DM Consislency
Pre-trained DM
Pre-trained DM +
Input
Data Consistency
Data Consistency
Data Dependent Methods
Vanilla DIP
SelF-Guided DIP
Referense-Guided
Random Input
Optimize Params.
Random Input
Optimize Params. and Input
Reference
Optimize
Image
Antoencoding Sequential DMP (Ours)
Mcasurements
Optimize Params,
Update Input
Optimize Params.
Update
Update
Optimize Params.
Input
Input
Data- Independett Methods
Figure 3: An overview of differences between aSeqDIP and prior arts in terms of data dependency. network architecture(s), and procedural requisites. 'Data-Dependency here indicates whether method depend on prior reference image or pre-trained models.
on selecting N K (total number of iterations) and À (regularization strength), which are necessary in most DIP-based methods.
Remark 3.3 (Computational Requirements). The computational requirements of aSeq are determined by two factors: (i) the NK gradient-based parameter updates, and (ii) the number of function evaluations necessary for updating z, which is K. In our experiments, we have found that setting 2 and K 2000 is generally sufficient. This configuration makes aIe nearly as efficient as Vanilla DIP.
Remark 3.4 (Relationship to DMs). aSeqDIP bears resemblance to the reverse process DMs due to their shared gradual denoising steps. However, despite these similarities, several distinctions emerge. Firstly, unlike the DM network, aSeqDIP does not require encoding a scalar representing time f. Secondly, and perhaps most significantly, aSeqDIP operates without requiring any training data or pre-trained networks. Thirdly, DSeqDIP operates in a truly sequential manner in terms of time, whereas in DMs, whether it's training (e.g., denoising score matching [39]) sampling, the prevalent technique involves sampling from time ~ (0, 1] (uniform distribution), which allows for non-sequential time points.
Figure illustrates how different approaches compare to aSeqDIP.
3.2.1 Mitigating Noise Overfitting in aIeqDIP
In DIP- approaches, noise overfitting occurs as the network attempts to fit its output to the noisy or subsampled measurements, y, as i increases during training. However, the specific value of k at which this PSNR decay begins is uncertain and varies across tasks and even among images within same task and distribution. aSeqDIP, when the output network fe, improves compared to that of Jo. the autoencoder term enforces similarity between the input and output of the network, thus delaying the onset of noise overfitting decay. This occurs because we are not only enforcing the network output to be mecaureement-consiseess become similar. Consequently, as increases, noise fitting delayed, and utilizing the autoencoder provides regularization against noise overfitting, In Section 4, will demonstrate how the proposed autoencoding term effectively regulates noise overfitting.
but also enforcing that the output and input
One might expect that incorporating the autoencoder could negatively impact reconstruction quality. However, empirical observations reveal that not only noise overfitting delayed with the autoencoder term, but also image reconstruction quality is enhanced. To further support this statement, in Appendix B, we investigate whether a trained autoencoder on clean images can as a reconstructor at testing time by optimizing the input.

---

Task
Setting
Data-independent baselines
Data-dependent baselines
Vanilla DIP [18], ES-DIP [27],
Ref-Guided DIP [29]
MRI
Ax (4x, 8x}
TV-DIP (24), Self-Guided DIP [20)
Score-MRI [14]
Vanilla DIP [18]. Self-Guided DIP [20).
Ref-Guided DIP [29)
CT
views E (18,30)
Filter Back Projection (FBP) [42]
MCG [15]
Vanilla DIP [18]. ES-DIP [27]. Self-Guided DIP [20].
Denoising
Ta C (15,30)
DPS [16]
TV-DIP [24]. Rethinking--DD DIP [43]. SGLD-DIP [25]
Vanilla DIP [18], ES-DIP [27], Self-Guided DIP [20].
In-painting
HIAR E (0.1,0.25)
DPS [16]
SGLD-DIP [25], TV-DIP [24]
Self-Guided DIP (20)
Dehlurring
BKSE [3]
DPS [16]
SGLD-DIP [25]
Table I: Tasks, settings, and baselines considered in our experiments. For MRI, we consider two Acceleration (Ax) factors, 4x and 8x, that determine the subsampling of the measurements. For 2D CT (parallel beam geometry), we use two sparse view settings: 18 and 30 views. For denoising. we perturb the ground truth images using two noise levels determined by aa- In in-painting. we use two hole-to-image area ratios (HTAR), 0.1 and 0.25. For non-linear deblurring, we use the Blurring Kernel Space Exploring (BKSF) setting [3), described in Equations (56) to (59) of [16]. Each baseline that utilizes pre-trained models reference image is considered data-dependent. Further details are provided in Appendix C.6.
4 Experimental Results
4.1 Settings, Datasets, and Baselines
In our experiments, we consider five tasks: MRI reconstruction from undersampled measurements, sparse-view CT image reconstruction, denoising, non-linear deblurring and in-painting. For MRI, we use the fastMRI dataset?. The forward model is y Ax'. The multi-coil data is obtained using 15 coils and is cropped to resolution of 320 x 320 pixels. To simulate undersampling of the MRI k-space, we use a Cartesian mask with 4x and 8x accelerations. Sensitivity maps for the coils are obtained using the BART toolbox [40]. For CT, we use the AAPM dataset'. For parallel beam CT, the input image with 512 x 512 pixels transformed into its sinogram representation using a Radon transform (the operator A). The forward model assuming a monoenergetic source and no scatter, noise is = Tue- [Ax"),, with Io denoting the number of incident photons per ray (assumed to be for simplicity) and indexing the ith measurement or detector pixel. We use the post-log measurements for reconstruction. We use a full set of 180 projection angles and simulate two different sparse view acquisition scenarios (with equispaced angles). Specifically, we created cases with 18 and 30 angles/views. The image resolution is kept at fixed size.
For the tasks of denoising. in-painting, and non-linear deblurring, we use the CBSD68 dataset'. For each task, we use 20 mcasurements/ we use the Peak Signal Noise Ratio (PSNR), and the Structural SIMilarity (SSIM) index [41]. For experimental settings and baselines, see Table 1 and its caption. Note that we consider data- dependent and inta-independent baselines as shown in the third and fourth columns of Table 1. All the experiments are conducted on single RTX5000 GPU machine. Further implementation details are provided in Appendix C.6.
images. To evaluate the reconstruction quality,
For the proposed DIP method in Algorithm 1, we use the Adam optimizer with learning rate of 3 0.0001. Furthermore, the regularization parameter is À following the ablation study in Appendix C.4. We select N = 2 and K = 2000 following the ablation study in Appendix C.S.
4.2 Impact of the Autoencoding term on Noise Overfitting
In this subsection, we showcase the impact of the proposed autoencoding regularization in DSeqDIP on noise null space (nuisance) overfitting.
We conducted experiments using 20 MRT scans and 20 CT scans, considering two cases of aSeqDIP as outlined in Algorithm 1. The first case sets X 1, consistent with the remainder of the paper,
artppo http::///gee //gitbbb .er....ooo cro.ottroooot//teee faattari haus/tzee/nain
hstpo: //wws, aapa. /ttgg///haaleggg///eeeeeoo
hrtpp:// /ttptt...bbb.../// www.ccowwwicceeeeeeeeeooo dataset

---

35
PSNR (dB)
30
aSeaDIP
25
regulseiation
20
SwikGuided DIP
Varilla DIP
15
1000
2000
3000
4000
1000
2000
3000
4000
5000
Reeation
Iteration
Figure 4: Average PSNR results w.r.t. iteration i of 20 MRI (with 4x) scans (leff) and 20 CT (with 18 views) scans (right) to show the impact of the proposed autoencoding regularization term on noise overfitting in aSeqDIP. Furthermore, average results of Vanilla DIP and Self- Guided DIP are also reported for comparison. For aSeqDIP, iteration E [NK], where. N Vertical lines approximately indicate the start of the PSNR decay for every case. In Appendix C.I, we include the PSNR curves aSeqDIP and other DIP-hased methods for the task of denoising.
while the second case sets À O, effectively disabling the autoencoding regularization term in (3). Additionally, for comparison, we report results for Vanilla DIP and Self-Guided DIP. The average PSNR results for these cases are depicted in Figure 4.
As observed, when the autoencoder term is disabled in aSeqDIP (blue dashed lines), noise overfitting in MRI, akin to Gelf-Guided DIP, begins after nearly 1600 iterations. For CT, note that aSe without regularization starts noise overfitting at around iteration 3800, whereas Self-Guided DIP experiences PSNR decay earlier, after approximately 1250 iterations. Importantly, when the autoen- coding term is utilized (blue solid lines), not only does the decay in noise overfitting not commence until after iteration 4000, but the reconstruction quality (measured by PSNR) also improves.
As expected, PSNR decay in Vanilla DIP begins early, at around iteration 500 and 750 for MRI and CT, respectively. In Appendix C.4, we provide ablation study to better show the impact of the value DIP
4.3
Main Results
Here, we present our primary results regarding the reconstruction quality, measured by PSNR and SSIM, as well the associated run-time. Table presents the results for the considered tasks in this paper. Column 3 indicates whether the baselines depend on prior data or pre-trained models. The last three columns provide the PSNR, SSIM, and run-time results where the arrows indicate favorable results. For PSNR and SSIM, the settings correspond to the second column of Table 1. The black (resp. blue) text corresponds to the first (resp. second) setting. Values after the + sign indicate standard deviation. Subsequently, we offer observations on the main results.
Compared data-independent methods, i.e., the baselines that do not depend on reference image or pre-trained models, aScqDIP demonstrates improved PSNR and SSIM scores. For example, DSPqDIP, apart from Self-Guided DIP, shows nearly 1dB improvement for MRI 8X acceleration compared to conventional methods. For the task of 30-views CT. DSPqDIP reports SSIM score of 0.92 which is 5% more than the second best, which is Guifed DIP with SSIM of 0.872. Although improvements against Self-Guided DIP are generally marginal in terms of reconstruction quality, our method proves be 2X faster for MRI and CT reconstruction and requires minute less than Self-Guided DIP for denoising and in in-painting. This speed-up is attributed to updating the input using one forward pass of the trained network at each iteration k instead of computing gradients with respect to the input for the update. Compared to Vanilla DIP, our method, on average, only requires an additional 30 to 60 seconds. When compared to ES-DIP [27], our method requires longer time, but on average achieves better reconstruction results across three tasks and different settings.
In comparison to data-dependent methods such as Score-MRI and MCG, our approach not only yields the best PSNR and SSIM but also requires reduced run-time, all without requiring any training data or pre-trained models. For instance, on average, aSeqDIP achieves nearly 2dB improvement in 30-views CT compared to MCG while being 2X faster. In comparison to DPS, on average, our method report higher SSIM. Our method requires slightly less run-time on average but enhances the PSNR by approximately 0.6dB for both denoising and in-painting. Notably, our method is an optimization- approach, whereas DM-based methods only require function evaluations. However, the generally larger run-time reported for DM-based methods is due to the necessity of
8

---

Data Independency
PSNR (dB) (f)
SSIM E 00. ())
Run-time (L)
Task
Method
(Setting Setting 2)
(Setting Setting
(minutes)
Score-MRI Lef--Gided DIP
(31.51 29 40.44
(0.891=00012, 0.802=00114)
6.2:0.122
(33.17+0.27, 00223=0.24)
(0,912=00021, 0.016=0.016)
2.5+:0,2
TV-DIP
(30.52::0.25, 29.20=0.37)
(0.872=0.022. 0.852=0.002)
2.5+0.11
ES-DIP
(31.02-0,34, 45)
00.882=00031, 0.858=0.028)
155::0.34
Vanilla DIP
(30.21+0.42, 28.75=0.333
(0.665+0.02. 0.842.4-.1122))
1,5:0,12
Self LuidGGidedd DIP
(33.6::0.23. 30,75:10.25)
00.922=00008.
0.874=00006)
4,5::.67
aSeqDIP (Ours)
(34.08=0.41, 31.34=0.47)
00.999 0.887:0.0099
2.240.12
MCG FBP
(3282=00.5, 31.35=0.449
(0.912=0000, 0.052=0.09)
6.4+0.2
(22.92::0.22. 19.52=0032)
(0.75=0.021.
0.68=00023)
0.2:0.011
Ref-Guided DIP
(31.21=0.24, 28.31=0.42)
00.892=0.023, 0.842=00021)
2.540.42
Vanilla DIP
(26.21 24.34=0.34)
00.791=0.021, 0.7122
1,4+021
Self-Guided DIP
(33.95 31.95=0.32)
(0.918++0002, 0.872--0.033)
4,540.56
aSeqDIP (Ours)
(34.88 0.36. 3..09
00.941 0.922100022
2.210.42
DFS Vanilla DIP
(31.02:0:25, 28,2
(0.912 0.852E0.021)
2.570.17
(30.48:0.28, 27.84=0.32)
(0.905=00021, 0.871=00030)
1,520,22
SGLD DIP
(30.58 2.212:0422)
00.908 0.077=0.017)
3,240.24
TV-DIP
(30.57 2.0
00.914=00022. 0.882-0.014)
2.540.24
Denoising
Rethinking-DIP
(30.98=0311, 28.67=0.25)
(0.912=:002, 0.887=0.03)
2.540.34
ES-DIP
(31.11+0.23 28.12=00.41
(0.914=00007,, 0.886=0.024)
145+0.44
Self-Guided DIP
031.21 28.31=0.35)
(0.916=0.02.
0.891=0.03)
3,54044
aScgDIP (Ours)
(3..11 28.97=0.44)
00.966=00021 0.008=00031)
2.410.45
DPS
(23,9:0 22.0310.36)
0087=00023
0.762=0.02)
2.500.3
Vanilla DIP
(22.5620.31, 2.6)
0..754=0.023. 0.0212
13:10,35
SGLD DIP
(23.09=0.55, =0.45)
00.772=0.0233, 0.732==00041
2,5:00455
In In-Painting
TV-DIP
(22.7 0..51)
(0.774++0004,
0.742++0.042)
2.5+0.31
ES-DIP
(23.33=0.44, 0.28)
00.781=00034, 0.745=00044)
1.25
Self-Guided DIP
(23.84 21,78=0,52)
(0.792=00042.
0,752==00064
3,540.45
aSeqDIP (Ours)
(24.56 0.45, 22.57-00447
(0.838 0.051, 1.778=0.045
2.4+0..54
DPS SGLD DIP
(23-00-0.556
(..76=000032)
2.2410.65
Deblurring
(0.720:20.03)
3.2410.55
Self Guided DIP aSeqDIP (Ours)
(20.34:10.55)
(0.732=00025)
3,44:1002
(23.89:10.40)
(0.792::0003))
2.5:0.778
Table Average PSNR, SSIM, and run-time results reported by our method against the selected baselines for the tasks of MRT reconstruction, CT reconstruction, image denoising, in-painting, and non-linear deblurring. Dntaa-nndeeendeecy in column indicates whether the methods depend on prior data pre-trained models. Setting and Setting 2, in the fourth and fifth columns correspond the scenarios in the second column of Table 1. For tasks with two settings, the run-time results are averaged over the two settings. Values past represent the standard deviation. See Appendix C.2 and Appendix C.3 for more comparison results.
running a large number of reverse sampling steps. When compared to Ref-Guide DIP, our method achieves higher PSNR and SSIM results without the need for any prior (or reference image) image.
4.4 Visualizations
Figure 5 shows reconstructed images for the five considered tasks using aSeqDIP and the other baselines. Each row corresponds task. The first column displays the ground truth (GT) image whereas the second column shows the degraded image. Column to the column before last present the reconstructed images by the baselines, while the last column shows the reconstructed images by DIPqDIP. PSNR values are provided at the bottom of each reconstructed image.
As observed, DIe achieves the highest PSNR scores. Additionally, the top right green boxes, which show the difference between the central region the reconstructed and GT images, indicate that for MRI and CT, our method visually exhibits the least difference, making it the closest to the GT.
A similar observation is seen for the denoising task for the zoomed in bottom box. For inpainting, we note that aSeqDIP introduces fewest unwanted artifacts as observed in the clouds (for DPS), and the left wing of the plane. While DSeqDIP contains artifacts for the task of Deblurring when compared to DPS, the latter generates a perceptually different image when compared the GT. Similar obscrvation noticed with the additional visualizations provided in Appendix C.7.

---

Degraded Image
Vanilla DIP
Ref-Guided DIP
SelFCuuded DIP
Scure-MRI
aSeyDIP
MRR
PSNR 36.12 6B
PSNR 33 78.8.88 GuldeG DIP
PSNR 52.94-BB
PSSR 1.46
GT
Degradd Image
Vanilla DIP
Ref-Guided DIP
MCG
DSe
PINK 29748
PSNit 12 am
FSNR 13.75 dB
PSNR 33.01
PSSR 34,34
Degraded
DIP
SelPGGudeed
aSe
Denoising
GT
Degraded Imaage
Vanilla DIP
SGLD DIP
Self-Guided DIP
DPS
la la-Paainting
PSKR HSNK 27.96 di DPS
PSNR: 28.35
GT
Degraded Image
Vanilla DIP
SGLD DIP
SoldeG DIP
aScqDIP
Deblurring
PSNR 25,7
PSNK 23 dB
PSNR 27.35 dB
Figure 5: Reconstructed/recoveeed images using proposed approach, aSeqDIP, and the baselines for the considered tasks. The ground truth (GT) and degraded images shown in the first and second columns, respectively, followed by three or baselines per task. The last column presents our method. PSNR results are given at the bottom of each reconstructed image. For MRI (8x undersampling) and CT (18 views), the top right box shows the absolute difference between the center region box of the reconstructed image and the same region in the GT image. Denoising and in painting used
25 and HIAR 0.25. For the task of Deblurring, aScqDIP contains artifacts when compared DPS. However, DPS generates perceptually different image when compared to the GT. For all other tasks, aSeqDIP reconstructions contain sharper and clearer image features than other methods.
5 Conclusions & Future Work
In this paper, we introduced Autoencoding Sequential Deep Image Prior (aSeqDIP), new unsuper- vised image recovery algorithm. Notably, aSeqDIP operates without the need of pre-trained models, relying solely sequential update of network parameters. These parameters are optimized using an anput-adaptive data consistency objective combined with autoencoding regularization, effectively mitigating noise overfitting. Our experimental results across various tasks highlight the competitive performance the proposed algorithm, matching (or outperforming) diffusion-baa -based methods in terms of reconstruction quality and required run time, all without the need for pre-trained models.
For future directions, we aim to explore the applicability of aSeqDIP other image recovery problems, thereby expanding its versatility and potential impact across diverse domains. Additionally, we are interested investigating the integration of network input update mechanism to dynamically adjust the autoencoding regularization parameter and the number of gradient updates iteration.
Acknowledgments
This work was supported by the National Science Foundation (NSF) grants CCF-2212065. CCF- 221 2066, and BCS-2215155. The authors would like to thank Xiang Li (University of Michigan), Avrajit Ghosh (Michigan State University). Huijie Zhang (University Michigan), and Siddhant Gautam (Michigan State University) for insightful discussions.
10

---

References
[1] Michael T McCann, Kyong Hwan Jin, and Michael Unser. Convolutional neural networks for inverse problems in imaging: A review. IEEE Signal Processing Magazine, 34(6):85-995 2017.
[2] B. Goyal, A. Dogra, S. Agrawal, B.S. Sohi, and A. Sharma. Image denoising review: From classical to state-of-the-ar approaches. Information Fusion, 55:220-244, 2020.
[3] Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 11956-11965, 2021.
[4] O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y. Akbari. Image inpainting: A review. Neural Processing Letters, 51:2007-2028, 2020.
[5] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero. Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[6] J. A. Fessler. Model-Based Image Reconstruction for MRI. IEEE Signal Processing Magazine, 27(4):81-89, 2010.
[7] I. A. Elbakri and J. A. Fessler. Statistical image reconstruction for polyenergetic X-ray computed tomography. IEEE Transactions Medical Imaging, 21(2):89-99, 2002.
[8] Lujun Zhai, Yonghui Wang, Suxia Cui, and Yu Zhou. A comprehensive review of deep learning-based real-world image restoration. IEEE Access, 11:21049-2108 2023.
[9] Hanene Ben Yedder, Ben Cardoen, and Ghassan Hamameh. Deep learning for biomedical image reconstruction: survey. Artificial intelligence review, 54(1):215-251, 2021.
[10] Shijun Liang, Anish Lahiri, and Saiprasad Ravishankar. Adaptive local neighborhood based neural networks for image reconstruction from undersampled data. IEEE Transactions Computational Imaging, 2024.
[11] Ling Chen, Zhishen Huang. Yong Long, and Saiprasad Ravishankar. Enhancing low-dose image reconstruction by integrating supervised and unsupervised learning. arXiv preprint arXiv:2311. / 2023.
[12] Bihan Wen, Saiprasad Ravishankar, Zhizhen Zhao, Raja Giryes, and Jong Chul Ye. Physics- driven machine learning for computational imaging: Part [from the guest editors]. IEEE Signal Processing Magazine, 40(2):13-15 2023.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68571 2020.
[14] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80: 102479, 2022.
[15] Hyungjin Chung, Bycongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems, 35:25683-25666, 2022.
[16] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusson posterior sampling for general noisy inverse problems. The Eleventh Interna- tional Conference on Learning Representations, 2023.
[17] Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning, 2024.
[18) Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9446-9454, 2018.
11

---

[19] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and cmpputrr-assister intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
[20] Shijun Liang, Evan Bell, Qing Qu, Rongrong Wang, and Saiprasad Ravishankar. Analysis of deep image prior and exploiting self- saf--guidance for image reconstruction. arXiv preprint urXiv:2402.04000 2024.
[21] Ismail R Alkhouri, George K Atia, and Alvaro Velasquez. A differentiable approach to the maximum independent set problem using dataless neural networks. Neural Networks, 155:168- 176, 2022.
[22] HH Barrett. The radon transform and its applications. PROGRESS IN OPTICS, 31:217-217, 1993.
[23] Avrajit Ghosh, Xitong Zhang, Kenneth K Sun, Qing Qu, Saiprasad Ravishankar, and Rongrong Wang- Optimal eye surgeon: Finding image priors through sparse generators at initialization. In Forty-first International Conference on Machine Learning, 2024.
[24] Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S Kamilov. Image restoration using total variation regularized deep image prior. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7715-7719. Ieee, 2019.
[25] Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective on the deep image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5443-5451, 2019.
[26] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-I1), pages 681-688, Citeseer, 2011.
[27] Hengkang Wang, Taihui Li, Zhong Zhuang, Tiancong Chen, Hengyue Liang, and Ju Sun. Early stopping for deep image prior. Transactions on Machine Learning Research, 2023.
[28] Taihui Li, Zhong Zhuang, Hengyue Liang, Le Peng, Hengkang Wang, and Ju Sun. Self- validation: Early stopping for single-instance deep generative priors. In Proceedings of the British Machine Vision Conference (BMVC), 2021., 2021.
[29] Di Zhao, Feng Zhao, and Yongjin Gan. Reference-drivee compressed sensing mr image recon- struction using deep convolutional neural networks without pre-training. Sensors, 20(1):308, 2020.
[30] Julián Tachella, Junqi Tang, and Mike Davies. The neural tangent link between enn denoisers and non-local filters. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8618-8621 2021.
[31] Xiang Li, Soo Min Kwon, Ismail R Alkhouri, Saiprasad Ravishankar, and Qing Qu De- coupled data consistency with diffusion purification for image restoration. arXiv preprint arXiv:2403. 06054, 2024.
[32] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfih International Conference on Learning Representutions, 2024.
[33] Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexan- dros G. Dimakis, Chul Ye, and Mauricio Delbracio. A survey on diffusion models for inverse problems. 2024.
[34] Y Xie and Li. Measurement-condditooned denoising diffusion probabilistic model for under- sampled medical image reconstruction. In International Conference on Medical Image Comput- ing and Cosputer-Assiseed Intervention, pages 655-664. Springer, 2022.
12

---

[35] A Güngör, S U Dar. ş Öztürk, Y Korkmaz, H A Bedel, G Elmas, M Ozbey, and T Çukur. Adaptive diffusion priors for accelerated MRI reconstruction. Medical Image Analysis, page 102872, 2023.
[36] C Peng, P Guo, S K Zhou, V M Patel, and R Chellappa. Towards performant and reliable undersampled MR reconstruction via diffusion model sampling. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 2022.
Springer,
[37] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[38] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
[39] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7): 167):166111177, 2011.
[40] J. L. Tamir, F. Ong. J. Y. Cheng, M. Uecker, and M. Lustig. Generalized magnetic resonance image reconstruction using the berkeley advanced reconstruction toolbox. In ISMRM Workshop on Data Sampling & Image Reconstruction, Sedona, AZ, 2016.
[41] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (3(4))600- 612, 2004.
[42] Gengsheng Zeng. Fast filtered backprojection algorithm for low-dose computed tomography. Journal of radiology and imaging, 4(7):45, 2020.
[43] Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5087-5096, 2021.
[44] A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd Neurips, NIPS' 18, page 8580-8589. Curran Associates Inc., 2018.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1026 1034, 2015.
[46] Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. -ucc E. Fox, and R. Garnett, editors, Advances in Neural Information Process- ing Systems, volume 32. Curran Associates, Inc., 2019.
[47] J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, et al. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv: 18/898, 2018.
[48] Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelcrated mri reconstruction. In Medical Image Computing and Computer Assisted Intervention-MICC. 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part 11 23, pages 13.73.60 Springer, 2020.
[49] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
13

---

Appendix
In the Appendix, we first shed more light on the impact of the DIP network input by studying the training dynamics using the neural tangent kernel for CNNs with residual connections. Next, show how trained autoencoders on clean images can be used as reconstructors at testing time. Lastly, provide additional experimental results and visualizations.
A
Case Study: Impact of the DIP Network Input through lens of Neural Tangent Kernel in Residual Networks
We show the impact of the DIP input through the lens of the Neural Tangent Kernel (NTK) [30, 44] for residual networks The NTK is a tool used to analyze the training dynamics of neural networks in the infinite width limit, where for CNNs the network width corresponds to the number of channels. In this limit, the change of any individual parameter during training becomes very small, which means that the change in the network's 's output during training can be accurately approximated by first order Taylor expansion around its initialization. In the context of DIP, we consider training a neural network f with parameters 0 and fixed input using gradient descent. At each training iteration, the network parameters are updated according to:
((++1) g( [VoC(ooc
where L is the loss function, and ,3 is the learning rate. We also consider the resulting change in the network's output due to this parameter update using the first order Taylor expansion:
fou+n (z) = foco (z) + Vefe(z) o= (0(++1) 8(()).
(8)
Substituting (7) into (8) and applying the chain rule to write
VoL(Jow (z)) = (Vofo(z)) oco) ( foce (x) ((Fcooz))
yields the equation:
Sgc (2) (z) (Vooo(z) (0 mcc))) ) ( lors (2) ((gor
(10)
1
In the infinite width limit, NTK theory states that the matrix ( (Vofo(z))e o- (Vo. (VoJ((z) a= )" stays fixed throughout training, so that O
()) for
all This matrix called the neural tangent kernel, and we denote it as θ. Moreover, because the parameters g(0) are initialized randomly, in the infinite width limit, the NTK θ becomes deterministic (as a function of z) due to the law of large numbers [30], and does not depend on the specific instantiation of g(0) In DIP, the loss function is the least squares loss given in (1). For simplicity, consider the denoising case, where the forward operator A I. Then, substituting the gradient the loss into (10) shows explicitly how the output of deep image prior evolves during training:
Sou () (z) = Soce (z) + () - Soco(()
(11)
Using this recursion relation, can derive closed form of the network output at iteration in terms of the initial output and NTK [20, 30]. The reconstruction at iteration t given by:
(z) = y - (y - foro (z)).
(12)
It is evident from (12) that the initial reconstruction of the network, feom (z), has important effects the training dynamics of DIP. Furthermore, networks used in DIP often feature skip connections from earlier layers to later ones, and it is natural to believe that these connections may cause the input to have a large effect on faco (z). In the following theorem, we analyze the training dynamics of CNNs with very similar architectural modification: residual connection that adds the input directly the network output.
"We note that skip and residual connections are not exactly the same as skip represents concatenation (typically from encoder to decoder end) and residual represents adding the input to the output. However, both operations correspond to sending initial input or features of network to its latter portion output.
14

---

Theorem A.1 (Dynamics of DIP with Residual Connections), Let be a convolutional neural network with parameters 6. We consider the complementary residual network f defined by fo(z) = + go(z). Suppose that J is trained using gradient descent with the loss [[fo(2))= 311=311100 fll - -yl.- Then, in the infinite width limit (number of channels), in expectation over the initialization of the parameters 0, we have that the ousput at training iteration is given by:
fars = -I- (0I-30)0 (y (y z).
(13)
The proof of Theorem A.I is provided in Appendix A.I, along with precise statement of the assumptions on the network architecture and parameter initialization. Additionally, in Appendix A.2 we provide simple experiment to validate that the training dynamics given in (13) hold for real networks.
Remark A.2. Theorem A.l can be used to understand how the choice of network input affects the performance of DIP for image denoising. To gain intuition, we consider two special cases. First, we consider using the noisy image y as the input z. In this case, equation (13) simplifies to Saus (y) =y for all iterations t. In this case absolutely no denoising occurs. On the other hand, we consider the oracle case where the clean image x is used as z. This gives us foce (x) =
y (I 30) - x). We see that at initialization (I = 0}, we already expect perfect demoising, since Soros (x) = y (I (y x) = x. These two cases support intuition thut using network inpul closer to the true image could result better performance in fewer training iterations.
A.1 Proof of Theorem A.1
Setting of Theorem A.I. We first precisely state the conditions of Theorem A. in particular the network architectures considered and the corresponding parameter initializations. The present setting is very similar to the setting considered in [30], but we provide the details here for completeness. We consider L layer CNN with Cn input channels and Cout output channels, with hidden channels in all intermediate layers. We assume that Ga, Cout << c. We assume all convolutions have a filter size of r. For simplicity, the network input and output are vectorized, so convolutions of any dimension are treated identically. For example, for a 2D CNN with 5 x 5 kernels, r = 25. Written explicitly, a network g with this architecture takes the form
go (z) = (x((CL1((( (CI.:(((CCCi)))))))))
where the operators C, represent convolutions with an additive bias, and & is a pointwise activation function such as ReLU. In this section, also consider the residual network architecture defined by fo(z) = z + (z).
We assume that the parameters are initialized using the He initialization [45]. With this initialization, the first layer convolutional filter weights are drawn from (0, and the filter weights for all other layers are drawn from (0, [i ), where the variance o depends the non-linearity used in the network. For ReLU networks, of 2 [45]. All biases are initialized to O.
Proof. In the setting described above, the NTK emerges in the limit e ~> co. A body of existing theory [30, 44, 46] establishes that in this limit the NTK is a deterministic matrix function of the network input z. This theory does not consider residual connections, but applies immediately to both g and f. For g, the NTK is given by
= (Vego ( ( (Vege(z) l0
(14)
However, we can see that (z)e(z)) |=0 = fefe -00000
Therefore, the linearization given in equation (10) holds for f using the same kernel θ and equation (12) describes the training dynamics of f.
Using equation (12), we can write: fece (z)
(I- 1-000(( feco) (z))
(15)
= - (I- --))0(s (y Joco (z))
(16)
To prove Theorem A. we consider the output face (z) in expectation over the initialization ((0)
Since all parameters ((0) are drawn from mean gaussian distributions, we find that Eggon [Moro] (z))
0 for any input z. Since is deterministic in the limit + oc, in expectation over g(0) equation (16)
reduces to faus (z) y (I 30) z), which proves Theorem A.I.
15

---

A.2 Example to support the results of Theorem A.1
We now provide a simple example using real networks to support the validity of Theorem A.1. This experiment substantiates both of the special cases considered in Remark A.2. Additionally, it shows that using the ground truth as the network input greatly inhibits overfitting. Indeed, in this experiment, find that residual network trained with the ground truth input takes approximately 25 times more training iterations to completely learn the noisy signal than residual network trained using a random noise input.
We use DIP for denoising a ID sinusoidal signal. We denote the clean signal x. The noisy signal is y
+ n, where .N(0, I). The network used is a five layer ReLU CNN with a residual connection. The full architecture can be written as f(z) = z - (RLLUC
where each C represents convolution (with bias). The signal has a size of 100, and the convolutions each have a filter size of 3, with 64 hidden channels. In all cases, the network is trained using gradient descent with learning rate of 5 x 10-1 The same seed was used to initialize the network in all cases.
Cs (ReLLUU()
We trained this network using three different inputs: the true signal x, the noisy signal y, and noise 2 ~ (0, 1). The results of training the network using these three inputs are shown Figure 6. We find that the results for training this real, reasonably sized network show the behavior predicted by Theorem A. which was obtained in the infinite width limit. Indeed, equation (13) predicts that when is used as the imput, the initial error will be small, with eventual overfitting to the noisy signal. This is observed in Figure 6, where the error is lowest at initialization, and it steadily increases throughout training. With this input, the network is highly resistant overfitting, requiring approximately 109 training iterations to completely fit the noisy signal. We also see that when y is used as the input, the error curve obtained is essentially flat and converges quickly to the error of the noisy signal. This agrees with the expectation that the network output will be y for all iterations t when y is used as the input. Finally, when random noise z is used as the input, the typical DIP behavior emerges.
Signals Used Denolsing Experiment
DI Denolsing Performance Using Different
12
Kakey Signely
Eror of Nuteork Ouquut 12, Narm
8-
Ground
Insut No 5igal y
Inout No x- AS5.0
-1
Noky Signal Erer (ly-lly-nn -Xib
20
60
1.00
100
101
101
1.0
Trining Rorazin
Figure 6: Ground truth signal and measurements (left), and results of the denoising experiment in Ap- pendix A.2 (right) to support the claims in Theorem A. and Remark A.2.
B Trained Autoencoders as Reconstructors
Here, we investigate how the autoencoder term in aSeqDIP is improving the reconstruction quality while mitigating the impact of noise overfitting, In particular, try to answer the question: Can an autoencoder trained on clean images operate as a reconstructor at testing time?
To address this question, we perform the following steps: (i) train an autoencoder on fully sampled measurements or clean data images and (ii) utilize the trained autoencoder with unseen subsampled or corrupted measurements, optimizing over the input using the DIP objective with the autoencoder term. This enables the autoencoder to function as an image reconstructor. Specifically, given training dataset, D. comprising unperturbed images or fully sampled MRI/CT data, denoted by x, we train an autoencoder U-Net g _ + R with parameters t.. The training process seeks to obtain & as
3 argmin
(x) -x1
(17)
XED
16

---

PONN (d8)
Average scars
1.0
1.5
2.0
2.5
3.0
Ap: Autoencader Regulanzztion
Figure 7: Average PSNR (y-axis) of 8 MRI images (with 4x undersampling) obtained by optimizing the input trained autoencoder (using (17)) w.r.t. different values the regularization parameter Xa (18) (x-axis).
Subsequently, given unseen measurements y and the learned autoencoder's parameters v. test the reconstruction of
f- argmin Agg (z) yl12 >allga(z) z112.
(18)
where AS R4 is regularization parameter. We perform this experiment by training & using 3000 fully sampled seans from fastMRI dataset [47]. then evaluate the reconstruction quality of the trained encoder using scans from the fastMRI testing set. The average PSNR results for different values Aa are depicted in Figure
observed, a trained autoencoder effectively serves reconstructor evidenced by the achieved PSNR. Thus, deduce that the autoencoder term in aSeqDIP not only mitigates noise overfitting but also enhances the reconstruction quality as an important prior.
C Additional Experiments
C.1 Robustness to Noise Overfitting for the Denoising Task
Denoising
32.5
30.0
27.5
25.0
22.5
DIP (Ours)
20.0
Self-Guided
17.5
SGLD DIP
Vanilla DIP
15.0
2000
4000
6000
8000
10000
Iterations
Figure 8: Average PSNR results of 20 images w.r.t. the optimization iteration using aSeqDIP and DIP-based baselines.
In this subsection, we illustrate aSeqDIP's robustness to the noise overfitting issue for the denoising task. The average PSNR for 20 images from the CBSD68 dataset denoising using aSeqDIP and other DIP-based methods are given in Figure 8. We observe two key points. First, in addition to higher PSNR, aSeqDIP shows higher robustness against noise overfitting compared other DIP-based methods, consistent with MRI and CT results in Figure 4. Second, unlike MRI and CT, the onset of noise overfitting occurs earlier, but the subsequent decay is very small.
C.2 Comparison with VarNet: An End-to-End MRT Supervised Method
Here, we compare DIe with the End-to-End (E2E) MRI supervised model [48] that uses the variational network (VarNet). Results are given Table As observed, we slightly under-perform when compared to VarNet (trained on 8000 data points) for the task of MRI reconstruction all without
17

---

requiring any labeled training data. It is important to note that, at inference, E2E models only require few unrolligg steps, whereas aSeqDIP an optimization method that requires train the network parameters for each new set of measurements.
Task
Method
Data Independency PSNR (1)
E2E VarNet (trained on 8000 data points from fastMRII (48)
34.89
MRI
E2E VarNet (Irained on 3000 data points from fastMRI) [48]
33,78
aSegDII (Ours)
34.08
Table 3: Average PSNR results (over 20 MRI scans at 4x undersampling from the testing set of fastMRI)
reported by our method against E2E VarNet [48] (pre-trained on fastMRI) for the task of MRI reconstruction.
C.3
Comparison with DM-based Methods on the FFHQ Dataset
Task
Method
Data Independency
PSNR (t)
DPS (trained FFHQ) [16]
31.45
Denoising
DDNM (trained on FFHQ) [49)
X
31.65
aSeDDIP (Ours)
DPS (trained on FFHQ) [16]
24.54
Random tu-Painting
DDNM (trained FFHQ)
aSeDDIP (Ours)
25.76
DPS (trined FFHQ) [16]
23.67
Deblurring
DDNM (trained on FFHQ) [49]
23.88
aScqDIP (Ouns)
24.02
(trained on FFHQ) [16]
22.67
Box Painting
DDNM (trained FFHQ)
22.89
aSeqDIP (Ours)
22.3
Table Average PSNR results reported by our method against DPS as well as more recent leading method DDNM [49] for four image restoration tasks: Denoising, Random In- Painting, non-linear Deblurring, and Box In In-Painting.
Here, we present average PSNR results (averaged over 20 images) for the tasks of denoising, random inpainting (97% missing pixels), box-in- eox-in-paintin (with HIAR of 0.25), and non- con-innerr deblurring of our method versus Denoising Diffusion Null-Space Model (DDNM) [49] and DPS [16] on the FFHQ testing dataset. For DPS and DDNM, we used a pre-trained model that was trained on the training set of FFHQ. As observed, our training-data-free method achieves competitive slightly improved results when compared to data-intensive methods on all tasks other than box-inpainting (for which we under-perform by less than dB), all without requiring pre-trained model.
C.4 Ablation Study on the Regularization Parameter in DSP
35
30
Ancrage mostrimerto Angulaization Tatemiior
Average messrrment Sepulattation fammmntrr
Aworage 22 messuremens with inguloritation Pammerer 2
2000
4000
6000
8000
100000
Iteracion
Figure 9: Average PSNR results of 20 MRI (with 4x undersampling) seans aSeqDIP for the cases where
A E {0.5, 1, 2} and [10000).
In this section, we conduct an ablation study on the choice the autoencoding regularization parameter, X, in (3), Specifically, conduct experiment using 20 MRI scans to examine the impact of À on the reconstruction quality and noise overfitting in aSeqDIP. We note that while the main results in Section 4 use NK K 4000, in these experiments, number of iterations to investigate the onset of noise overfitting. We set = 5000 and = 2 for this purpose.
algorithm for extended
In this experiment, we run DSeqDIP with values of À E {0.5, 1,2). Average PSNR results are given in Figure 9. It is evident that, average, using À 1 yields the most favorable results in terms of
18

---

PSNR values, which is our selected choice. Furthermore, we observe that for À 0.5 (red), the start of the PSNR decay (the onset of noise overfitting) precedes that of À 1 and À 2 (blue and black).
C.5 Ablation Study on N and K in DIP
In this section, we conduct an ablation study to investigate the impact of the number of gradient updates (N) per one set of parameter (K) in aSeqDIP. Specifically, we report the PSNR results across the tasks of MRI, CT. denoising, and in- nn-painting for the case of NK 4000, considering combinations of (N, K) as (1, 4000), (2, 2000), and (4, 1000). The results, presented in Figure 10, reveal that across all tasks considered, the combination of N = 2 and K 2000 consistently yields the most favorable results terms of PSNR.
MRI Ax
HAIR 0.25
35.00
34.95
(dt)
34.90
25.25
34.85
2SWR
33.00
32.75
33.25
34.65
36.70
36.75
36.00
31.9
25.03
25.35
25.10
[1,4000] 12,2000) (6,1000)
(1,4000) (2,2000) (4,1000)
(3,4000] (2,2000) (4,1000)
[1,4000] (2,2000) (4,1000)
(0.
Figure 10: Ablation study the choice of the number of gradient updates N per U-Net, and the number of U-Nets K in terms PSNR using the four considered tasks the case of NK 4000.
C.6 Additional Implementation Details
For denoising, each noisy RGB image is 512 x 512 and is generated by adding an additive Gaussian white noise with noise levels descired in Table 1. For in-painting, consider central region mask, and we evaluate two hole-to-image area ratios (HIAR) with image size 512 x 512.
For Vanilla DIP, Self-Guided DIP, Reference- guffre DIP, TV-DIP, Rethinking DIP, and SGLD DIP, we use 4000 iterations. For TV-DIP, we set the regularization parameter For ES-DIP, we use the default configuration provided by the authors for the three considered tasks.
The reference image in Reference Reference-guided guided DIP is chosen as (using distance metric such as Eu- clidean distance or other metric) that which is most similar to an estimated test reconstruction from undersampled data or sparse-view data.
For DM-based approaches, we use the codes attached to the authors' papers. Specifically, Score-MRI [14], MCG [15], and DPS [16]. For experiments with natural images (denoising. in-painting, and non-linear deblurring) in Table 2, we used the CBSD68 dataset. As such, for DPS (the DM-based method), utilized a pre-trained model that was trained on very large and diverse dataset which is ImageNet 128 x 128, 256 x 256, and 512 x 512. This pre-trained model is much more generalizable when compared to the other option which was trained on FFHQ (a dataset of faces). According to [17], the ImageNet pre- pre-trained model has high generalizability. For the FFHQ comparison results in Appendix C.3, we used an FFHQ- pre-traieed DM for DPS and DDNM.
For MRI, the pre-trained model used in Score-MRI was originally trained on natural images then fine-tuned using the training set of fastMRI. Similar approach was used for the CT pre-trained model used MCG.
C.7 Additional Visualizations
Figures 11 and 12 present additional MRI visualisations, whereas Figures 13 and 14 present CT visualizations. Samples from the natural image restoration tasks are given in Figure 15, Figure 16, and Figure 17 for box- box-inpainting, denoising, and non-linee linear deblurring respectively.
19

---

Ground Truth
Input
Self-Guided
Reference Guided
PSNR = oc dB Score MRI
PSNR 22112 dB
PSNR 34.01 dB Vanilla DIP
PSNR = 32.45 DB
DSeqDIP
PSNR 31.2
PSNR 34.68 dB
PSNR 30.26 dB
Figure 1I: Visualization of ground-truti -rutt and reconstructed images using different methods of knee image from the fastMRI datasct with 4x k-space undersampling. A region of interest is shown with a green box and its eтror (magnitude) is shown in the panel the top right. DIP provides the sharpest and clearest reconstruction of image features.
Ground Truth
Input
Self-Guided
Reference Guided
PSNR dB
PSNR 16.67 aSeqDIP
PSNR 32,95 dB
PSNR 32.31 dB
Score MRI
Vanilla DIP
Figure 12: Visualization of ground-truth and reconstructed images using different methods of a knee
image from the fastMRI dataset with k-space undersampling. region of interest shown with green box and its error (magnitude) is shown in the panel on the top right. aSeqDIP provides the clearest reconstruction of image features amongst the methods.
20

---

Ground Truth
Input
Self-Guided
Reference Guided
PSNR oc dB MCG
PSNR 17.12
PSNR 33.05
PSNR = 32.31
aSeqDIP
Vanilla DIP
PSNR = 32.47 dB
PSNR = 33.48 dB
PSNR = 29.77 dB
Figure 13: Visualization of ground-truth and reconstructed images using different methods scan from the AAPM dataset with 18 views. A region of interest is shown with green box and its error (magnitude) is shown in the panel on the top right. aIPqDIP provides the sharpest and clearest reconstruction of image fcatures.
Ground Truth
Input
Self-Guided
Reference Guided
oo dB MCG
PSNR 22.67 DIP
PSNR
PSNR 33.31
Vanilla DIP
PSNR = 34.15 JB
PSNR = 35.18 JB
PSNR = 30.16 dB
Figure 14: Visualization of ground-truth and reconstructed images using different methods of CT scan from the AAPM dataset with with 30 views. A region interest is shown with a green box and its error (magnitude) is shown in the panel the top right. aSeqDIP provides better reconstruction of small and low-contrast image features.
21

---

Ground Truth
Input
Self-Guided
SGLD
PSNR dB DPS
PSNR 14.56
PSNR 24.21 dB Vanilla DIP
PSNR 23.02
aScqDIP
PSNR
Figure 15: In-painting example with 0.1 HIAR where image restorations of different methods given using example from the CBSD66 dataset. The diffusion-hased DPS produces spurious (although sharp) content the hole region while aSeqDIP much better preserves features original ground truth.
Ground Truth
Input
Self-Guided
SGLD
U.S.A
U.A
PSNR = oc dB DPS
PSNR 19.68 dB
PSNR = 30.19 dB Vanilla DIP
PSNR = 29.62 dB
DIP
PSNR
PSNR
PSNR
Denoising example with 25 where image restorations of different methods given using example from the CBSD68 dataset. aSeqDIP provides the sharpest and clear reconstruction of image features.
Figure 16:
22

---

Ground Truth
Input
Self-Guided
SGLD
PSNR dB DPS
PSNR 14.24 dB
PSNR 27.82 dB
PSNR 22.62 dB
DIP
Vanilla DIP
PSNR 19.82
PSNR 28.67
PSNR 25.24
Non-linear deblurring samples where image restorations of different methods are given using an example from the FFHQ datasct.
Figure 17:
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The Abstract and Introduction highlight the challenges of previous works, leading the motivation of the our proposed approach, followed by the contributions the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 3, we show that while we use the autoencoding regularization term to mitigate the issue of noise overfitting, the problem still exists. Nevertheless, is highly mitigated when compared previous works, This was supported by experiments.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: For the first remark, we provide the assumptions needed to state the results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the detailed settings, baselines, and forward models for each imaging task we consider (Table 1). Furthermore, the implementation details for baselines are provided in Appendix C.6.
5. Open access to data and code
23

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code that contains all the experiments of our method and most baselines are included in the repo link we provide.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section provide all the details needed to understand the results.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We report average PSNR and SSIM scores well standard deviation in Section 3 for the main experiments.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: In the beginning Section we exactly specify the GPU machine used to run our method and all other baselines. Furthermore, in Table 2, we report run time results.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http:://///erree http///eerrrpspeeeeeee http:///....pp.cccccccstttteeeea
Answer: [Yes]
Justification: Authors have reviewed the code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper addresses the problem of image reconstruction, and our method can not be used for image manipulation. Therefore, there is no societal impact of the work performed.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We cite the code used for baselines, and provide the links used for each dataset.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
24

---

Justification: We cite the code used for baselines, and provide the links used for each dataset.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification: We only release code for our algorithm.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
25


Reviews:
[{"review_id": "qQKLTEQ7Et", "reviewer": "Reviewer_qnNG", "summary": "This paper investigates how to prevent deep image prior (DIP) from overfitting to the noise or compressed measurements, which is a classic problem of DIP. To address the problem, the authors proposed Autoencoding Sequential DIP (aSeqDIP). The general idea of aSeqDIP is to cuts the overall training process into K sequential blocks, in each of which the input ($\\mathbf{z_k}$) of the network is the previous output ($f(\\mathbf{z_{k-1}})$), and the loss is defined as $\\mathcal{L} = ||Af(\\mathbf{z}_k) - y||_2^2 + || z_k - f(\\mathbf{z}_k)||_2^2$, where the second term forces the output to be alike the input. As k increases, the network aims to find the network $f$ such that $z = f(z)$ and $||Af(\\mathbf{z}_k) - y||_2^2$ is minimized. In a word, aSeqDIP forces the output to be a fixed point of the network that fits the measurements as much as possible.", "questions": "1. Please consider including a baseline like DIP+TV in the experiment.\n\n2. What is the runtime of aSeqDIP? A comparison between aSeqDIP and other baselines is preferred.\n\n3. [Subjective] The proposition seems redundant: 1) it assumes very strong conditions (the convergence of the network), which, in my opinion distracts the reader from the key message; 2) it kinda shows how aSeqDIP prevent itself from overfitting, but it is not very clear without text. I suggest to write the proposition in the form of plain text with equations.", "limitations": "Please discuss the limitations such as runtime, practical convergence, etc.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n\n2. The proposed approach for enforcing DIP to find its fixed point is interesting and novel\n\n3. Superior performance has been demonstrated again current diffusion-model-based approaches for image reconstruction tasks.", "weaknesses": "1. Although the experiment is thorough in terms of the variety of image reconstruction tasks, I still think it misses an important baseline that uses explicit regularizer (e.g. TV) to stablize the learning process. It would strengthen the argument if the authors could include a comparison like that.\n\n2. The current introduction is concise, but it does not convey the message that the key benefit of aSeqDIP is to address the problem of overfitting.\n\n3. The authors do not discuss the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1+Q1: Comparison with TV-DIP**: We thank the reviewer for their comment. In what follows, we include a comparison with TV-DIP in terms of the average PSNR over 15 scans in MRI (with 4x and the fastMRI test dataset) and 20 images from the CBSD68 dataset for denoising and in-painting. As observed, our method outperforms TV-DIP.\n\n| Method | MRI | Denoising | In-painting |\n|----------|----------|----------|----------|\n| TV-DIP | 31.04 | 30.57 | 22.67 |\n| aSeqDIP (Ours) | 34.08 | 31.51 | 24.56 |\n\n**C2: The current introduction does not convey the message that the key benefit of aSeqDIP is to address the problem of overfitting**: We acknowledge the reviewer's comment. In the revised Introduction Section of the paper, we will emphasize that the main problem of DIP-based methods is noise overfitting as the selection of the number of optimization steps in DIP can differ not only from task to task but also from image to image within the same task. We will add a discussion that the main goal of introducing aSeqDIP is to mitigate the issue of noise over-fitting through the input-adaptive procedure and the use of the auto-encoding term.\n\n**C3 Discussing the limitations such as runtime and practical convergence**: Please see our global response.  \n\n**Q2 Runtime of aSeqDIP**: We thank the reviewer for their comment. In the last column of Table 2, we report the average run-time (minutes) of all the methods. For the second best PSNR and SSIM (self-guided DIP), our method is 2X faster for MRI and CT reconstruction and requires 1 minute less than self-Guided DIP for denoising and in-painting. When compared to DM-based methods, aSeqDIP requires a slightly less run time while achieving an improvement in terms of the reconstruction quality (PSNR and SSIM). While DM-based methods only require function evaluations and our method is an optimization-based approach, the generally larger run-time reported for DM-based methods is due to the necessity of running a large number of reverse sampling steps.\n\n**Q3 [Subjective] The convergence assumption in the Proposition**: We agree with the reviewer that the convergence assumption in Proposition 3.1 is strong. The main point we'd like to convey in this proposition is that aSeqDIP is trying to solve the optimization problem in (5), which is different that Vanilla DIP. In Remark 3.2, we discuss this point by comparing aSeqDIP and Vanilla DIP. In the revised manuscript, we will present the point of the proposition as a remark for further clarification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DCwAUKg0Hf", "reviewer": "Reviewer_MAra", "summary": "The authors in this paper propose an Autoencoding Sequential DIP (aSeqDIP) which aims to address the overfitting issue of DIP while without introducing extra parameters. The idea is very simple, the authors simply feed the output of the DIP into DIP model after each N updates. The authors validate the efficiency of their methods on several different image restoration tasks.", "questions": "I have several questions which have been listed in [Weaknesses].", "limitations": "I have listed the limitations in [Weaknesses].", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1) The presentation of the paper is good and it is very easy to understand.\n2) The authors have conducted experiments on for different image restoration tasks.\n3) The authors provide numerical comparisons as well as visual comparisons.", "weaknesses": "I have several concerns about this paper.\n\n1) The novelty is significantly limited. It is almost the original DIP. The only difference is: the original DIP each iteration uses a random noise as its input; while here, the output of DIP is fed into the DIP for the next N iteration's updates.\n\n2) The current experiments are all linear image restoration tasks. It would be interesting to see how this model works for non-linear image restoration tasks such as image delurring.\n\n3) The authors should compare their method with more advanced DIPs such as Ref1.\n\n4) How does this model compare with SOTA diffusion models Ref2?\n\n5) The current results (Table2), the improvement is very small. And I am wondering what if we run the experiments many rounds and report the mean and std. \n\n\nRef1: Jo, Yeonsik, Se Young Chun, and Jonghyun Choi. \"Rethinking deep image prior for denoising.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nRef2: Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1 Novelty and Differences with the original DIP**: We appreciate the reviewer's comment. We would like to emphasize that our proposed method differs significantly from other DIP-based methods, including Vanilla DIP. These distinctions, which we believe set our work apart, are highlighted below.\n\n- Formulation and Algorithmic Perspective: In addition to the input-adaptive nature of Algorithm 1 (which is motivated by the discussion in Section 3.1), we propose the use of the auto-encoding term (Section 3.2.1) which is not used in Vanilla DIP. Furthermore, the original DIP does not require the input to be an image, whereas in our case, it is an image-to-image mapping.\n- Noise Overfitting Perspective: The proposed input-adaptive procedure along with the auto-encoding term result in the major key benefit of significantly delaying the noise overfitting decay which is the main challenge in all of the DIP-based approaches. See Figure 4, Figure 10, and the first figure in the PDF attached in the global response of this rebuttal.\n- Applicability Perspective: In our paper, we presented experimental results using multiple reconstruction and restoration tasks. Most other DIP-based methods consider at most one to three tasks. For example, ref-guided DIP only considered MRI, whereas TV-DIP considered denoising and in-painting. Furthermore, in addition to the four tasks in the paper, in this rebuttal, we included a non-linear inverse imaging task which is non-uniform deblurring (see our response to the following comment).\n\nBased on the highlighted remarks, we respectfully ask the reviewer to reconsider their opinion regarding novelty. \n\n\n**C2 It would be interesting to see how this model works for non-linear image restoration tasks such as image deblurring**: We thank the reviewer for their comment. Here, we include results of the non-linear non-uniform image deblurring task using the setting in the DPS code. In particular, we use the ''blur-kernel-space-exploring'' setting. In what follows, we report the achieved PSNR (averaged over 25 images form the CBSD68 dataset) of aSeqDIP when compared to DPS, Self-guided DIP, and SGLD-DIP. As observed, our method significantly outperforms all DIP-based methods while reporting improved results when compared to DPS.\n\n| Task | DPS | Self-Guided DIP | SGLD-DIP | aSeqDIP (Ours)\n|----------|----------|----------|----------|----------|\n| Non-uniform Deblurring | 23.4 | 20.3 | 19.8 | 23.89 |\n\n**C3 Comparison with \"Rethinking DIP for denoising\"**: Thank you for your comment. In what follows, we compare aSeqDIP with the suggested paper. We use the task of denoising and report the average PSNR over 25 images from the CBSD68 dataset. As observed, our method reports higher PSNR. We believe that our approach, exploiting autoencoding regularization, keenly taps into the generative and denoising nature of CNNs for more explicit regularization to alleviate overfitting.\n\n| Task | Rethinking DIP for Denoising | aSeqDIP (Ours)\n|----------|----------|----------|\n| Denoising | 30.98 | 31.51 |\n\n**C4 Comparison with SOTA DM-based method, DDNM**: Thank you for your question. In what follows, we present average PSNR results (averaged over 20 images) for the tasks of denoising (with $\\sigma_d=25$), random in-painting (97\\% missing pixels), box-in-painting (with HIAR of 25), and non-uniform deblurring of our method versus DDNM  (the suggested paper) and DPS on the FFHQ testing dataset. For DPS and DDNM, we used a pre-trained model that was trained on the training set of FFHQ. As observed, our training-data-free method achieves competitive or slightly improved results when compared to data-intensive methods on all tasks other than box-inpainting (for which we slightly under-perform), all without requiring a pre-trained model.\n| Method | Denoising | Random In-painting | Non-uniform Deblurring | Box In-painting\n|----------|----------|----------|----------|----------|\nDDNM (using FFHQ-trained DM) | 31.45 | 25.34 |23.88 | 22.89|\nDPS (using FFHQ-trained DM) | 31.65 | 25.54 | 23.67 | 22.67|\naSeqDIP (Ours) | 31.77 | 25.76 | 24.02 | 22.3 |\n\n**C5 The current results (Table2), the improvement is very small**: Thank you for your comment. While the PSNR and SSIM improvements compared to other baselines are not very significant, we would like to highlight the following points. First, compared to DIP-based methods, our approach not only achieves higher reconstruction quality but also significantly improves robustness to noise overfitting. See the PSNR curves in Figures 4 and 10, as well as in the figure in the attached PDF in the global response. Second, when compared to DM-based methods, our approach not only achieves comparable or slightly improved PSNR and SSIM scores but also has the significant advantage of being independent of any training data and pre-trained models. We hope that emphasizing these points will highlight the additional advantages offered by aSeqDIP.\n\nRegarding running the experiments for many rounds, do you mean for our method or the other baselines? Or do you mean running our method with different initializations of $\\phi_1$?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l1rwnhC0Pu", "reviewer": "Reviewer_hjHc", "summary": "This manuscript describes a variant of the seminar Deep Image Prior (DIP) work that incorporates aspects and mindset of the iterative prediction workflow in diffusion models.\nIn particular, the proposed procedure (method?) aSeqDIP is training a network to predict a single and fixed but distorted (e.g. noisy) image when given a pixel-wise random input image (just as DIP has initially proposed). The key novelty is to only train for a few training steps before switching to feeding the current prediction as an updated input (instead of the initial noisy input image). The similarity to predictions with diffusion models is apparent.\nThe approach is to the best of my knowledge novel and the idea, in my point of view, the biggest contribution of the paper.", "questions": "* I cannot judge the quality or sufficiency of the used baseline methods. If for any of the 4 tasks the current SOTA (or industry standard) method would also be given and compared against, I would find that very useful.\n* When does the proposed approach work (when is it applicable) and what are known limitations of its applicability? (When does the inductive bias of the setup not suit the desired task?)", "limitations": "* I would expect that the inductive biases that comes with the used network and training procedure dictate for what problems the presented method/procedure can produce good results. If the authors have any thoughts, I think the manuscript would much benefit from a short discussion in that vein.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The idea is fantastic and thought provoking. \n* Introduction of the regularization term (autoencoding term) during iterative training and showing that it is useful and that tuning it is important. (Figure 7 from the supplement could make it into the main paper.)\n* Application to 4 different tasks (MRI and CT (important real-world use-cases) and denoising and in-painting (to make the CV community happy… ;)\n* Partially informative appendix.", "weaknesses": "* It surprises me that the manuscript does not argue more about WHY this approach leads to better results even when compared to data-dependent baselines. How strong are these baselines? Do better pre-trained methods exist? If so, why not show results with them as well?\n   * An answer to the above “WHY?” question would go right to the heart of why diffusion works well and would therefore be very interesting.\n   * Since the presented method/procedure is not dependent on any amount of available training data (but the single compromised target image) I wonder how this manuscript can avoid talking about inductive biases of the used network architecture (a UNet, hence, a CNN).\n* The manuscript is overall rather compact in content, the text certainly not too dense but maybe even a bit too repetitive and wordy, and the figures not very legible (adequately sized). I also much regret not to see the compromised target image used by aSeqDIP, but only GT and final predictions.\n* Figure 1: when seeing it first it really did not help me understand the paper any better. After reading the entire manuscript and coming back to Figure 1 I can confirm the figure makes sense, but one would hope that Figure 1 is more educational as it currently is.\n* Figure captions are in general not terrible but also not in all cases making the figures self-contained. It is necessary to find some potentially distant place in the manuscript to fully grasp the visuals (e.g. Fig. 2).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1 WHY this approach achieves better results? Strength of the DM-based baselines. Inductive bias discussion**: We thank the reviewer for their constructive comment. The answers to these questions will definitely strengthen our paper. We divided our response to this comment into the following three parts. \n\n- Answer to why we achieve better results when compared to diffusion models: Our approach and DM-based methods are conceptually different. Provided a set of training images, DMs are trained to approximate the underlying distribution $p(x)$. Subsequently, when employed to solve imaging inverse problems (IPs), the problem becomes sampling from the conditional distribution $p(x\\mid y)$ where $y$ denotes the measurements. To this end, DM-based IP solvers attempt to approximate this conditional distribution (which may not be accurate and/or suitable for some tasks) using different approaches for which the reverse sampling steps are modified to achieve this target. aSeqDIP is different in the sense that it is an optimization-based method that depends solely on an input-adaptive Unet architecture and the generative power of the network, and a loss function that is designed to mitigate irrelevant noise overfitting (which is the major obstacle with DIP-based methods). We argue that the reverse sampling modifications needed to sample from the conditional distribution in DM-based methods present a more challenging design choice when compared to our approach.\n\n- Answer to how strong the data-dependent baselines are: We believe that the DM-based methods that we used as baselines are very strong on the tasks they were considered for. For MRI and CT, we used the following criteria: Select a method that perform strongly for every task, and use the test dataset with the same training dataset distribution for which these DMs were trained on. For example, for MRI, we used Score-MRI as baseline. This method utilizes a pre-trained DM that was originally trained on natural images and then fine-tuned with the training set of the fastMRI dataset. In our MRI experiments, we used the testing set of fastMRI. Similar approach was used in CT where we used the MCG method (with an AAPM dataset-trained DM) and the AAPM testing dataset. \n\nFor our experiments with natural images (denoising, in-painting, and non-uniform deblurring which is added in this rebuttal), we used the CBSD68 dataset. As such, for DPS (the DM-based method), we utilized a pre-trained model that was trained on a very large and diverse dataset which is ImageNet 128X128, 256X256, and 512X512. This pre-trained model is much more generalizable when compared to the other option which was trained on FFHQ (a dataset of faces). According to `The Emergence of Reproducibility and Consistency in Diffusion Models', the ImageNet pre-trained model has high generalizability. In our response to Reviewer MAra Comment 4, we experimented with the testing set of the FFHQ dataset where the pre-trained models of DPS and DDNM (the DM-based baselines) were trained on the training set of FFHQ.\n\nThe main message of our comparison with data-centric methods is to demonstrate that our method can achieve competitive or superior results compared to DM-based methods, all without requiring pre-trained models and training data.\n\n- Answer to why we generally achieve better results: We think that our method, similar to all DIP-based methods, benefits from the implicit bias inherited in the Unet architecture. The structure of a randomly initialized CNN is used as a prior as it was shown in the original DIP paper. The architecture of a generator network alone is capable of capturing a significant amount of low-level image statistics even before any learning takes place. However, the number of optimization steps required in DIP represents a challenge. *Therefore, we believe that our method, exploiting autoencoding regularization and input-updating, keenly taps into the generative and denoising nature of CNNs for more explicit regularization to alleviate overfitting*.\n\n**C2 Text is not too dense but a bit too repetitive and wordy. Including compromised target images**: We do agree that some points are sort of repeated in the Introduction and other places in the paper. In the revised manuscript, we will improve the writing and readability of the figures. In the PDF attached in the global response of this rebuttal, we included the degraded images in our visualizations of Figure 5. We will do the ones in the Appendix in the revised manuscript.\n\n**C3 Figure 1 location and explanation**: We acknowledge the reviewer's comment. In the revised manuscript, we will either elaborate more in the caption of this figure (or add more context to the diagram itself) or re-locate it until after we present our method.\n\n**C4 Figures captions**: Thank you for your comment. We will improve the captions in the revised manuscript.\n\n**Q1 Sufficiency of the used baseline methods**: To address your question about how strong the DM-based baselines are, please refer to our response to your first comment. See also the second part of our response to comment 3 of Reviewer Q1ff where we experimented with a leading end-2-end supervised reconstruction model. Regarding the DIP-based baselines, we would like to highlight that we considered the recent self-guided DIP work which has demonstrated highly competitive performance in terms of reconstruction quality and robustness to noise overfitting across multiple tasks.\n\n**Q2 known limitations of aSeqDIP**: Thank you for your question. Please see our global response. Additionally, we evaluate our approach using different additional tasks, baselines, and settings. We hope that these results will shed more light into the capabilities and limitations of our method. In particular, for run-time and practical convergence, see our response to comment 3 of Reviewer qnNG. For testing with a non-linear task, see our response to Reviewer MAra comments 2 and 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PRmQHsNwd0", "reviewer": "Reviewer_Q1ff", "summary": "The paper proposes Autoencoding Sequential Deep Image Prior (aSeqDIP) an extension to the family of untrained (no training data required) DIP methods. \nAs in previous work a combination of data consistency loss and autoencoding regularization (a loss between network input and output) is used. However, instead of further regularizing the optimization by adding noise to the network input in every iteration as proposed in previous work, this paper varies the network input by setting it to the network output after every two iterations. \nExperimental evidence is provided that demonstrates superior robustness to overfitting and overall reconstruction accuracy compared to other DIP approaches and trained diffusion models.", "questions": "1. In the Figures (like Fig. 5) showing qualitative reconstruction results it would be helpful to show the measurement, i.e. noise image for denoising, mask fr in-painting and zero-filled reconstruction for MRI.\n\n2. I personally find the formulation in line 10 \"sequential optimization of multiple network architectures\" a bit missleading as the networks are not changed or re-initialized. Only the network input is adapted. \n\n3. Regarding Section 3.1 and Figure 2 it makes sense to me why the blue curve is decaying, but do you know why the red curve is decaying? Does the red curve decaying imply that a constant input (like all zeros) to the DIP would work best?", "limitations": "Limitations are not discussed in the paper. This could be for example the limitted interpretability of the comparison to trained baseline methods as current results in the paper significantly fall behind the results reported in the baseline papers or the long reconstruction times compared to end-to-end deep learning based methods.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Significance: Untrained neural networks are an important technique used in many fields, where training data is scarce. Mitigating the problem of overfitting is an important contribution. \n\nOriginality: Despite the relatively small difference compared to previous work (regularize network input by setting it equal the output instead of adding noise to it as in the self-guided DIP) the importance and effectiveness of this idea seems to be justified by the experimental results in terms of reconstruction accuracy and robustness to overfitting.\n\nClarity: Overall the paper is written in a clear way and related work in terms of DIP approaches and diffusion models are discussed.\n\nQuality: The experiments towards evaluating the reconstruction performance and overfitting robustness of the proposed DIP approach are well designed and ablation studies for the most important hyperparameters are provided. \n\nOverall an interesting paper. However, some questions remain and some important points remain unclear (see weeknesses). If those can be addressed adequately, I would consider raising the score.", "weaknesses": "**1. No results are provided for the most interesting case: Overfitting robustness for the task of denoising.**\nTo my understanding Prop. 3.1 says that in the limit the proposed approach converges to a network that has learned the identity and to a network output (which is identical to the network input) that perfectly fulfills data consistency under the forward operator and the given measurement. \nFor the tasks of MRI and inpainting (in the absence of noise) solutions that fulfill Prop. 3.1 include the ground truth image and the empirical evidence provided in the paper shows that indeed the solution found by the proposed method is closer to the ground truth image than that of other DIP based approaches (why it is closer remains unclear up to intuitions/speculations, but answering this question is non-trivial and does not have to be the scope of the paper). \n\nAlso, the robustness to overfitting seems reasonable for the aforementioned tasks. My intuition would be that repeatedly setting the network input to the output combined with the autoencoding regularization loss lets the network converge to identity relatively quickly. If at the same time data consistency is reached, there is no incentive for the network to change its weights anymore and a stable solution is reached. \n\n**However, things change in the presence of noise or in general for the task of denoising.** \nTo my understanding, for denoising Prop. 3.1 is fulfilled, when the network perfectly reconstructs the noisy measurement $\\mathbf{y}$, which is the very definition of overfitting. So, here it would be interesting to see curves as presented in Figure 4 for MRI and CT also for denoising to demonstrate to what extend overfitting is still prevented.\n \n**2. In general, I am not sure about the usefulness of Prop. 3.1.** \nAs discussed above for the tasks of MRI and inpainting reconstructions that fulfill Prop 3.1 comprise good and bad solutions however without making a statement why a good solution should be preferably reached over a bad solution, whereas for the task of denoising it implies convergence to a bad solution. \n\n**3. The learning based \"SOTA\" baselines perform very badly, which is not discussed in the paper and no information regarding the used training sets is provided.** \nDiffusion model-based methods are introduced as the state-of-the-art in the paper (line 113) and indeed the works referenced there report impressive results in terms of quantitative and qualitative reconstruction performance. \nYet, in this work the reconstruction scores and especially the reconstructed images (Figure 5, 12, 13) contain severe artifacts. I understand that training and inference of those diffusion models is sensitive to hyperparameters, but then potential reasons for this bad performance should at least be discussed in the paper. Also, no information is provided regarding the training data used for those baseline models.\n\nIf there is no comparison to state-of-the-art end-to-end approaches (which definitely should give better results than DIP based reconstruction if trained properly on enough data) like the end-to-end VarNet (https://arxiv.org/abs/2004.06688) then at least the diffusion model based baseline should show reasonable results or if not it should be discussed why. \n\n**4. Code is only provided for the task of MRI reconstruction and the implemented method seems to differ from the description in the paper.** Checking the code provided via a link (https://anonymous.4open.science/r/Aseq_DIP-E728/README) in the paper I can only find a notebook that performs MRI reconstruction *Aseq_DIP.ipynb*. \nIn the code the step where the network output is set to be the network input for the following steps is implemented as follows:\n```\n    pred_ksp = mps_and_gt_to_ksp(mps1.to(device), net_output)\n\n    new_pred_ksp = (1 - mask_from_file).to(device) * pred_ksp.detach() / scale_factor + mask_from_file * ksp1\n\n    new_ref = ksp_and_mps_to_gt(new_pred_ksp, mps1)\n    random_smoothing_temp = torch.zeros_like(new_ref).to(device)\n\n    for jj in range(1):\n        random_smoothing = new_ref + 1e-5 * torch.rand((640, 372), dtype=torch.complex64).to(device)\n        random_smoothing_temp += random_smoothing\n\n    random_smoothing_final = random_smoothing_temp \n    ref[:, 0, :, :] = random_smoothing_final.real\n    ref[:, 1, :, :] = random_smoothing_final.imag\n```\nIf I understand it correctly the new input (ref) is first set to fulfill perfect data consistency and then processed with random smoothing, two things not mentioned anywhere in the paper. Also, the new input (ref) is updated before the computation of the loss, which means that in fact the autoencoding regularizer is not really computed between the network input and the network applied to this same input.\nIt is unclear how essential these two steps (hard data consistency and smoothing) are to the proposed approach.\n\nFurther, \n```\n    loss.backward()\n    for i in range(2):\n        optimizer.step()\n```\nimplies that the network weights are updated twice without updating the loss functions, which is not what is outlined in Algorithm 1 of the paper.\n\nIf that is really how the mehtod is implemented that corresponds to the results in the paper the ablation study regarding the values for the number of iterations $N$ per network in Appendix C.4 seems meaningless as the loss is not updated as $N$ increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1 Overfitting robustness for the task of denoising**: Since our method is an unsupervised approach optimizing a single image, our observations suggest that repeatedly setting the network input to the output, along with autoencoding regularization, enables convergence to autoencoding-specific images instead of the identity.\n\nThe attached PDF in the global response presents the average PSNR for 20 images from the CBSD68 dataset for denoising. We observe two key points. First, aSeqDIP shows higher robustness against noise overfitting compared to other DIP-based methods, consistent with MRI and CT findings. Second, unlike MRI and CT, the onset of noise overfitting occurs earlier, but the subsequent decay is very small. Please refer to our response to the following comment for the proposition.\n\n**C2 Usefulness of Prop. 3.1**: The main point of this proposition is to highlight that the aSeqDIP algorithm minimizes a different optimization problem compared to Vanilla DIP. Proposition 3.1 is not intended to indicate solution quality or analyze RMSE during optimization but to state that, under strong convergence assumptions, our algorithm minimizes the optimization problem in (5). Solving (5) explicitly is challenging due to the equality constraint and the network's non-linearity. In other words, the proposition illustrates the types of solutions our algorithm converges to. Our empirical results show that our algorithm converges quickly and remains stable. \n\n**C3 Performance of DM-based methods and their training data**: Regarding the difference in PSNR values between our reported results and those in Score-MRI and MCG, we'd like to clarify that for our MRI experiments, we used Cartesian sampling, a more common scheme, while Score-MRI used Uniform 1D, Gaussian 1D, Gaussian 2D, and VD Poisson disk sampling. For CT, our aSeqDIP results (and other baselines except MCG) were obtained using the full 512x512 pixels in the AAPM dataset, whereas MCG downsized the images to 256x256 pixels for faster training and sampling (see the caption of Table 3 in the MCG paper). Therefore, we compared the ground truth after resizing the MCG results back to 512x512. We will include this discussion in the revised paper. In the following table, we compare aSeqDIP (a dataless method) versus MCG (a data-centric method) at their downsized pixel space, as reported in Table 3 of their paper. As observed, we achieve very competitive results.\n\n| Task | MCG 256X256 | aSeqDIP 256X256 |\n|----------|----------|----------|\n| Sparse View CT with 18 views | 33.75 | 33.86 | \n| Sparse View CT with 30 views | 36.09 | 35.89 | \n\nWe use the testing images from the fastMRI and AAPM datasets. The pre-trained models for Score-MRI and MCG were trained on the training sets of these datasets.\n\nFor our experiments with natural images (denoising, in-painting, and non-uniform deblurring), we used the CBSD68 dataset. For DPS (a DM-based method), we utilized a pre-trained model from ImageNet (128x128, 256x256, and 512x512), known for its high generalizability according to \"The Emergence of Reproducibility and Consistency in Diffusion Models.\" This model is more generalizable than the alternative trained on FFHQ (faces dataset). In response to Reviewer MAra Comment 4, we experimented with the FFHQ testing set, using pre-trained models of DPS and DDNM trained on the FFHQ training set.\n\nFor the DM-based baselines, we used the default hyper-parameters provided by DPS (natural images), Score-MRI, and MCG (sparse view CT).\n\nNote that we are not the first to observe unwanted artifacts in DPS (see Figure 3 in \"Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency\"). We believe these artifacts arise because DM-based approaches, which learn $p(x)$, require modifications to sample from $p(x | y)$. These modifications, often approximations, may not be accurate or suitable for all tasks and images.\n\nA major point of our comparison is to show that our method achieves competitive or superior results compared to DM-based methods, without needing pre-trained models or training data.\n\nEnd-to-end (E2E) supervised models outperform many DIP-based methods like VarNet because the best DIP optimization steps can vary significantly. This motivates our aSeqDIP approach. The table below shows that our method achieves higher average PSNR than VarNet for 15 MRI scans (4x). While E2E models are faster at inference with a single forward pass, our method's advantage is that it is fully data-independent.\n\n| Task | VarNet | aSeqDIP (Ours) |\n|----------|----------|----------|\n| MRI | 33.78 | 34.08 | \n\n**C4 Code**: Thank you for bringing this to our attention. We mistakenly uploaded an older version of our code. We have fixed the link and uploaded .py files for all the tasks.\n\n\n**Q1 Including measurements**: See our global response. \n\n**Q2 Comment on using \"sequential optimization of multiple network architectures\"**: In our algorithm, each network is initialized by the optimized parameters of the previous network, and then optimized using (3) such that the input of the network is the output of the previously optimized network (see line 157). No re-initialization or architecture change are needed. In the revised manuscript, we will re-word it for further clarification.\n\n**Q3 Curves of Figure 2**: Larger variance in the standard Gaussian distribution corresponds to larger additive perturbations even for the case of $\\mathbf{x}^*=\\mathbf{0}$ (the red curve). We conjecture that this still leads to larger distances from the GT and hence worst performance. In regard to the case where the input is the all-zero vector, we ran Vanilla DIP with $\\mathbf{z}=\\mathbf{0}$, and the reconstruction quality is very low. We conjecture that this is due to (i) the impact of the first convolutional layer, which is only its bias, and (ii) the output of the first layer is concatenated to later layers through the skip connection.\n\n**Limitation Comment**: See our global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

We thank the reviewers for their constructive comments. Many reviewers raised questions about the limitations and capabilities of the proposed approach, and requested experiments with different tasks, settings, and baselines. Below, we discuss these limitations and summarize the additional experiments and results obtained over the past week. These discussions will be included in the revised manuscript.

**Limitation Discussions**:

- Reviewer qnNG - Comment 3: We thank the reviewer for their comment. In terms of run time, we would like to point out that in Remark 3.4, we discuss the computational requirements of the proposed method, which is determined by (i) the $N\times K$ parameter updates, and (ii) the number of function evaluations necessary for updating the input of every network which is $K$. Furthermore, in Table 2, we include average run-time results of aSeqDIP as compared to the considered baselines. Furthermore, we have shown that our method is not that sensitive to the selection of $N$ and $K$ (along with $\lambda$) as, for the four considered tasks, we selected the same values. As our adoption of the autoencoding term delays the start of the PSNR decay, the empirical convergence plots for our method reaches nearly 5\% of its best possible PSNR at around iteration 2000 (see Figure 4).
- Reviewer hjHc - Question 2: A known limitation of our method is its slow run-time when compared to the inference time of End-to-End (E2E) supervised reconstruction models. However, it is important to note that our method, as it is a DIP-based approach, operates without any training data and pre-trained models.
- Reviewer Q1ff - Limitation Comment: Thank you for your comment. See our response to your third Comment that (i) justifies why the reported results in our paper and the DM-based baselines are slightly different, and (ii) includes additional aSeqDIP CT results on MCG's downsized pixel space. In regards to run-time, we agree with the reviewer that the inference of E2E methods is very fast when compared to our method and the DM-based methods. The reason is that E2E methods requires only one forward pass (or a few in the case of unrolling networks). However, it is important to note that our method, as it is a DIP-based approach, operates without any training data and pre-trained models. 


**Summary of Additional Results**: 
- MRI, Denoising, and In-painting PSNR results with aSeqDIP (Ours) vs. TV-DIP. (Reviewer qnNG). 
- Non-uniform image Deblurring (a non-linear inverse imaging task) PSNR results of our method as compared with DPS, Self-guided DIP, and SGLD. (Reviewer MAra)
- Denoising PSNR results of our method as compared to Rethinking DIP. (Reviewer MAra)
- Denoising, non-uniform deblurring, and in-painting results of our method as compared with DDNM and DPS using the FFHQ testing dataset. (Reviewer MAra)
- CT reconstruction results of aSeqDIP as compared to MCG using the downsized 256X256 setting. (Reviewer Q1ff)
- MRI PSNR results of our method as compared with a supervised E2E method, VarNet. (Reviewer Q1ff)
- Average PSNR curves for the task of denoising with respect to iteration $i$ for aSeqDIP, Vanilla DIP, and Self-Guided DIP is given in the **attached PDF**. (Reviewer Q1ff)
- The degraded images in Figure 5 are given in the **attached PDF**. (Reviewers hjHc and Q1ff)

Meta-review: Authors present a novel DIP method that uses a sequence of DIPs akin
to diffusion models. The idea is backed up by theory and as far as I
can see, experimental results also support the idea.

Reviewers raised concerns regarding the experimental setup and as far
as I can see, authors addressed them as good as possible. Most
reviewers are happy with the rebuttal.

One reviewer raise concerns even after the rebuttal about comparisons
and experimental results. While these concerns have valid points, I
think the work is quite good. 

Given the novelty in the article, supported by 3 reviewers, supportive
experimental results and the fact that it attained 1BA, 1WA and 1A, I
think the paper would be a great contribution to the conference.
Decision: Accept (poster)