Image Reconstruction Via Autoencoding Sequential Deep Image Prior

Authors: ismail r alkhouri
shijun  liang
evan  bell
qing  qu
rongrong  wang
saiprasad  ravishankar
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:16:40

Abstract: Recently, Deep Image Prior (DIP) has emerged as an effective unsupervised one-shot learner, delivering competitive results across various image recovery problems. This method only requires the noisy measurements and a forward operator, relying solely on deep networks initialized with random noise to learn and restore the structure of the data. However, DIP is notorious for its vulnerability to overfitting due to the overparameterization of the network. Building upon insights into the impact of the DIP input and drawing inspiration from the gradual denoising process in cutting-edge diffusion models, we introduce Autoencoding Sequential DIP (aSeqDIP) for image reconstruction. This method progressively denoises and reconstructs the image through a sequential optimization of network weights. This is achieved using an input-adaptive DIP objective, combined with an autoencoding regularization term. Compared to diffusion models, our method does not require training data and outperforms other DIP-based methods in mitigating noise overfitting while maintaining a similar number of parameter updates as Vanilla DIP. Through extensive experiments, we validate the effectiveness of our method in various image reconstruction tasks, such as MRI and CT reconstruction, as well as in image restoration tasks like image denoising, inpainting, and non-linear deblurring.

Main paper: 
 introduction: 
 Inverse imaging problems arise across various real-world applications [1] . These include tasks such as image denoising [2] , image deblurring [3] , restoring missing portions of images [4] (in-painting), super-resolution [5] , magnetic resonance imaging (MRI) [6] , and reconstructing X-ray computed tomography (CT) scans [7] . The common challenge in these scenarios is the reconstruction of images from limited and/or corrupted measurements. their application to data-limited or data-corrupted tasks such as in medical imaging (MRI and CT). To address this challenge, previous DM-based MRI and CT methods typically fine-tune pre-trained DMs (trained on natural images) using limited fully-sampled data points, which are not as readily available as natural images [14, 15] . As such, there is a pressing need to develop methods that can alleviate the reliance on large, clean datasets and/or pre-trained models.One well-known method is Deep Image Prior (DIP) [18] , which does not require pre-trained models and relies on the parameters of an architecture (e.g., U-Net [19] ). DIP finds a solution without needing any data other than the limited/corrupted task-specific measurements and the forward operator. DIP has shown competitive reconstruction/restoration results in inverse imaging problems such as MRI [20] and image denoising [18] . However, DIP (and most of its variants) suffer from the problem of overfitting.In this paper, we introduce Autoencoding Sequential DIP (aSeqDIP) to mitigate noise overfitting artifacts. Our approach comprises a sequential update of network weights. A significant advantage of our approach, in contrast to DM-based methods, is its ability to operate without the need for training data or pre-trained models, thus fitting into the framework of dataless training [21] . Notably, the total number of parameter updates in aSeqDIP remains equivalent to that of Vanilla DIP [18] . Furthermore, unlike a recent work [20] , aSeqDIP eliminates the need for gradient-based input updates.• Building upon an insight about the impact of the DIP network input, we introduce Autoencoding Sequential DIP (aSeqDIP), which incorporates a U-Net architecture whose weights are updated sequentially. These updates are based on objective functions that consist of (i) an input-adaptive data consistency term and (ii) an autoencoding regularization term used for noise overfitting mitigation. • Our extensive experimental evaluations, in terms of standard image reconstruction metrics and required run-time, highlight the superior (or competitive) performance of aSeqDIP compared to DIP-based and leading DM-based methods for the tasks of MRI and CT reconstruction, denoising, in-painting, and non-linear deblurring.Organization: In Section 2, we provide an overview of the considered tasks, and review recent advancements in both DIP-based and DM-based methods. Section 3 introduces our approach, aSeqDIP. Our experimental results are presented in Section 4, followed by discussions on conclusions and future directions in Section 5.2 Preliminaries & Related Work 
the image reconstruction problem: 
 Image reconstruction tasks can be defined as recovering an image x * ∈ R n from measurements y ∈ R m , where m ≤ n, governed by the forward operator A. For multi-coil MRI, the forward operator is A = MFS, where M denotes coil-wise undersampling, F is the coil-by-coil Fourier transform, and S represents the sensitivity encoding with multiple coils 2 . For CT, a simplified forward operator will be used to study the sparse-views setting: A = CR, where C is an undersampling operator that selects specific projection views or angles, and R denotes the radon transform [22] (corresponding to parallel-beam geometry in CT). For both MRI and CT with limited acquired data (done to accelerate scan or reduce X-ray dose, etc.), the condition m < n applies.of low-level image statistics even before any learning takes place. Specifically, the DIP image reconstruction is obtained through the minimization of the following objective:EQUATIONwhere x is the reconstructed image, and θ corresponds to the parameters of network f : R n → R n , which is typically implemented using a U-Net architecture [19] . The input to the network, z ∈ R n , is randomly chosen and remains fixed throughout the optimization process. While standard DIP was shown to perform well in many tasks, selecting the number of iterations to optimize objective (1) poses a challenge as the network would eventually fit the noise present in y or could fit to undesired images based on the null space of A.To mitigate the problem of noise overfitting, previous studies considered different approaches such as regularization, early stopping (ES), and network pruning [23] . For regularization-based methods, the work in [24] enhanced the standard DIP by introducing a total variation (TV) regularization term for denoising and deblurring tasks, whereas the study in [25] proposed combining DIP with stochastic gradient Langevin dynamics (SGLD) [26] . The authors in [27] use running variance as the criterion for ES, whereas the authors of [28] propose combining self-validation and training to apply ES.The input to the standard DIP (or Vanilla DIP) network is a random noise vector that, in most works, remains fixed during the optimization. Nevertheless, other works, such as those in [29] and [30] , have explored cases where the input contains some structure of the ground truth. The approach employed in reference-guided DIP (Ref-Guided DIP) [29] follows the same objective as standard DIP in (1). However, instead of using a fixed random noise vector as input, it utilizes a reference image closely resembling the one undergoing reconstruction. This method was applied to the task of MRI. This methodology proves particularly effective when datasets comprising structurally similar data points are available. The reference required here makes this method a data-dependent approach.Inspired by the departure from using a random fixed input, the authors in [20] recently introduced Self-Guided DIP. Unlike Ref-Guided DIP [29] , a prior image that closely resembles the unknown (to be estimated) image is not needed, and the optimization occurs simultaneously with respect to both the input and the parameters of the network. Specifically, Self-Guided DIP employs the following objective:EQUATION)where η is random noise, and α is a regularization parameter. The first (resp. second) term is used for data consistency (resp. denoising regularization) and final reconstruction is obtained asx = E η [f θ (ẑ + η)]. aSeqDIP is different from Self-Guided DIP as our method does not require gradient-based updates for the input, making it computationally less expensive. Self-Guided DIP has demonstrated superior performance compared to Vanilla DIP, TV-DIP, and SGLD-DIP, thus serving as a primary baseline for comparison.In recent years, there has been an abundance of DM-based methods proposed to address inverse imaging problems [14, 15, 16, 31, 32, 33] . A well-known method for natural images is Diffusion Posterior sampling (DPS) [16] . DPS incorporates a gradient step into the reverse sampling process of pre-trained DMs, ensuring data consistency and enabling sampling from the conditional distribution. In the context of image reconstruction and restoration tasks, numerous diffusion-based approaches have emerged, as evidenced by works such as [34, 35, 36] . Notably, the authors in [14] and [15] introduced a SOTA DM-based approach for addressing the MRI and CT reconstruction inverse problems, respectively. They propose incorporating the predictor-corrector sampling algorithm [37] for data consistency, akin to DPS, thereby facilitating the sampling from a conditional distribution.One clear distinction between aSeqDIP and DM-based methods is that our approach does not necessitate pre-trained models. For our experiments in MRI, CT, and denoising (as well as in-painting and deblurring) tasks, we will utilize Score-MRI [14] , Manifold Constrained Gradient (MCG) [15] , and DPS [16] , respectively, as DM-based baselines. 
autoencoding sequential dip: 
 In this section, we begin by investigating the impact of the input on DIP. Then, we introduce our method, aSeqDIP. We note that while we consider linear and non-linear inverse problems, in our formulations, we use a linear forward model to simplify notation.The Impact of the Network Input in Vanilla DIPHere, we aim to address the question: How does employing a noisy version of the ground truth image, which retains some structure of the ground truth, as the fixed input to the Vanilla DIP objective in (1), affect performance? To investigate, we conduct the following experiment.Consider the MRI task defined as y ≈ Ax * . Let the input to the standard DIP objective in (1) be denoted as z = x * + δ, where δ ∼ N (0, σ 2 I). Here, σ controls the magnitude of the perturbations added to the ground truth image, indicating that a larger σ results in a greater deviation between z and x * . We optimize (1) for various values of σ, recording the best possible PSNR compared to the ground truth, i.e., prior to the start of the noise overfitting decay.Figure 1 displays the average results for 8 images. Notably, for all images, a closer similarity of the DIP network input to x * , as indicated by σ, corresponds to higher reconstruction quality, measured by PSNR. Larger variance in the standard Gaussian distribution corresponds to larger additive perturbations even for the case of x * = 0 (the red curve). We conjecture that this still leads to larger distances from the ground truth and hence worse performance.0.2 0.5 0. Based on this discussion, a notable insight emerges:The proximity of the DIP network input to the ground truth correlates with the quality of the reconstruction. This promotes the question: Can we develop an input-adaptive DIP method that mitigates noise overfitting?We proceed to address this question by proposing our method, which we refer to as Autoencoding Sequential DIP (aSeqDIP). In Appendix A, we provide a case study and theory on the impact of the DIP network input through the lens of the Neural Tangent Kernel in residual networks. The onset of severe noise overfitting therein is delayed for better inputs (Appendix A.2).Consider that we have a U-Net architecture defined by f : R n → R n whose weights are given by ϕ k , where k ∈ [K], and [K] := {1, . . . , K}. Each set of parameters in f ϕ k takes an input z k and outputs f ϕ k (z k ). Based on the insight from the previous subsection, we initially set z 0 to y (resp. A H y) for denoising, in-painting, and deblurring (resp. MRI and CT). The initialization of ϕ 1 follows the same initialization as any other DIP-based method. The parameters in f ϕ k , and the input, z k , are then updated sequentially throughEQUATIONEQUATIONwhere λ ∈ R + is a regularization parameter, and the initialization of ϕ k is the optimized ϕ k-1 in (3). The final reconstruction is given as:EQUATIONThe proposed procedure outlined in (3) and (4) consists of two key components. First, the optimization of each set of weights in f ϕ k using an objective that consists of the data consistency term and the Algorithm 1 Autoencoding Sequential Deep Image Prior (aSeqDIP).Input: Measurements y, forward operator A, number of input updates K, number of gradient updates N per input update, regularization parameter λ, and learning rate β. Output: Reconstructed image x. Initialization: z0 = A H y; ϕ0 ∼ N (0, I).1: For each k ∈ [K] 2: Initialize ϕ (0) k ← ϕ (N )k-1 for k ∈ {2, . . . , K}, and ϕk ← ϕ (0) for k = 1.For each i ∈ [N ]. (Network parameters update) 4: second autoencoding term that aims to alleviate noise overfitting. Second, the update of the input, z k , after optimizing each set of weights f ϕ k , so that our method is input-adaptive.ϕ (i) k = ϕ (i-1) k -β∇ ϕ k ∥Af ϕ k (z k-1 ) -y∥ 2 2 + λ∥f ϕ k (z k-1 ) -z k-1 ∥ 2 2 ϕ k =ϕ (i-1) k . 5: Obtain z k := f ϕ (N ) k (z k-1Algorithm 1 presents the procedure of our proposed approach. As inputs, the algorithm takes y, A, K, N , λ, and the learning rate β. Apart from the measurements and the forward operator, the remaining parameters are considered hyper-parameters, typical in most DIP-based methods. The parameters in f ϕ k are set to ϕ k-1 (step 2) and subsequently optimized for N iterations using a gradient-based optimizer, such as gradient descent (as depicted in Algorithm 1) or Adam [38] . A block diagram of our proposed aSeqDIP method is presented in Figure 2 .In the following remarks, we provide insights into our proposed aSeqDIP method. Remark 3.1 (Differences from Vanilla DIP [18] ). Assume that the iterates of ( 3) and ( 4) converge, i.e., as k → ∞, z k → z * and ϕ k → ϕ * . Then, according to (4), for a continuous mapping f , we have z * = f ϕ * (z * ). Substituting this into (3) in the limit, we getEQUATIONThe limit points of aSeqDIP correspond to the solution of a constrained version of the Vanilla DIP objective in (1) . The constraint enforces additional prior that could alleviate overfitting. While its not straightforward to use a gradient-based algorithm for (6) given the hard constraint, the aSeqDIP scheme's limit points nevertheless minimize (6) . Furthermore, aSeqDIP automatically estimates the network input by a sequential feed forward process without needing expensive updates. The main point is to show that aSeqDIP is solving the optimization problem in (6) , which is different than Vanilla DIP in (1).Remark 3.2 (Differences from Self-Guided DIP [20] ). While both aSeqDIP and Self-Guided DIP [20] update the input and network parameters simultaneously, there exist fundamental differences. Firstly, Self-Guided DIP solves the optimization problem in (2) which does not strictly enforce the auto-encoder constraint z = f ϕ (z) as in (6) . Secondly, aSeqDIP only requires a network forward pass to update z, resulting in significantly fewer computations as will further be demonstrated in our experimental results. Thirdly, the second term in the aSeqDIP objective does not require computing the expectation, as it is an auto-encoder rather than a denoiser that results in higher resistance to noise overfitting. Lastly, our method does not require initializing z randomly and generating random vectors (η in ( 2)). The selection of η introduces an additional hyper-parameter that we avoid, focusing solely Optimize Params. on selecting N K (total number of iterations) and λ (regularization strength), which are necessary in most DIP-based methods.Remark 3.3 (Computational Requirements). The computational requirements of aSeqDIP are determined by two factors: (i) the N K gradient-based parameter updates, and (ii) the number of function evaluations necessary for updating z, which is K. In our experiments, we have found that setting N = 2 and K = 2000 is generally sufficient. This configuration makes aSeqDIP nearly as efficient as Vanilla DIP.Remark 3.4 (Relationship to DMs). aSeqDIP bears resemblance to the reverse process in DMs due to their shared gradual denoising steps. However, despite these similarities, several distinctions emerge. Firstly, unlike the DM network, aSeqDIP does not require encoding a scalar representing time t. Secondly, and perhaps most significantly, aSeqDIP operates without requiring any training data or pre-trained networks. Thirdly, aSeqDIP operates in a truly sequential manner in terms of time, whereas in DMs, whether it's training (e.g., denoising score matching [39] ) or sampling, the prevalent technique involves sampling from time t ∼ U[0, 1] (uniform distribution), which allows for non-sequential time points.Figure 3 illustrates how different approaches compare to aSeqDIP.In DIP-based approaches, noise overfitting occurs as the network attempts to fit its output to the noisy or subsampled measurements, y, as k increases during training. However, the specific value of k at which this PSNR decay begins is uncertain and varies across tasks and even among images within the same task and distribution. In aSeqDIP, when the output of network f ϕ k improves compared to that of f ϕ k-1 , the autoencoder term enforces similarity between the input and output of the network, thus delaying the onset of noise overfitting decay. This occurs because we are not only enforcing the network output to be measurement-consistent, but also enforcing that the output and input become similar. Consequently, as k increases, noise fitting is delayed, and utilizing the autoencoder provides regularization against noise overfitting. In Section 4, we will demonstrate how the proposed autoencoding term effectively regulates noise overfitting.One might expect that incorporating the autoencoder could negatively impact reconstruction quality. However, empirical observations reveal that not only is noise overfitting delayed with the autoencoder term, but also image reconstruction quality is enhanced. To further support this statement, in Appendix B, we investigate whether a trained autoencoder on clean images can act as a reconstructor at testing time by optimizing the input. Vanilla DIP [18] , ES-DIP [27] , Ref-Guided DIP [29] TV-DIP [24] , Self-Guided DIP [20] Score-MRI [14] CT views ∈ {18, 30}Vanilla DIP [18] , Self-Guided DIP [20] , Ref-Guided DIP [29] Filter Back Projection (FBP) [42] MCG [15] Denoising σd ∈ {15, 30}Vanilla DIP [18] , ES-DIP [27] , Self-Guided DIP [20] , DPS [16] TV-DIP [24] , Rethinking-DIP [43] , SGLD-DIP [25] In-painting HIAR ∈ {0.1, 0.25}Vanilla DIP [18] , ES-DIP [27] , Self-Guided DIP [20] , DPS [16] SGLD-DIP [25] , TV-DIP [24] Deblurring BKSE [3] Self-Guided DIP [20] DPS [16] SGLD-DIP [25] Table 1 : Tasks, settings, and baselines considered in our experiments. For MRI, we consider two Acceleration (Ax) factors, 4x and 8x, that determine the subsampling of the measurements. For 2D CT (parallel beam geometry), we use two sparse view settings: 18 and 30 views. For denoising, we perturb the ground truth images using two noise levels determined by σ d . In in-painting, we use two hole-to-image area ratios (HIAR), 0.1 and 0.25. For non-linear deblurring, we use the Blurring Kernel Space Exploring (BKSE) setting [3] , described in Equations ( 56) to (59) of [16] . Each baseline that utilizes pre-trained models or a reference image is considered data-dependent. Further details are provided in Appendix C.6. 
settings, datasets, and baselines: 
 In our experiments, we consider five tasks: MRI reconstruction from undersampled measurements, sparse-view CT image reconstruction, denoising, non-linear deblurring and in-painting. For MRI, we use the fastMRI dataset 3 . The forward model is y ≈ Ax * . The multi-coil data is obtained using 15 coils and is cropped to a resolution of 320 × 320 pixels. To simulate undersampling of the MRI k-space, we use a Cartesian mask with 4x and 8x accelerations. Sensitivity maps for the coils are obtained using the BART toolbox [40] . For CT, we use the AAPM dataset 4 . For parallel beam CT, the input image with 512 × 512 pixels is transformed into its sinogram representation using a Radon transform (the operator A). The forward model assuming a monoenergetic source and no scatter, noise is y i = I 0 e -[Ax * ]i , with I 0 denoting the number of incident photons per ray (assumed to be 1 for simplicity) and i indexing the ith measurement or detector pixel. We use the post-log measurements for reconstruction. We use a full set of 180 projection angles and simulate two different sparse view acquisition scenarios (with equispaced angles). Specifically, we created cases with 18 and 30 angles/views. The image resolution is kept at a fixed size.For the tasks of denoising, in-painting, and non-linear deblurring, we use the CBSD68 dataset 5 . For each task, we use 20 measurements/corrupted images. To evaluate the reconstruction quality, we use the Peak Signal to Noise Ratio (PSNR), and the Structural SIMilarity (SSIM) index [41] . For experimental settings and baselines, see Table 1 and its caption. Note that we consider datadependent and data-independent baselines as shown in the third and fourth columns of Table 1 . All the experiments are conducted on a single RTX5000 GPU machine. Further implementation details are provided in Appendix C.6.For the proposed aSeqDIP method in Algorithm 1, we use the Adam optimizer with learning rate of β = 0.0001. Furthermore, the regularization parameter is set to λ = 1 following the ablation study in Appendix C.4. We select N = 2 and K = 2000 following the ablation study in Appendix C.5.In this subsection, we showcase the impact of the proposed autoencoding regularization in aSeqDIP on noise or null space (nuisance) overfitting.We conducted experiments using 20 MRI scans and 20 CT scans, considering two cases of aSeqDIP as outlined in Algorithm 1. The first case sets λ = 1, consistent with the remainder of the paper, while the second case sets λ = 0, effectively disabling the autoencoding regularization term in (3). Additionally, for comparison, we report results for Vanilla DIP and Self-Guided DIP. The average PSNR results for these cases are depicted in Figure 4 .As observed, when the autoencoder term is disabled in aSeqDIP (blue dashed lines), noise overfitting in MRI, akin to Self-Guided DIP, begins after nearly 1600 iterations. For CT, we note that aSeqDIP without regularization starts noise overfitting at around iteration 3800, whereas Self-Guided DIP experiences PSNR decay earlier, after approximately 1250 iterations. Importantly, when the autoencoding term is utilized (blue solid lines), not only does the decay in noise overfitting not commence until after iteration 4000, but the reconstruction quality (measured by PSNR) also improves.As expected, PSNR decay in Vanilla DIP begins early, at around iteration 500 and 750 for MRI and CT, respectively. In Appendix C.4, we provide an ablation study to better show the impact of the value of λ in aSeqDIP.Here, we present our primary results regarding the reconstruction quality, measured by PSNR and SSIM, as well as the associated run-time. Table 2 presents the results for the considered tasks in this paper. Column 3 indicates whether the baselines depend on prior data or pre-trained models.The last three columns provide the PSNR, SSIM, and run-time results where the arrows indicate favorable results. For PSNR and SSIM, the settings correspond to the second column of Table 1 . The black (resp. blue) text corresponds to the first (resp. second) setting. Values after the ± sign indicate standard deviation. Subsequently, we offer observations on the main results.Compared to data-independent methods, i.e., the baselines that do not depend on a reference image or pre-trained models, aSeqDIP demonstrates improved PSNR and SSIM scores. For example, aSeqDIP, apart from Self-Guided DIP, shows nearly a 1dB improvement for MRI 8X acceleration compared to conventional methods. For the task of 30-views CT, aSeqDIP reports SSIM score of 0.92 which is 5% more than the second best, which is Self-Guided DIP with SSIM of 0.872. Although improvements against Self-Guided DIP are generally marginal in terms of reconstruction quality, our method proves to be 2X faster for MRI and CT reconstruction and requires 1 minute less than Self-Guided DIP for denoising and in-painting. This speed-up is attributed to updating the input using one forward pass of the trained network at each iteration k, instead of computing gradients with respect to the input for the update. Compared to Vanilla DIP, our method, on average, only requires an additional 30 to 60 seconds. When compared to ES-DIP [27] , our method requires longer time, but on average achieves better reconstruction results across three tasks and different settings.In comparison to data-dependent methods such as Score-MRI and MCG, our approach not only yields the best PSNR and SSIM but also requires reduced run-time, all without requiring any training data or pre-trained models. For instance, on average, aSeqDIP achieves nearly a 2dB improvement in 30-views CT compared to MCG while being 2X faster. In comparison to DPS, on average, our method report higher SSIM. Our method requires slightly less run-time on average but enhances the PSNR by approximately 0.6dB for both denoising and in-painting. Notably, our method is an optimization-based approach, whereas DM-based methods only require function evaluations. However, the generally larger run-time reported for DM-based methods is due to the necessity of running a large number of reverse sampling steps. When compared to Ref-Guided DIP, our method achieves higher PSNR and SSIM results without the need for any prior (or reference image) image.Figure 5 shows reconstructed images for the five considered tasks using aSeqDIP and the other baselines. Each row corresponds to a task. The first column displays the ground truth (GT) image whereas the second column shows the degraded image. Column 3 to the column before last present the reconstructed images by the baselines, while the last column shows the reconstructed images by aSeqDIP. PSNR values are provided at the bottom of each reconstructed image.As observed, aSeqDIP achieves the highest PSNR scores. Additionally, the top right green boxes, which show the difference between the central region of the reconstructed and GT images, indicate that for MRI and CT, our method visually exhibits the least difference, making it the closest to the GT.A similar observation is seen for the denoising task for the zoomed in bottom box. For inpainting, we note that aSeqDIP introduces the fewest unwanted artifacts as observed in the clouds (for DPS), and the left wing of the plane. While aSeqDIP contains artifacts for the task of Deblurring when compared to DPS, the latter generates a perceptually different image when compared to the GT. Similar observation are noticed with the additional visualizations provided in Appendix C.7.  
conclusions & future work: 
 In this paper, we introduced Autoencoding Sequential Deep Image Prior (aSeqDIP), a new unsupervised image recovery algorithm. Notably, aSeqDIP operates without the need of pre-trained models, relying solely on a sequential update of network parameters. These parameters are optimized using an input-adaptive data consistency objective combined with autoencoding regularization, effectively mitigating noise overfitting. Our experimental results across various tasks highlight the competitive performance of the proposed algorithm, matching (or outperforming) diffusion-based methods in terms of reconstruction quality and required run time, all without the need for pre-trained models.For future directions, we aim to explore the applicability of aSeqDIP to other image recovery problems, thereby expanding its versatility and potential impact across diverse domains. Additionally, we are interested in investigating the integration of a network input update mechanism to dynamically adjust the autoencoding regularization parameter and the number of gradient updates per iteration. 
 
 Appendix: In the Appendix, we first shed more light on the impact of the DIP network input by studying the training dynamics using the neural tangent kernel for CNNs with residual connections. Next, we show how trained autoencoders on clean images can be used as reconstructors at testing time. Lastly, we provide additional experimental results and visualizations.A Case Study: Impact of the DIP Network Input through lens of Neural Tangent Kernel in Residual NetworksWe show the impact of the DIP input through the lens of the Neural Tangent Kernel (NTK) [30, 44] for residual networks 6 . The NTK is a tool used to analyze the training dynamics of neural networks in the infinite width limit, where for CNNs the network width corresponds to the number of channels. In this limit, the change of any individual parameter during training becomes very small, which means that the change in the network's output during training can be accurately approximated by a first order Taylor expansion around its initialization. In the context of DIP, we consider training a neural network f with parameters θ and a fixed input z using gradient descent. At each training iteration, the network parameters are updated according to:EQUATIONwhere L is the loss function, and β is the learning rate. We also consider the resulting change in the network's output due to this parameter update using the first order Taylor expansion:EQUATIONSubstituting ( 7) into ( 8) and applying the chain rule to writeEQUATIONyields the equation:EQUATIONIn the infinite width limit, NTK theory states that the matrixΘ (t) := (∇ θ f θ (z) θ=θ (t) )(∇ θ f θ (z) θ=θ (t) )T stays fixed throughout training, so that Θ (t) = Θ (0) for all t. This matrix is called the neural tangent kernel, and we denote it as Θ. Moreover, because the parameters θ (0) are initialized randomly, in the infinite width limit, the NTK Θ becomes deterministic (as a function of z) due to the law of large numbers [30] , and does not depend on the specific instantiation of θ (0) . In DIP, the loss function is the least squares loss given in (1) . For simplicity, we consider the denoising case, where the forward operator A = I. Then, substituting the gradient of the loss into (10) shows explicitly how the output of deep image prior evolves during training:EQUATIONUsing this recursion relation, one can derive a closed form of the network output at iteration t in terms of the initial output and NTK [20, 30] . The reconstruction at iteration t is given by:EQUATIONIt is evident from (12) that the initial reconstruction of the network, f θ (0) (z), has important effects on the training dynamics of DIP. Furthermore, networks used in DIP often feature skip connections from earlier layers to later ones, and it is natural to believe that these connections may cause the input z to have a large effect on f θ (0) (z). In the following theorem, we analyze the training dynamics of CNNs with a very similar architectural modification: a residual connection that adds the input directly to the network output.Theorem A.1 (Dynamics of DIP with Residual Connections). Let g be a convolutional neural network with parameters θ. We consider the complementary residual network f defined by f θ (z) = z + g θ (z). Suppose that f is trained using gradient descent with the lossL(f θ (z)) = 1 2 ||f θ (z) -y|| 2 2 .Then, in the infinite width limit (number of channels), in expectation over the initialization of the parameters θ, we have that the output at training iteration t is given by:EQUATIONThe proof of Theorem A.1 is provided in Appendix A.1, along with a precise statement of the assumptions on the network architecture and parameter initialization. Additionally, in Appendix A.2 we provide a simple experiment to validate that the training dynamics given in (13) hold for real networks.Remark A.2. Theorem A.1 can be used to understand how the choice of network input affects the performance of DIP for image denoising. To gain intuition, we consider two special cases. First, we consider using the noisy image y as the input z. In this case, equation (13) simplifies to f θ (t) (y) = y for all iterations t. In this case absolutely no denoising occurs. On the other hand, we consider the oracle case where the clean image x is used as z. This gives usf θ (t) (x) = y -(I -βΘ) t (y -x).We see that at initialization (t = 0), we already expect perfect denoising, since f θ (0) (x) = y -(I -βΘ) 0 (y -x) = x. These two cases support intuition that using a network input closer to the true image could result in better performance in fewer training iterations.Setting of Theorem A.1. We first precisely state the conditions of Theorem A.1, in particular the network architectures considered and the corresponding parameter initializations. The present setting is very similar to the setting considered in [30] , but we provide the details here for completeness. We consider an L layer CNN with c in input channels and c out output channels, with c hidden channels in all intermediate layers. We assume that c in , c out << c. We assume all convolutions have a filter size of r. For simplicity, the network input and output are vectorized, so convolutions of any dimension are treated identically. For example, for a 2D CNN with 5 × 5 kernels, r = 25. Written explicitly, a network g with this architecture takes the formg θ (z) = C L (φ(C L-1 (φ(• • • φ(C 1 (z)))))), where the operators C i represent convolutions with an additive bias, and φ is a pointwise activation function such as ReLU. In this section, we also consider the residual network architecture defined by f θ (z) = z + g θ (z).We assume that the parameters are initialized using the He initialization [45] . With this initialization, the first layer convolutional filter weights are drawn from N (0, σ 2 w cinr ), and the filter weights for all other layers are drawn from N (0, σ 2 w cr ), where the variance σ 2 w depends on the non-linearity used in the network. For ReLU networks, σ 2 w = 2 [45] . All biases are initialized to 0.Proof. In the setting described above, the NTK emerges in the limit c → ∞. A body of existing theory [30, 44, 46] establishes that in this limit the NTK is a deterministic matrix as a function of the network input z. This theory does not consider residual connections, but applies immediately to both g and f . For g, the NTK is given byEQUATIONHowever, we can see that∇ θ g θ (z) θ=θ (0) = ∇ θ f θ (z) θ=θ (0) .Therefore, the linearization given in equation ( 10) holds for f using the same kernel Θ, and equation (12) describes the training dynamics of f .Using equation (12) , we can write:EQUATIONTo prove Theorem A.1, we consider the output f θ (t) (z) in expectation over the initialization θ (0) . Since all parameters θ (0) are drawn from mean 0 gaussian distributions, we find that E θ (0) [g θ (0) (z)] = 0 for any input z. Since Θ is deterministic in the limit c → ∞, in expectation over θ (0) equation ( 16) reduces to f θ (t) (z) = y -(I -βΘ) t (y -z), which proves Theorem A.1.We now provide a simple example using real networks to support the validity of Theorem A.1. This experiment substantiates both of the special cases considered in Remark A.2. Additionally, it shows that using the ground truth as the network input greatly inhibits overfitting. Indeed, in this experiment, we find that a residual network trained with the ground truth as input takes approximately 25 times more training iterations to completely learn the noisy signal than a residual network trained using a random noise input.We use DIP for denoising a 1D sinusoidal signal. We denote the clean signal x. The noisy signal is y = x + n, where n ∼ N (0, I). The network used is a five layer ReLU CNN with a residual connection. The full architecture can be written asf (z) = z + C 5 (ReLU(C 4 (• • • ReLU(C 2 (ReLU(C 1 (z))))))), where each C i represents a convolution (with bias).The signal has a size of 100, and the convolutions each have a filter size of 3, with 64 hidden channels. In all cases, the network is trained using gradient descent with a learning rate of 5 × 10 -4 . The same seed was used to initialize the network in all cases.We trained this network using three different inputs: the true signal x, the noisy signal y, and noise z ∼ N (0, I). The results of training the network using these three inputs are shown in Figure 6 . We find that the results for training this real, reasonably sized network show the behavior predicted by Theorem A.1, which was obtained in the infinite width limit. Indeed, equation (13) predicts that when x is used as the input, the initial error will be small, with eventual overfitting to the noisy signal. This is observed in Figure 6 , where the error is lowest at initialization, and it steadily increases throughout training. With this input, the network is highly resistant to overfitting, requiring approximately 10 5 training iterations to completely fit the noisy signal. We also see that when y is used as the input, the error curve obtained is essentially flat and converges quickly to the error of the noisy signal. This agrees with the expectation that the network output will be y for all iterations t when y is used as the input. Finally, when random noise z is used as the input, the typical DIP behavior emerges. Here, we investigate how the autoencoder term in aSeqDIP is improving the reconstruction quality while mitigating the impact of noise overfitting. In particular, we try to answer the question: Can an autoencoder trained on clean images operate as a reconstructor at testing time?To address this question, we perform the following steps: (i) train an autoencoder on fully sampled measurements or clean data images and (ii) utilize the trained autoencoder with unseen subsampled or corrupted measurements, optimizing over the input using the DIP objective with the autoencoder term. This enables the autoencoder to function as an image reconstructor. Specifically, given a training dataset, D, comprising unperturbed images or fully sampled MRI/CT data, denoted by x, we train an autoencoder U-Net g : R n → R n with parameters ψ. The training process seeks to obtain ψ as Subsequently, given unseen measurements y and the learned autoencoder's parameters ψ, we test the reconstruction of z ← argminEQUATIONEQUATIONwhere λ a ∈ R + is a regularization parameter. We perform this experiment by training ψ using 3000 fully sampled scans from the fastMRI dataset [47] . We then evaluate the reconstruction quality of the trained encoder using 8 scans from the fastMRI testing set. The average PSNR results for different values of λ a are depicted in Figure 7 . As observed, a trained autoencoder effectively serves as a reconstructor as evidenced by the achieved PSNR. Thus, we deduce that the autoencoder term in aSeqDIP not only mitigates noise overfitting but also enhances the reconstruction quality as an important prior. In this subsection, we illustrate aSeqDIP's robustness to the noise overfitting issue for the denoising task. The average PSNR for 20 images from the CBSD68 dataset for denoising using aSeqDIP and other DIP-based methods are given in Figure 8 . We observe two key points. First, in addition to higher PSNR, aSeqDIP shows higher robustness against noise overfitting compared to other DIP-based methods, consistent with MRI and CT results in Figure 4 . Second, unlike MRI and CT, the onset of noise overfitting occurs earlier, but the subsequent decay is very small.Here, we compare aSeqDIP with the End-to-End (E2E) MRI supervised model [48] that uses the variational network (VarNet). Results are given in Table 3 . As observed, we slightly under-perform when compared to VarNet (trained on 8000 data points) for the task of MRI reconstruction all without requiring any labeled training data. It is important to note that, at inference, E2E models only require a few unrolling steps, whereas aSeqDIP is an optimization method that requires to train the network parameters for each new set of measurements. Here, we present average PSNR results (averaged over 20 images) for the tasks of denoising , random inpainting (97% missing pixels), box-in-painting (with HIAR of 0.25), and non-linear deblurring of our method versus Denoising Diffusion Null-Space Model (DDNM) [49] and DPS [16] on the FFHQ testing dataset. For DPS and DDNM, we used a pre-trained model that was trained on the training set of FFHQ. As observed, our training-data-free method achieves competitive or slightly improved results when compared to data-intensive methods on all tasks other than box-inpainting (for which we under-perform by less than 1 dB), all without requiring a pre-trained model. In this section, we conduct an ablation study on the choice of the autoencoding regularization parameter, λ, in (3). Specifically, we conduct an experiment using 20 MRI scans to examine the impact of λ on the reconstruction quality and noise overfitting in aSeqDIP. We note that while the main results in Section 4 use N K = 4000, in these experiments, we run our algorithm for an extended number of iterations to investigate the onset of noise overfitting. We set K = 5000 and N = 2 for this purpose.In this experiment, we run aSeqDIP with values of λ ∈ {0.5, 1, 2}. Average PSNR results are given in Figure 9 . It is evident that, on average, using λ = 1 yields the most favorable results in terms of In practice, the entries of x * and y in MRI are complex numbers. However, to generalize the problem's definition, we use real numbers.https://github.com/microsoft/fastmri-plus/tree/mainhttps://www.aapm.org/grandchallenge/lowdosect/https://github.com/clausmichele/CBSD68-datasetWe note that skip and residual connections are not exactly the same as skip represents concatenation (typically from encoder to decoder end) and residual represents adding the input to the output. However, both operations correspond to sending initial input or features of a network to its latter portion or output. 


Decision: Authors present a novel DIP method that uses a sequence of DIPs akin
to diffusion models. The idea is backed up by theory and as far as I
can see, experimental results also support the idea.

Reviewers raised concerns regarding the experimental setup and as far
as I can see, authors addressed them as good as possible. Most
reviewers are happy with the rebuttal.

One reviewer raise concerns even after the rebuttal about comparisons
and experimental results. While these concerns have valid points, I
think the work is quite good. 

Given the novelty in the article, supported by 3 reviewers, supportive
experimental results and the fact that it attained 1BA, 1WA and 1A, I
think the paper would be a great contribution to the conference.

Reviews: review: paper_summary: paper_summary: This paper investigates how to prevent deep image prior (DIP) from overfitting to the noise or compressed measurements, which is a classic problem of DIP. To address the problem, the authors proposed Autoencoding Sequential DIP (aSeqDIP). The general idea of aSeqDIP is to cuts the overall training process into K sequential blocks, in each of which the input ($\mathbf{z_k}$) of the network is the previous output ($f(\mathbf{z_{k-1}})$), and the loss is defined as $\mathcal{L} = ||Af(\mathbf{z}_k) - y||_2^2 + || z_k - f(\mathbf{z}_k)||_2^2$, where the second term forces the output to be alike the input. As k increases, the network aims to find the network $f$ such that $z = f(z)$ and $||Af(\mathbf{z}_k) - y||_2^2$ is minimized. In a word, aSeqDIP forces the output to be a fixed point of the network that fits the measurements as much as possible.
strength_weakness: strength_weakness: 1. Although the experiment is thorough in terms of the variety of image reconstruction tasks, I still think it misses an important baseline that uses explicit regularizer (e.g. TV) to stablize the learning process. It would strengthen the argument if the authors could include a comparison like that.

2. The current introduction is concise, but it does not convey the message that the key benefit of aSeqDIP is to address the problem of overfitting.

3. The authors do not discuss the limitations of the proposed method.
questions: questions: 1. Please consider including a baseline like DIP+TV in the experiment.

2. What is the runtime of aSeqDIP? A comparison between aSeqDIP and other baselines is preferred.

3. [Subjective] The proposition seems redundant: 1) it assumes very strong conditions (the convergence of the network), which, in my opinion distracts the reader from the key message; 2) it kinda shows how aSeqDIP prevent itself from overfitting, but it is not very clear without text. I suggest to write the proposition in the form of plain text with equations.
limitations: limitations: Please discuss the limitations such as runtime, practical convergence, etc.
paper_summary: paper_summary: The authors in this paper propose an Autoencoding Sequential DIP (aSeqDIP) which aims to address the overfitting issue of DIP while without introducing extra parameters. The idea is very simple, the authors simply feed the output of the DIP into DIP model after each N updates. The authors validate the efficiency of their methods on several different image restoration tasks.
strength_weakness: strength_weakness: I have several concerns about this paper.

1) The novelty is significantly limited. It is almost the original DIP. The only difference is: the original DIP each iteration uses a random noise as its input; while here, the output of DIP is fed into the DIP for the next N iteration's updates.

2) The current experiments are all linear image restoration tasks. It would be interesting to see how this model works for non-linear image restoration tasks such as image delurring.

3) The authors should compare their method with more advanced DIPs such as Ref1.

4) How does this model compare with SOTA diffusion models Ref2?

5) The current results (Table2), the improvement is very small. And I am wondering what if we run the experiments many rounds and report the mean and std. 


Ref1: Jo, Yeonsik, Se Young Chun, and Jonghyun Choi. "Rethinking deep image prior for denoising." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

Ref2: Wang, Yinhuai, Jiwen Yu, and Jian Zhang. "Zero-shot image restoration using denoising diffusion null-space model." arXiv preprint arXiv:2212.00490 (2022).
questions: questions: I have several questions which have been listed in [Weaknesses].
limitations: limitations: I have listed the limitations in [Weaknesses].
paper_summary: paper_summary: This manuscript describes a variant of the seminar Deep Image Prior (DIP) work that incorporates aspects and mindset of the iterative prediction workflow in diffusion models.
In particular, the proposed procedure (method?) aSeqDIP is training a network to predict a single and fixed but distorted (e.g. noisy) image when given a pixel-wise random input image (just as DIP has initially proposed). The key novelty is to only train for a few training steps before switching to feeding the current prediction as an updated input (instead of the initial noisy input image). The similarity to predictions with diffusion models is apparent.
The approach is to the best of my knowledge novel and the idea, in my point of view, the biggest contribution of the paper.
strength_weakness: strength_weakness: * It surprises me that the manuscript does not argue more about WHY this approach leads to better results even when compared to data-dependent baselines. How strong are these baselines? Do better pre-trained methods exist? If so, why not show results with them as well?
   * An answer to the above “WHY?” question would go right to the heart of why diffusion works well and would therefore be very interesting.
   * Since the presented method/procedure is not dependent on any amount of available training data (but the single compromised target image) I wonder how this manuscript can avoid talking about inductive biases of the used network architecture (a UNet, hence, a CNN).
* The manuscript is overall rather compact in content, the text certainly not too dense but maybe even a bit too repetitive and wordy, and the figures not very legible (adequately sized). I also much regret not to see the compromised target image used by aSeqDIP, but only GT and final predictions.
* Figure 1: when seeing it first it really did not help me understand the paper any better. After reading the entire manuscript and coming back to Figure 1 I can confirm the figure makes sense, but one would hope that Figure 1 is more educational as it currently is.
* Figure captions are in general not terrible but also not in all cases making the figures self-contained. It is necessary to find some potentially distant place in the manuscript to fully grasp the visuals (e.g. Fig. 2).
questions: questions: * I cannot judge the quality or sufficiency of the used baseline methods. If for any of the 4 tasks the current SOTA (or industry standard) method would also be given and compared against, I would find that very useful.
* When does the proposed approach work (when is it applicable) and what are known limitations of its applicability? (When does the inductive bias of the setup not suit the desired task?)
limitations: limitations: * I would expect that the inductive biases that comes with the used network and training procedure dictate for what problems the presented method/procedure can produce good results. If the authors have any thoughts, I think the manuscript would much benefit from a short discussion in that vein.
paper_summary: paper_summary: The paper proposes Autoencoding Sequential Deep Image Prior (aSeqDIP) an extension to the family of untrained (no training data required) DIP methods. 
As in previous work a combination of data consistency loss and autoencoding regularization (a loss between network input and output) is used. However, instead of further regularizing the optimization by adding noise to the network input in every iteration as proposed in previous work, this paper varies the network input by setting it to the network output after every two iterations. 
Experimental evidence is provided that demonstrates superior robustness to overfitting and overall reconstruction accuracy compared to other DIP approaches and trained diffusion models.
strength_weakness: strength_weakness: **1. No results are provided for the most interesting case: Overfitting robustness for the task of denoising.**
To my understanding Prop. 3.1 says that in the limit the proposed approach converges to a network that has learned the identity and to a network output (which is identical to the network input) that perfectly fulfills data consistency under the forward operator and the given measurement. 
For the tasks of MRI and inpainting (in the absence of noise) solutions that fulfill Prop. 3.1 include the ground truth image and the empirical evidence provided in the paper shows that indeed the solution found by the proposed method is closer to the ground truth image than that of other DIP based approaches (why it is closer remains unclear up to intuitions/speculations, but answering this question is non-trivial and does not have to be the scope of the paper). 

Also, the robustness to overfitting seems reasonable for the aforementioned tasks. My intuition would be that repeatedly setting the network input to the output combined with the autoencoding regularization loss lets the network converge to identity relatively quickly. If at the same time data consistency is reached, there is no incentive for the network to change its weights anymore and a stable solution is reached. 

**However, things change in the presence of noise or in general for the task of denoising.** 
To my understanding, for denoising Prop. 3.1 is fulfilled, when the network perfectly reconstructs the noisy measurement $\mathbf{y}$, which is the very definition of overfitting. So, here it would be interesting to see curves as presented in Figure 4 for MRI and CT also for denoising to demonstrate to what extend overfitting is still prevented.
 
**2. In general, I am not sure about the usefulness of Prop. 3.1.** 
As discussed above for the tasks of MRI and inpainting reconstructions that fulfill Prop 3.1 comprise good and bad solutions however without making a statement why a good solution should be preferably reached over a bad solution, whereas for the task of denoising it implies convergence to a bad solution. 

**3. The learning based "SOTA" baselines perform very badly, which is not discussed in the paper and no information regarding the used training sets is provided.** 
Diffusion model-based methods are introduced as the state-of-the-art in the paper (line 113) and indeed the works referenced there report impressive results in terms of quantitative and qualitative reconstruction performance. 
Yet, in this work the reconstruction scores and especially the reconstructed images (Figure 5, 12, 13) contain severe artifacts. I understand that training and inference of those diffusion models is sensitive to hyperparameters, but then potential reasons for this bad performance should at least be discussed in the paper. Also, no information is provided regarding the training data used for those baseline models.

If there is no comparison to state-of-the-art end-to-end approaches (which definitely should give better results than DIP based reconstruction if trained properly on enough data) like the end-to-end VarNet (https://arxiv.org/abs/2004.06688) then at least the diffusion model based baseline should show reasonable results or if not it should be discussed why. 

**4. Code is only provided for the task of MRI reconstruction and the implemented method seems to differ from the description in the paper.** Checking the code provided via a link (https://anonymous.4open.science/r/Aseq_DIP-E728/README) in the paper I can only find a notebook that performs MRI reconstruction *Aseq_DIP.ipynb*. 
In the code the step where the network output is set to be the network input for the following steps is implemented as follows:
```
    pred_ksp = mps_and_gt_to_ksp(mps1.to(device), net_output)

    new_pred_ksp = (1 - mask_from_file).to(device) * pred_ksp.detach() / scale_factor + mask_from_file * ksp1

    new_ref = ksp_and_mps_to_gt(new_pred_ksp, mps1)
    random_smoothing_temp = torch.zeros_like(new_ref).to(device)

    for jj in range(1):
        random_smoothing = new_ref + 1e-5 * torch.rand((640, 372), dtype=torch.complex64).to(device)
        random_smoothing_temp += random_smoothing

    random_smoothing_final = random_smoothing_temp 
    ref[:, 0, :, :] = random_smoothing_final.real
    ref[:, 1, :, :] = random_smoothing_final.imag
```
If I understand it correctly the new input (ref) is first set to fulfill perfect data consistency and then processed with random smoothing, two things not mentioned anywhere in the paper. Also, the new input (ref) is updated before the computation of the loss, which means that in fact the autoencoding regularizer is not really computed between the network input and the network applied to this same input.
It is unclear how essential these two steps (hard data consistency and smoothing) are to the proposed approach.

Further, 
```
    loss.backward()
    for i in range(2):
        optimizer.step()
```
implies that the network weights are updated twice without updating the loss functions, which is not what is outlined in Algorithm 1 of the paper.

If that is really how the mehtod is implemented that corresponds to the results in the paper the ablation study regarding the values for the number of iterations $N$ per network in Appendix C.4 seems meaningless as the loss is not updated as $N$ increases.
questions: questions: 1. In the Figures (like Fig. 5) showing qualitative reconstruction results it would be helpful to show the measurement, i.e. noise image for denoising, mask fr in-painting and zero-filled reconstruction for MRI.

2. I personally find the formulation in line 10 "sequential optimization of multiple network architectures" a bit missleading as the networks are not changed or re-initialized. Only the network input is adapted. 

3. Regarding Section 3.1 and Figure 2 it makes sense to me why the blue curve is decaying, but do you know why the red curve is decaying? Does the red curve decaying imply that a constant input (like all zeros) to the DIP would work best?
limitations: limitations: Limitations are not discussed in the paper. This could be for example the limitted interpretability of the comparison to trained baseline methods as current results in the paper significantly fall behind the results reported in the baseline papers or the long reconstruction times compared to end-to-end deep learning based methods.

Comments: title: Thank you for your response
A friendly and gentle reminder
A friendly and gentle reminder
Thank you for your response and raising your score
Thank you for your response. Addressing the Reviewer's Concern
Thanks for the clarification. Final summary of revision.
Thank you for the response
Thank you for your response. Response to the VarNet experiment
Some more details regarding the VarNet experiment?
Thank you for your response. Addressing the remaining concerns with baselines results
Rebuttal is online - please respond
Reponse to author rebuttal - remaning concerns with baseline results
comment: We're glad the reviewer is satisfied with our testing of aSeqDIP on a non-linear task. Below, we address the remaining concerns.

### 1) The image to image mapping in DIP is not new, [ref1: DeepRED] also used the image as its input instead of random seed. 

In our rebuttal, we outlined the key differences between our work and the original DIP paper. We agree with the reviewer that we are not the first to consider a DIP network input containing some structure of the ground truth, **as discussed in line 88 of our paper, where we cite two other works**. Below, we discuss how aSeqDIP differs from [Ref1].

We agree that DeepRED initializes the algorithm with the noised image $x_0 = y$. However, in DeepRED, the DIP network input is still random noise that remains fixed. This is stated in Section 4 as ***"In all the reported tests the same network as in [Original DIP] is used with an i.i.d. uniform (∼[0, 0.1]) random input tensor of size 32×W×H, where W×H is the size of the output image to synthesize"***. This input is $z$ in their algorithm which remains unchanged as given in Eq.(7), (11), and (12). 

In our case, the DIP network input is the image we are iteratively estimating, whereas for DeepRED, its a random tensor unrelated to the reconstruction.

Additionally, we see three major differences between our work and DeepRED:

- **Variables**: Due to their adoption of the ADMM algorithm, DeepRED requires updating three variables: The network parameters, variable $x$, and the Lagrange multipliers vector. In aSeqDIP, we only update the parameters and the input (updated by a single pass of the network after every few parameters update).

-  **External Denoiser**: Due to the use of RED (The Little Engine that Could: Regularization by Denoising), in addition to the DIP network, $\textrm{T}_\Theta$, DeepRED requires an external denoiser $f$. $f$ is used for updating the network input (Eq.(11) or (12)). Table 1 in DeepRED shows the external denoisers (NLM and BM3D) used in their experiments. **In aSeqDIP, no external denoisers is needed**. 

- **Applicability**: We considered a diverse array of tasks including two medical image reconstruction tasks (MRI and CT) and three natural image restoration tasks, whereas DeepRED considered three image restoration tasks. 

We will include this discussion in the revised manuscript. 



### 2) I do agree that DIP-based model suffers from the overfitting issues, others paper addressed this issue. [ref2]: Early Stopping for Deep Image Prior, and [ref3]: Self-Validation: Early Stopping for Single-Instance Deep Generative Priors

We would like to thank the reviewer for sharing these papers. As noise over-fitting is the major drawback of DIP-based methods, we agree that there are several works that address the noise overfitting issue including the ones we discuss (**and include as baselines**) such as (**TV-DIP, Ref-DIP, SGLD-DIP, & Self-Guided DIP**). 

**Due to time constraints, we are unable to compare with [ref2] and [ref3]. However, we will include discussions about [ref2,ref3] in the revised related work section as below**. 

In [ref2], the authors categorized DIP methods based on addressing noise over-fitting into: (i) Regularization, (ii) Noise modeling, and (iii) Early stopping (ES). The methods in [ref2] and [ref3] are both ES approaches, whereas aSeqDIP belongs to the "Regularization" category due to the use of the input-adaptive auto-encoding term. In [ref2], the authors use the running variance in Eq. (3) as the criteria for ES, whereas the authors of [ref3] propose combining self-validation and training to apply ES. 

Most importantly, we believe that aSeqDIP achieves high robustness to noise over-fitting. In Figure 10 ($\lambda = 1$), on average, noise over-fitting does not start until iteration 8000 with a subsequent very minimal decay for the task of MRI. The minimal decay in noise over-fitting is also observed in the PSNR curves for denoising in the attached PDF. In these experiments, we compared to SGLD-DIP, Self-guided DIP, and Vanilla DIP. 



### 3) Comparison with DIP-based deblurring models such as [ref4]: Neural Blind Deconvolution Using Deep Priors

In our rebuttal to the reviewer's comment (see **C2 It would be...**), we compared with two DIP-based methods (self-guided DIP and SGLD-DIP) for the task of non-uniform image deblurring. 

In an attempt to fully address the reviewer's comment, we ran the code of [ref4] using 20 FFHQ images, and report the results below. 


| Method | Forward Operator | Non-uniform Deblurring |
|----------|----------|----------|
DDNM (using FFHQ-trained DM)| Known |23.88 |
DPS (using FFHQ-trained DM)|Known | 23.67 |
SelfDeblur [ref4]|Unknown |  22.35 |
aSeqDIP (Ours)| Known |  24.02 |

While we achieve better results, we emphasize that [ref4] operates in a blind setting without access to the forward operator.

**We hope that our responses have addressed the reviewer's concerns and kindly ask if they could reconsider their score.**
Thanks to the authors for the detailed rebuttal. And it is glad to know that the authors added the experiments on nonlinear tasks. However, to 1) the image to image mapping in DIP is not new at all, this paper [ref1] also used the image as its input instead of random seed; 2) I do agree that DIP-based model suffers from the overfitting issues, however, there are several papers have addressed this issue, see ref2, ref3. 3) for the nonlinear tasks, the authors may want to compare with DIP-based deblurring models such as ref4.


Ref1: https://openaccess.thecvf.com/content_ICCVW_2019/papers/LCI/Mataev_DeepRED_Deep_Image_Prior_Powered_by_RED_ICCVW_2019_paper.pdf

Ref2: https://arxiv.org/abs/2112.06074

Ref3:https://arxiv.org/abs/2110.12271 

Ref4: https://openaccess.thecvf.com/content_CVPR_2020/html/Ren_Neural_Blind_Deconvolution_Using_Deep_Priors_CVPR_2020_paper.html
We would like to express our sincere gratitude to the reviewer once again for their insightful comments.

As the open discussion period is drawing to a close, we would be deeply grateful if the reviewer could kindly respond to our rebuttal. This would provide us with the opportunity to address or clarify any remaining concerns thoroughly.
We would like to sincerely thank the reviewer once again for their insightful comments.

As the open discussion period is drawing to a close, we would be deeply grateful if the reviewer could kindly respond to our rebuttal. This would provide us with the opportunity to address or clarify any remaining concerns thoroughly.

We would also like to respectfully highlight that other reviewers have acknowledged the novelty and strengths of aSeqDIP, with remarks such as "The idea is fantastic and thought-provoking", "The proposed approach for enforcing DIP to find its fixed point is interesting and novel", "The introduction of the regularization term (autoencoding term) during iterative training and showing that it is useful and that tuning it is important", and "Robustness towards overfitting demonstrating the benefits of the proposed approach".
We would like to thank the reviewer for their response and raising their score. We are glad that the reviewer is satisfied with our rebuttal and the additional experiments. 

Following the reviewer's recommendation, we will add the additional experiments and more details about the baselines configurations in the revised manuscript.
First, we would like to thank the reviewer for their response.

For TV-DIP, during the rebuttal, we used "https://arxiv.org/pdf/1810.12864" following the reviewer's suggestion. We coded the TV regularization term in Equation (3) and used Equation (5) to tune the parameters $\Theta$. According to Section 4.1 of the TV-DIP paper, we used 5000 optimization steps. For the architecture, we used the U-Net architecture (with skip connections) from the original DIP paper, which is the same as the one the authors describe in Figure 3.

In regard to the hyper-parameters (including the regularization parameter), in Section 4.1, the authors of TV-DIP mentions: "All algorithmic hyperparameters were optimized in each experiment for the best signal-to-noise ratio (SNR) performance with respect to the ground truth test image". 

As the authors do not provide the regularization parameter, in the results we provided in the rebuttal, we used $\lambda = 1$, which is similar to aSeqDIP. 

To fully address the reviewer's concern and ensure that the choice of $\lambda$ we used (in our rebuttal) in TV-DIP is sufficient, we run TV-DIP with additional values of the regularization parameter for the task of denoising and report the average PSNR results for 20 images from the CBSD68 dataset. As observed, the TV-DIP results with $\lambda$ near 1 is better than other values we considered here. In all cases, aSeqDIP achieves better PSNR.

| Task | TV-DIP ($\lambda = 0.1$) | TV-DIP ($\lambda = 0.8$) | TV-DIP ($\lambda = 1$)| TV-DIP ($\lambda = 1.2$) |  TV-DIP ($\lambda = 3$) |TV-DIP ($\lambda = 10$) | aSeqDIP (Ours) | 
|----------|----------|----------|----------|----------|----------|----------|----------|
| Denoising | 30.02 | 30.54 | 30.57 | 30.61 | 30.43 |28.89 | 31.51 |

Following the reviewer's suggestion, we will add this discussion and the TV-DIP results in the revised paper. We are happy to address any other concerns the reviewer might have.

*We hope that, in light of our responses, the reviewer might consider raising their score.*


Thanks,

Authors
Thanks for the clarification. I trust that in the revised version all the additional experimental settings and baseline configurations are explained in detail at least somewhere in the Appendix.

Overall I am very happy with the rebuttal and the additional experiments and clarifications provided by the authors.
In particular, the experiment on robustness towards overfitting in the denoising case demonstrates the benfits of the proposed approach, and the experiments with the end-to-end VarNet help to put the results into the right perspective. 

I raised my score accordinly.
I thank the authors for their response, and they have addressed most of my concerns. One only question remaining is how the DIP+TV is finetuned? I suggest the authors to include a description on this in the camera-ready.
We would like to thank you again for your response. We are glad that you found the results of aSeqDIP impressive. We hope that our responses can further convince you to raise the score.

We double checked and the reviewer is correct that the full training/validation set is larger. We would like to clarify the settings we used for the rebuttal. We used a subset of data and removed peripheral slices in each volume (around 10 per volume) during training. We followed similar setup as recent works "Blind Primed Supervised (BLIPS) Learning for MR Image Reconstruction, TMI 2021" (Fig. 3) that showed supervised model results with varying training sizes from ~1K to ~8K. We believe its promising that a data-free approach can compete with supervised networks trained with many knee slices.
Thank you for providing the additional information regarding the DM baseline experiments. I guess Score-MRI looses more performance through this shift from training on magnitude images only to multi-coil evaluation than I expceted. Still impressive that your aSeqDIP can perform on the same or better level.


Regarding the VarNet experiment, I have a last question. 
If I understand it correctly, you consider the problem of multi-coil knee slice reconstruction. 
Do you focus on a certain subset of slices or which data do you mean when you say 3000 out of the 8000 available fastMRI multi-coil datapoints?
The fastMRI knee dataset contains alsmot 35k slices, see https://arxiv.org/pdf/1811.08839 Table 4.
We would like to thank the reviewer for their prompt response to our rebuttal. We hope that the following will address the reviewer's remaining concerns with baselines. 

We would like to emphasize that the main point of comparing our method with data-intensive approaches (VarNet, Score-MRI, DPS, DDNM, and MCG) is to demonstrate that we achieve competitive results, all without the need for training data or pre-trained models by appropriately setting up the optimization and regularization of deep image prior. In what follows, we address your concerns in three parts.

**If Score-MRI is so unstable that when trained on the entire fastMRI brain training set is still outperformed by a data-free method on the fastMRI brain validation set, then maybe it is not a good DM baseline.** 

Score-MRI DM was originally trained on natural images and then fine-tuned on the entire training set of fastMRI using the single-coil setting. The reason for this pre-training + fine-tuning is that DMs require a significant amount of training data to enter the generalization regime (see Figure 2b in "The Emergence of Reproducibility and Consistency in Diffusion Models"), which may not be available for tasks such as MRI and CT. During testing, they used the multicoil real setting DM on both the real and imaginary parts. In their paper, they mention: "Our model requires magnitude images only for training, and yet is able to reconstruct complex-valued data, and even extends to parallel imaging."

To the best of our knowledge, Score-MRI and CCDF (which we came across after submitting the paper) are the two best DM-based MRI baselines. In CCDF, they demonstrated slightly improved PSNR scores while requiring fewer sampling steps, making it faster. Refer to Table 5 in the CCDF paper, "Come-Closer-Diffuse-Faster: Accelerating Conditional Diffusion Models for Inverse Problems through Stochastic Contraction." In that paper, Score-MRI is referred to as "Score-POCS." For the 4x Gaussian 1D sampling, CCDF reports 32.51 dB, whereas Score-MRI achieved 31.45 dB.

**The type of undersampling mask should not make up for the difference in PSNR as Uniform 1D and Gaussian 1D are also Cartesian masks.**

Thank you for your comment. Note that in the Score-MRI paper (multicoil setting in Table 2), the PSNR values vary by nearly 4 dB depending on the mask, ranging from 29.17 dB (Gaussian 2D) to 34.25 dB (Gaussian 1D).

We acknowledge the reviewer's comment that the masks they used are indeed Cartesian. Thank you for the correction. We needed to review the details of their code implementation to confirm this.

The first column of Figure 4 (and its caption) in their paper describes the exact masks they used. Our sampling mask is the 1D Poisson Disk, which the Score-MRI authors did not use. The Poisson Disk mask in their results is 2D Poisson Disk. 

To fully address the reviewer's concern, we ran aSeqDIP with the 1D Uniform setting, the sampling mask used in the first row of Table 2 in Score-MRI. The results are averaged over 20 MRI knee scans (with 4x undersampling). As observed, we achieve competitive PSNR results with Score-MRI in this setting as well.

| Task | Score-MRI (reported from their paper) | Score-MRI (us running their code)| aSeqDIP (Ours) | 
|----------|----------|----------|----------|
| MRI (1D Uniform Sampling) | 33.25 | 33.45 | 34.05 |


**Further, I find it extremely strange that the VarNet is outperformed by a data-free method as the VarNet provides stable SOTA results (see fastMRI challenge) if trained and tested on the same type of data. So what was the training and testing setup for this experiment with the VarNet?**

The reviewer is correct; VarNet indeed achieves very competitive results. Since VarNet does not provide a pre-trained model, we initially trained their architecture from scratch using 3,000 fastMRI multicoil datapoints and tested it with the fastMRI testing dataset. Please note that due to time constraints and the additional experiments conducted during the rebuttal, in our initial response, we used 3,000 knee images for training instead of the available 8,000 datapoints. 

To fully address the reviewer's concern, we trained VarNet with the full training set and their PSNR results do indeed improve when compared to the results of VarNet trained with only 3K points. See the following table. Our method is still quite competitive and could prove beneficial in limited training data regimes. 

| Task |	VarNet (trained with 8k points)	| VarNet (trained with 3k points) | aSeqDIP (Ours) |
|----------|----------|----------|----------|
|MRI	| 34.89 | 33.78	 | 34.08 |
 

Note that Score-MRI paper also reported VarNet results (7th column of Table 2), and the results slightly varied when different masks were used.

*We are happy to address any more concerns*.

Thanks,

Authors
Dear Reviewers, 

Authors carefully prepared their rebuttal - trying to address the concerns you have raised. Please check the rebuttals and join the discussion about the paper. 

Regards,
I thank the authors for the clarifications and additional experiments and for answerning my qustions.

My remaining concerns are with the baseline results. If Score-MRI is so unstabel that when trained on the entire fastMRI brain training set is still outperformed by a data-free method on the fastMRI brain validation set, then maybe it is not a good DM baseline. Also the type of undersampling mask should not make up for the difference in PSNR as Uniform 1D and Gaussian 1D are also Cartesian masks. 

Further, I find it extremely strange that the VarNet is outperformed by a data-free method as the VarNet provides stable SOTA results (see fastMRI challenge) if trained and tested on the same type of data.
So what was the training and testing setup for this experiment with the VarNet?