Title: Image Reconstruction Via Autoencoding Sequential Deep Image Prior
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Reconstruction

Deep Image Prior

Generative Models
Abstract: Recently, Deep Image Prior (DIP) has emerged as an effective unsupervised one-shot learner, delivering competitive results across various image recovery problems. This method only requires the noisy measurements and a forward operator, relying solely on deep networks initialized with random noise to learn and restore the structure of the data. However, DIP is notorious for its vulnerability to overfitting due to the overparameterization of the network. Building upon insights into the impact of the DIP input and drawing inspiration from the gradual denoising process in cutting-edge diffusion models, we introduce Autoencoding Sequential DIP (aSeqDIP) for image reconstruction. This method progressively denoises and reconstructs the image through a sequential optimization of network weights. This is achieved using an input-adaptive DIP objective, combined with an autoencoding regularization term. Compared to diffusion models, our method does not require training data and outperforms other DIP-based methods in mitigating noise overfitting while maintaining a similar number of parameter updates as Vanilla DIP. Through extensive experiments, we validate the effectiveness of our method in various image reconstruction tasks, such as MRI and CT reconstruction, as well as in image restoration tasks like image denoising, inpainting, and non-linear deblurring.

Image Reconstruction Via Autoencoding Sequential Deep Image Prior

Abstract

Recently, Deep Image Prior (DIP) has emerged as an effective unsupervised oneshot learner, delivering competitive results across various image recovery problems. This method only requires the noisy measurements and a forward operator, relying solely on deep networks initialized with random noise to learn and restore the structure of the data. However, DIP is notorious for its vulnerability to overfitting due to the overparameterization of the network. Building upon insights into the impact of the DIP input and drawing inspiration from the gradual denoising process in cutting-edge diffusion models, we introduce Autoencoding Sequential DIP (aSeqDIP) for image reconstruction. This method progressively denoises and reconstructs the image through a sequential optimization of network weights. This is achieved using an input-adaptive DIP objective, combined with an autoencoding regularization term. Compared to diffusion models, our method does not require training data and outperforms other DIP-based methods in mitigating noise overfitting while maintaining a similar number of parameter updates as Vanilla DIP. Through extensive experiments, we validate the effectiveness of our method in various image reconstruction tasks, such as MRI and CT reconstruction, as well as in image restoration tasks like image denoising, inpainting, and non-linear deblurring. Our code is available at the GitHub repository aSeqDIP.

1 Introduction

Inverse imaging problems arise across various real-world applications [1]. These include tasks such as image denoising [2], image deblurring [3], restoring missing portions of images [4] (in-painting), super-resolution [5], magnetic resonance imaging (MRI) [6], and reconstructing X-ray computed tomography (CT) scans [7]. The common challenge in these scenarios is the reconstruction of images from limited and/or corrupted measurements.

Recently, a plethora of Deep Neural Network (DNN) techniques have emerged to solve inverse imaging problems, including supervised models that are trained to process measurements and generate estimates approximating the true image [8, 9, 10, 11, 12]. Generative model, such as Diffusion models (DMs) [13], have shown state-of-the-art (SOTA) performance in various imaging inverse problems such as the works in [14, 15, 16]. However, training diffusion models is not only computationally expensive but also demands a significant amount of clean (or fully-sampled) data points [17], limiting

their application to data-limited or data-corrupted tasks such as in medical imaging (MRI and CT). To address this challenge, previous DM-based MRI and CT methods typically fine-tune pre-trained DMs (trained on natural images) using limited fully-sampled data points, which are not as readily available as natural images [14, 15]. As such, there is a pressing need to develop methods that can alleviate the reliance on large, clean datasets and/or pre-trained models.

One well-known method is Deep Image Prior (DIP) [18], which does not require pre-trained models and relies on the parameters of an architecture (e.g., U-Net [19]). DIP finds a solution without needing any data other than the limited/corrupted task-specific measurements and the forward operator. DIP has shown competitive reconstruction/restoration results in inverse imaging problems such as MRI [20] and image denoising [18]. However, DIP (and most of its variants) suffer from the problem of overfitting.

In this paper, we introduce Autoencoding Sequential DIP (aSeqDIP) to mitigate noise overfitting artifacts. Our approach comprises a sequential update of network weights. A significant advantage of our approach, in contrast to DM-based methods, is its ability to operate without the need for training data or pre-trained models, thus fitting into the framework of dataless training [21]. Notably, the total number of parameter updates in aSeqDIP remains equivalent to that of Vanilla DIP [18]. Furthermore, unlike a recent work [20], aSeqDIP eliminates the need for gradient-based input updates.

Contributions:

Building upon an insight about the impact of the DIP network input, we introduce Autoencoding Sequential DIP (aSeqDIP), which incorporates a U-Net architecture whose weights are updated sequentially. These updates are based on objective functions that consist of ( i ) an input-adaptive data consistency term and ( ii ) an autoencoding regularization term used for noise overfitting mitigation.

Our extensive experimental evaluations, in terms of standard image reconstruction metrics and required run-time, highlight the superior (or competitive) performance of aSeqDIP compared to DIP-based and leading DM-based methods for the tasks of MRI and CT reconstruction, denoising, in-painting, and non-linear deblurring.

Organization: In Section 2, we provide an overview of the considered tasks, and review recent advancements in both DIP-based and DM-based methods. Section 3 introduces our approach, aSeqDIP. Our experimental results are presented in Section 4, followed by discussions on conclusions and future directions in Section 5.

2 Preliminaries & Related Work

2.1 The Image Reconstruction Problem

Image reconstruction tasks can be defined as recovering an image x ∗ ∈ R n from measurements y ∈ R m , where m ≤ n , governed by the forward operator A . For multi-coil MRI, the forward operator is A = MFS , where M denotes coil-wise undersampling, F is the coil-by-coil Fourier transform, and S represents the sensitivity encoding with multiple coils 2 . For CT, a simplified forward operator will be used to study the sparse-views setting: A = CR , where C is an undersampling operator that selects specific projection views or angles, and R denotes the radon transform [22] (corresponding to parallel-beam geometry in CT). For both MRI and CT with limited acquired data (done to accelerate scan or reduce X-ray dose, etc.), the condition m<n applies.

For the denoising task, the forward operator is the identity matrix, whereas for in-painting, it is equivalent to applying a binary mask element-wise that masks corrupted or missing pixels. For deblurring, the non-linear forward operator is the neural network approximated kernel in [3]. Next, we review prior arts within the context of DIP-based and DM-based methods.

2.2 Related Work

DIP-based Methods: Deep Image Prior (DIP) was first introduced by [18]. The authors demonstrated that the architecture of a generator network alone is capable of capturing a significant amount

of low-level image statistics even before any learning takes place. Specifically, the DIP image reconstruction is obtained through the minimization of the following objective:

where ˆ x is the reconstructed image, and θ corresponds to the parameters of network f : R n → R n , which is typically implemented using a U-Net architecture [19]. The input to the network, z ∈ R n , is randomly chosen and remains fixed throughout the optimization process. While standard DIP was shown to perform well in many tasks, selecting the number of iterations to optimize objective (1) poses a challenge as the network would eventually fit the noise present in y or could fit to undesired images based on the null space of A .

To mitigate the problem of noise overfitting, previous studies considered different approaches such as regularization, early stopping (ES), and network pruning [23]. For regularization-based methods, the work in [24] enhanced the standard DIP by introducing a total variation (TV) regularization term for denoising and deblurring tasks, whereas the study in [25] proposed combining DIP with stochastic gradient Langevin dynamics (SGLD) [26]. The authors in [27] use running variance as the criterion for ES, whereas the authors of [28] propose combining self-validation and training to apply ES.

The input to the standard DIP (or Vanilla DIP) network is a random noise vector that, in most works, remains fixed during the optimization. Nevertheless, other works, such as those in [29] and [30], have explored cases where the input contains some structure of the ground truth. The approach employed in reference-guided DIP (Ref-Guided DIP) [29] follows the same objective as standard DIP in (1). However, instead of using a fixed random noise vector as input, it utilizes a reference image closely resembling the one undergoing reconstruction. This method was applied to the task of MRI. This methodology proves particularly effective when datasets comprising structurally similar data points are available. The reference required here makes this method a data-dependent approach.

Inspired by the departure from using a random fixed input, the authors in [20] recently introduced Self-Guided DIP. Unlike Ref-Guided DIP [29], a prior image that closely resembles the unknown (to be estimated) image is not needed, and the optimization occurs simultaneously with respect to both the input and the parameters of the network. Specifically, Self-Guided DIP employs the following objective:

where η is random noise, and α is a regularization parameter. The first (resp. second) term is used for data consistency (resp. denoising regularization) and final reconstruction is obtained as ˆ x = E η [ f ˆ θ (ˆ z + η )] . aSeqDIP is different from Self-Guided DIP as our method does not require gradient-based updates for the input, making it computationally less expensive. Self-Guided DIP has demonstrated superior performance compared to Vanilla DIP, TV-DIP, and SGLD-DIP, thus serving as a primary baseline for comparison.

DM-based Methods: In recent years, there has been an abundance of DM-based methods proposed to address inverse imaging problems [14, 15, 16, 31, 32, 33]. A well-known method for natural images is Diffusion Posterior sampling (DPS) [16]. DPS incorporates a gradient step into the reverse sampling process of pre-trained DMs, ensuring data consistency and enabling sampling from the conditional distribution. In the context of image reconstruction and restoration tasks, numerous diffusion-based approaches have emerged, as evidenced by works such as [34, 35, 36]. Notably, the authors in [14] and [15] introduced a SOTA DM-based approach for addressing the MRI and CT reconstruction inverse problems, respectively. They propose incorporating the predictor-corrector sampling algorithm [37] for data consistency, akin to DPS, thereby facilitating the sampling from a conditional distribution.

One clear distinction between aSeqDIP and DM-based methods is that our approach does not necessitate pre-trained models. For our experiments in MRI, CT, and denoising (as well as in-painting and deblurring) tasks, we will utilize Score-MRI [14], Manifold Constrained Gradient (MCG) [15], and DPS [16], respectively, as DM-based baselines.

3 Autoencoding Sequential DIP

In this section, we begin by investigating the impact of the input on DIP. Then, we introduce our method, aSeqDIP. We note that while we consider linear and non-linear inverse problems, in our formulations, we use a linear forward model to simplify notation.

3.1 Motivation of aSeqDIP: The Impact of the Network Input in Vanilla DIP

Here, we aim to address the question: How does employing a noisy version of the ground truth image, which retains some structure of the ground truth, as the fixed input to the Vanilla DIP objective in (1) , affect performance? To investigate, we conduct the following experiment.

Consider the MRI task defined as y ≈ Ax ∗ . Let the input to the standard DIP objective in (1) be denoted as z = x ∗ + δ , where δ ∼ N ( 0 , σ 2 I ) . Here, σ controls the magnitude of the perturbations added to the ground truth image, indicating that a larger σ results in a greater deviation between z and x ∗ . We optimize (1) for various values of σ , recording the best possible PSNR compared to the ground truth, i.e., prior to the start of the noise overfitting decay.

Figure 1 displays the average results for 8 images. Notably, for all images, a closer similarity of the DIP network input to x ∗ , as indicated by σ , corresponds to higher reconstruction quality, measured by PSNR. Larger variance in the standard Gaussian distribution corresponds to larger additive perturbations even for the case of x ∗ = 0 (the red curve). We conjecture that this still leads to larger distances from the ground truth and hence worse performance.

Based on this discussion, a notable insight emerges:

The proximity of the DIP network input to the ground truth correlates with the quality of the reconstruction. This promotes the question: Can we develop an input-adaptive DIP method that mitigates noise overfitting?

We proceed to address this question by proposing our method, which we refer to as Autoencoding Sequential DIP (aSeqDIP). In Appendix A, we provide a case study and theory on the impact of the DIP network input through the lens of the Neural Tangent Kernel in residual networks. The onset of severe noise overfitting therein is delayed for better inputs (Appendix A.2).

3.2 The Proposed aSeqDIP Algorithm

0.2

0.5

0.8

1.1

1.4

1.7

2.0

: Input Noise Level

30

31

32

33

Best Possible PSNR (dB)

DIP with input

z

=

x

* +

DIP with input

z

=

Consider that we have a U-Net architecture defined by f : R n → R n whose weights are given by ϕ k , where k ∈ [ K ] , and [ K ] := { 1 , . . . , K } . Each set of parameters in f ϕ k takes an input z k and outputs f ϕ k ( z k ) . Based on the insight from the previous subsection, we initially set z 0 to y (resp. A H y ) for denoising, in-painting, and deblurring (resp. MRI and CT). The initialization of ϕ 1 follows the same initialization as any other DIP-based method. The parameters in f ϕ k , and the input, z k , are then updated sequentially through

where λ ∈ R + is a regularization parameter, and the initialization of ϕ k is the optimized ϕ k -1 in (3). The final reconstruction is given as:

The proposed procedure outlined in (3) and (4) consists of two key components. First, the optimization of each set of weights in f ϕ k using an objective that consists of the data consistency term and the

Algorithm 1 A utoencoding Seq uential D eep I mage P rior ( aSeqDIP ).

Input : Measurements y , forward operator A , number of input updates K , number of gradient updates N per input update, regularization parameter λ , and learning rate β .

Output : Reconstructed image ˆ x .

Initialization : z 0 = A H y ; ϕ 0 ∼ N ( 0 , I ) .

1: For each k ∈ [ K ]

2: Initialize ϕ (0) k ← ϕ ( N ) k -1 for k ∈ { 2 , . . . , K } , and ϕ (0) k ← ϕ (0) for k = 1 .

3: For each i ∈ [ N ] . (Network parameters update)

4: ϕ ( i ) k = ϕ ( i -1) k -β ∇ ϕ k [ ∥ A f ϕ k ( z k -1 ) -y ∥ 2 2 + λ ∥ f ϕ k ( z k -1 ) -z k -1 ∥ 2 2 ]∣ ∣ ∣ ϕ k = ϕ ( i -1) k .

5: Obtain z k := f ϕ ( N ) k ( z k -1 ) . (Network input update)

6: Reconstructed image: ˆ x = z K = f ϕ ( N ) K ( z K -1 )

𝐳0

= 𝐀 𝐻 𝐲

𝐳1

= 𝑓 𝜙1

(𝐳0)

𝐳2

= 𝑓 𝜙2

(𝐳1 )

𝐳𝐾-1

= 𝑓 𝜙𝐾-1

(𝐳𝐾-2)

𝐳𝐾

= 𝑓 𝜙𝐾

(𝐳𝐾-1)

𝑓 𝜙 1

(⋅)

Initialize

Optimize for

𝑁

steps

𝑓 𝜙 2

(⋅)

Initialize

Optimize for

𝑁

steps

𝜙1

𝑓 𝜙 𝐾

(⋅)

Initialize

Optimize for

𝑁

steps

randomly

second autoencoding term that aims to alleviate noise overfitting. Second, the update of the input, z k , after optimizing each set of weights f ϕ k , so that our method is input-adaptive .

Algorithm 1 presents the procedure of our proposed approach. As inputs, the algorithm takes y , A , K , N , λ , and the learning rate β . Apart from the measurements and the forward operator, the remaining parameters are considered hyper-parameters, typical in most DIP-based methods. The parameters in f ϕ k are set to ϕ k -1 (step 2) and subsequently optimized for N iterations using a gradient-based optimizer, such as gradient descent (as depicted in Algorithm 1) or Adam [38]. A block diagram of our proposed aSeqDIP method is presented in Figure 2.

In the following remarks, we provide insights into our proposed aSeqDIP method.

Remark 3.1 (Differences from Vanilla DIP [18]) . Assume that the iterates of (3) and (4) converge, i.e., as k → ∞ , z k → z ∗ and ϕ k → ϕ ∗ . Then, according to (4), for a continuous mapping f , we have z ∗ = f ϕ ∗ ( z ∗ ) . Substituting this into (3) in the limit, we get ϕ ∗ = { argmin ϕ ∥ A f ϕ ( z ∗ ) -y ∥ 2 2 : f ϕ ( z ∗ ) = z ∗ } , which corresponds to the minimizer of

The limit points of aSeqDIP correspond to the solution of a constrained version of the Vanilla DIP objective in (1). The constraint enforces additional prior that could alleviate overfitting. While its not straightforward to use a gradient-based algorithm for (6) given the hard constraint, the aSeqDIP scheme's limit points nevertheless minimize (6). Furthermore, aSeqDIP automatically estimates the network input by a sequential feed forward process without needing expensive updates. The main point is to show that aSeqDIP is solving the optimization problem in (6), which is different than Vanilla DIP in (1).

Remark 3.2 (Differences from Self-Guided DIP [20]) . While both aSeqDIP and Self-Guided DIP [20] update the input and network parameters simultaneously, there exist fundamental differences. Firstly, Self-Guided DIP solves the optimization problem in (2) which does not strictly enforce the auto-encoder constraint z = f ϕ ( z ) as in (6). Secondly, aSeqDIP only requires a network forward pass to update z , resulting in significantly fewer computations as will further be demonstrated in our experimental results. Thirdly, the second term in the aSeqDIP objective does not require computing the expectation, as it is an auto-encoder rather than a denoiser that results in higher resistance to noise overfitting. Lastly, our method does not require initializing z randomly and generating random vectors ( η in (2)). The selection of η introduces an additional hyper-parameter that we avoid, focusing solely

Vanilla DIP

Optimize Params.

Random

Input

Self-Guided DIP

Optimize Params.

Random

Input

Reference-Guided DIP

Optimize Params.

Reference

Image

DM-based Approaches

Pre-trained DM +

Random

Input

Data Consistency

Pre-trained DM +

Data Consistency

Pre-trained DM +

Data Consistency

Autoencoding Sequential DIP (Ours)

Measurements

Optimize Params.

Update

Input

Update

Input

Update

Input

Optimize Params.

Optimize Params.

Data-Independent Methods

Data-Dependent Methods

on selecting NK (total number of iterations) and λ (regularization strength), which are necessary in most DIP-based methods.

Remark 3.3 (Computational Requirements) . The computational requirements of aSeqDIP are determined by two factors: ( i ) the NK gradient-based parameter updates, and ( ii ) the number of function evaluations necessary for updating z , which is K . In our experiments, we have found that setting N = 2 and K = 2000 is generally sufficient. This configuration makes aSeqDIP nearly as efficient as Vanilla DIP.

Remark 3.4 (Relationship to DMs) . aSeqDIP bears resemblance to the reverse process in DMs due to their shared gradual denoising steps. However, despite these similarities, several distinctions emerge. Firstly, unlike the DM network, aSeqDIP does not require encoding a scalar representing time t . Secondly, and perhaps most significantly, aSeqDIP operates without requiring any training data or pre-trained networks. Thirdly, aSeqDIP operates in a truly sequential manner in terms of time, whereas in DMs, whether it's training (e.g., denoising score matching [39]) or sampling, the prevalent technique involves sampling from time t ∼ U [0 , 1] (uniform distribution), which allows for non-sequential time points.

3.2.1 Mitigating Noise Overfitting in aSeqDIP

In DIP-based approaches, noise overfitting occurs as the network attempts to fit its output to the noisy or subsampled measurements, y , as k increases during training. However, the specific value of k at which this PSNR decay begins is uncertain and varies across tasks and even among images within the same task and distribution. In aSeqDIP, when the output of network f ϕ k improves compared to that of f ϕ k -1 , the autoencoder term enforces similarity between the input and output of the network, thus delaying the onset of noise overfitting decay. This occurs because we are not only enforcing the network output to be measurement-consistent, but also enforcing that the output and input become similar. Consequently, as k increases, noise fitting is delayed, and utilizing the autoencoder provides regularization against noise overfitting. In Section 4, we will demonstrate how the proposed autoencoding term effectively regulates noise overfitting.

One might expect that incorporating the autoencoder could negatively impact reconstruction quality. However, empirical observations reveal that not only is noise overfitting delayed with the autoencoder term, but also image reconstruction quality is enhanced. To further support this statement, in Appendix B, we investigate whether a trained autoencoder on clean images can act as a reconstructor at testing time by optimizing the input.

4 Experimental Results

4.1 Settings, Datasets, and Baselines

In our experiments, we consider five tasks: MRI reconstruction from undersampled measurements, sparse-view CT image reconstruction, denoising, non-linear deblurring and in-painting. For MRI, we use the fastMRI dataset 3 . The forward model is y ≈ Ax ∗ . The multi-coil data is obtained using 15 coils and is cropped to a resolution of 320 × 320 pixels. To simulate undersampling of the MRI k-space, we use a Cartesian mask with 4x and 8x accelerations. Sensitivity maps for the coils are obtained using the BART toolbox [40]. For CT, we use the AAPM dataset 4 . For parallel beam CT, the input image with 512 × 512 pixels is transformed into its sinogram representation using a Radon transform (the operator A ). The forward model assuming a monoenergetic source and no scatter, noise is y i = I 0 e -[ Ax ∗ ] i , with I 0 denoting the number of incident photons per ray (assumed to be 1 for simplicity) and i indexing the i th measurement or detector pixel. We use the post-log measurements for reconstruction. We use a full set of 180 projection angles and simulate two different sparse view acquisition scenarios (with equispaced angles). Specifically, we created cases with 18 and 30 angles/views. The image resolution is kept at a fixed size.

For the tasks of denoising, in-painting, and non-linear deblurring, we use the CBSD68 dataset 5 . For each task, we use 20 measurements/corrupted images. To evaluate the reconstruction quality, we use the Peak Signal to Noise Ratio (PSNR), and the Structural SIMilarity (SSIM) index [41]. For experimental settings and baselines, see Table 1 and its caption. Note that we consider datadependent and data-independent baselines as shown in the third and fourth columns of Table 1. All the experiments are conducted on a single RTX5000 GPU machine. Further implementation details are provided in Appendix C.6.

For the proposed aSeqDIP method in Algorithm 1, we use the Adam optimizer with learning rate of β = 0 . 0001 . Furthermore, the regularization parameter is set to λ = 1 following the ablation study in Appendix C.4. We select N = 2 and K = 2000 following the ablation study in Appendix C.5.

4.2 Impact of the Autoencoding term on Noise Overfitting

In this subsection, we showcase the impact of the proposed autoencoding regularization in aSeqDIP on noise or null space (nuisance) overfitting.

We conducted experiments using 20 MRI scans and 20 CT scans, considering two cases of aSeqDIP as outlined in Algorithm 1. The first case sets λ = 1 , consistent with the remainder of the paper,

0

1000

2000

3000

4000

Iteration

i

15

20

25

30

35

PSNR (dB)

MRI

0

1000

2000

3000

4000

5000

Iteration

i

CT

aSeqDIP

Self-Guided DIP

Vanilla DIP

while the second case sets λ = 0 , effectively disabling the autoencoding regularization term in (3). Additionally, for comparison, we report results for Vanilla DIP and Self-Guided DIP. The average PSNR results for these cases are depicted in Figure 4.

As observed, when the autoencoder term is disabled in aSeqDIP (blue dashed lines), noise overfitting in MRI, akin to Self-Guided DIP, begins after nearly 1600 iterations. For CT, we note that aSeqDIP without regularization starts noise overfitting at around iteration 3800, whereas Self-Guided DIP experiences PSNR decay earlier, after approximately 1250 iterations. Importantly, when the autoencoding term is utilized (blue solid lines), not only does the decay in noise overfitting not commence until after iteration 4000, but the reconstruction quality (measured by PSNR) also improves.

As expected, PSNR decay in Vanilla DIP begins early, at around iteration 500 and 750 for MRI and CT, respectively. In Appendix C.4, we provide an ablation study to better show the impact of the value of λ in aSeqDIP.

4.3 Main Results

Here, we present our primary results regarding the reconstruction quality, measured by PSNR and SSIM, as well as the associated run-time. Table 2 presents the results for the considered tasks in this paper. Column 3 indicates whether the baselines depend on prior data or pre-trained models. The last three columns provide the PSNR, SSIM, and run-time results where the arrows indicate favorable results. For PSNR and SSIM, the settings correspond to the second column of Table 1. The black (resp. blue) text corresponds to the first (resp. second) setting. Values after the ± sign indicate standard deviation. Subsequently, we offer observations on the main results.

Compared to data-independent methods, i.e., the baselines that do not depend on a reference image or pre-trained models, aSeqDIP demonstrates improved PSNR and SSIM scores. For example, aSeqDIP, apart from Self-Guided DIP, shows nearly a 1dB improvement for MRI 8X acceleration compared to conventional methods. For the task of 30-views CT, aSeqDIP reports SSIM score of 0.92 which is 5% more than the second best, which is Self-Guided DIP with SSIM of 0.872. Although improvements against Self-Guided DIP are generally marginal in terms of reconstruction quality, our method proves to be 2X faster for MRI and CT reconstruction and requires 1 minute less than Self-Guided DIP for denoising and in-painting. This speed-up is attributed to updating the input using one forward pass of the trained network at each iteration k , instead of computing gradients with respect to the input for the update. Compared to Vanilla DIP, our method, on average, only requires an additional 30 to 60 seconds. When compared to ES-DIP [27], our method requires longer time, but on average achieves better reconstruction results across three tasks and different settings.

In comparison to data-dependent methods such as Score-MRI and MCG, our approach not only yields the best PSNR and SSIM but also requires reduced run-time, all without requiring any training data or pre-trained models. For instance, on average, aSeqDIP achieves nearly a 2dB improvement in 30-views CT compared to MCG while being 2X faster. In comparison to DPS, on average, our method report higher SSIM. Our method requires slightly less run-time on average but enhances the PSNR by approximately 0.6dB for both denoising and in-painting. Notably, our method is an optimization-based approach, whereas DM-based methods only require function evaluations. However, the generally larger run-time reported for DM-based methods is due to the necessity of

running a large number of reverse sampling steps. When compared to Ref-Guided DIP, our method achieves higher PSNR and SSIM results without the need for any prior (or reference image) image.

4.4 Visualizations

Figure 5 shows reconstructed images for the five considered tasks using aSeqDIP and the other baselines. Each row corresponds to a task. The first column displays the ground truth (GT) image whereas the second column shows the degraded image. Column 3 to the column before last present the reconstructed images by the baselines, while the last column shows the reconstructed images by aSeqDIP. PSNR values are provided at the bottom of each reconstructed image.

As observed, aSeqDIP achieves the highest PSNR scores. Additionally, the top right green boxes, which show the difference between the central region of the reconstructed and GT images, indicate that for MRI and CT, our method visually exhibits the least difference, making it the closest to the GT.

A similar observation is seen for the denoising task for the zoomed in bottom box. For inpainting, we note that aSeqDIP introduces the fewest unwanted artifacts as observed in the clouds (for DPS), and the left wing of the plane. While aSeqDIP contains artifacts for the task of Deblurring when compared to DPS, the latter generates a perceptually different image when compared to the GT. Similar observation are noticed with the additional visualizations provided in Appendix C.7.

MRI

CT

Denoising

In-Painting

Deblurring

GT

Degraded Image

Vanilla DIP

PSNR = 22.65 dB

PSNR = 30.12 dB

PSNR = 32.78 dB

PSNR = 33.78 dB

PSNR = 32.98 dB

PSNR = 34.44 dB

GT

Vanilla DIP

MCG

aSeqDIP

PSNR = 29.7 dB

PSNR = 32.57 dB

PSNR = 33.78 dB

PSNR = 33.01 dB

PSNR = 34.34 dB

GT

Vanilla DIP

SGLD DIP

Self-Guided DIP

DPS

aSeqDIP

PSNR = 27.88 dB

PSNR = 28.67 dB

PSNR = 29.11 dB

PSNR = 29.21 dB

PSNR = 30.77 dB

GT

aSeqDIP

PSNR = 26.65 dB

PSNR = 28.01 dB

PSNR = 28.12 dB

PSNR = 27.98 dB

PSNR = 28.35 dB

Ref-Guided DIP

Self-Guided DIP

Vanilla DIP

SGLD DIP

Self-Guided DIP

DPS

PSNR = 27.25 dB

aSeqDIP

DPS

SGLD DIP

Self-Guided DIP

Vanilla DIP

Degraded Image

Degraded Image

Degraded Image

Degraded Image

GT

Ref-Guided DIP

PSNR = 21.79 dB

Self-Guided DIP

PSNR = 25.67 dB

Score-MRI

PSNR = 23.42 dB

5 Conclusions & Future Work

In this paper, we introduced Autoencoding Sequential Deep Image Prior (aSeqDIP), a new unsupervised image recovery algorithm. Notably, aSeqDIP operates without the need of pre-trained models, relying solely on a sequential update of network parameters. These parameters are optimized using an input-adaptive data consistency objective combined with autoencoding regularization, effectively mitigating noise overfitting. Our experimental results across various tasks highlight the competitive performance of the proposed algorithm, matching (or outperforming) diffusion-based methods in terms of reconstruction quality and required run time, all without the need for pre-trained models.

For future directions, we aim to explore the applicability of aSeqDIP to other image recovery problems, thereby expanding its versatility and potential impact across diverse domains. Additionally, we are interested in investigating the integration of a network input update mechanism to dynamically adjust the autoencoding regularization parameter and the number of gradient updates per iteration.

Acknowledgments

This work was supported by the National Science Foundation (NSF) grants CCF-2212065, CCF2212066, and BCS-2215155. The authors would like to thank Xiang Li (University of Michigan), Avrajit Ghosh (Michigan State University), Huijie Zhang (University of Michigan), and Siddhant Gautam (Michigan State University) for insightful discussions.

aSeqDIP

References

Michael T McCann, Kyong Hwan Jin, and Michael Unser. Convolutional neural networks for inverse problems in imaging: A review. IEEE Signal Processing Magazine , 34(6):85-95, 2017.

B. Goyal, A. Dogra, S. Agrawal, B.S. Sohi, and A. Sharma. Image denoising review: From classical to state-of-the-art approaches. Information Fusion , 55:220-244, 2020.

Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11956-11965, 2021.

O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y. Akbari. Image inpainting: A review. Neural Processing Letters , 51:2007-2028, 2020.

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4681-4690, 2017.

J. A. Fessler. Model-Based Image Reconstruction for MRI. IEEE Signal Processing Magazine , 27(4):81-89, 2010.

I. A. Elbakri and J. A. Fessler. Statistical image reconstruction for polyenergetic X-ray computed tomography. IEEE Transactions on Medical Imaging , 21(2):89-99, 2002.

Lujun Zhai, Yonghui Wang, Suxia Cui, and Yu Zhou. A comprehensive review of deep learning-based real-world image restoration. IEEE Access , 11:21049-21067, 2023.

Hanene Ben Yedder, Ben Cardoen, and Ghassan Hamarneh. Deep learning for biomedical image reconstruction: A survey. Artificial intelligence review , 54(1):215-251, 2021.

Shijun Liang, Anish Lahiri, and Saiprasad Ravishankar. Adaptive local neighborhood-based neural networks for mr image reconstruction from undersampled data. IEEE Transactions on Computational Imaging , 2024.

Ling Chen, Zhishen Huang, Yong Long, and Saiprasad Ravishankar. Enhancing low-dose ct image reconstruction by integrating supervised and unsupervised learning. arXiv preprint arXiv:2311.12071 , 2023.

Bihan Wen, Saiprasad Ravishankar, Zhizhen Zhao, Raja Giryes, and Jong Chul Ye. Physicsdriven machine learning for computational imaging: Part 2 [from the guest editors]. IEEE Signal Processing Magazine , 40(2):13-15, 2023.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis , 80:102479, 2022.

Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Ye. Improving diffusion models for inverse problems using manifold constraints. Advances in Neural Information Processing Systems , 35:25683-25696, 2022.

Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. In The Eleventh International Conference on Learning Representations , 2023.

Huijie Zhang, Jinfan Zhou, Yifu Lu, Minzhe Guo, Peng Wang, Liyue Shen, and Qing Qu. The emergence of reproducibility and consistency in diffusion models. In Forty-first International Conference on Machine Learning , 2024.

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 9446-9454, 2018.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Shijun Liang, Evan Bell, Qing Qu, Rongrong Wang, and Saiprasad Ravishankar. Analysis of deep image prior and exploiting self-guidance for image reconstruction. arXiv preprint arXiv:2402.04097 , 2024.

Ismail R Alkhouri, George K Atia, and Alvaro Velasquez. A differentiable approach to the maximum independent set problem using dataless neural networks. Neural Networks , 155:168176, 2022.

HH Barrett. The radon transform and its applications. PROGRESS IN OPTICS , 31:217-217, 1993.

Avrajit Ghosh, Xitong Zhang, Kenneth K Sun, Qing Qu, Saiprasad Ravishankar, and Rongrong Wang. Optimal eye surgeon: Finding image priors through sparse generators at initialization. In Forty-first International Conference on Machine Learning , 2024.

Jiaming Liu, Yu Sun, Xiaojian Xu, and Ulugbek S Kamilov. Image restoration using total variation regularized deep image prior. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7715-7719. Ieee, 2019.

Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. A bayesian perspective on the deep image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5443-5451, 2019.

Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11) , pages 681-688. Citeseer, 2011.

Hengkang Wang, Taihui Li, Zhong Zhuang, Tiancong Chen, Hengyue Liang, and Ju Sun. Early stopping for deep image prior. Transactions on Machine Learning Research , 2023.

Taihui Li, Zhong Zhuang, Hengyue Liang, Le Peng, Hengkang Wang, and Ju Sun. Selfvalidation: Early stopping for single-instance deep generative priors. In Proceedings of the British Machine Vision Conference (BMVC), 2021. , 2021.

Di Zhao, Feng Zhao, and Yongjin Gan. Reference-driven compressed sensing mr image reconstruction using deep convolutional neural networks without pre-training. Sensors , 20(1):308, 2020.

Julián Tachella, Junqi Tang, and Mike Davies. The neural tangent link between cnn denoisers and non-local filters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8618-8627, 2021.

Xiang Li, Soo Min Kwon, Ismail R Alkhouri, Saiprasad Ravishankar, and Qing Qu. Decoupled data consistency with diffusion purification for image restoration. arXiv preprint arXiv:2403.06054 , 2024.

Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. In The Twelfth International Conference on Learning Representations , 2024.

Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexandros G. Dimakis, Chul Ye, and Mauricio Delbracio. A survey on diffusion models for inverse problems. 2024.

Y Xie and Q Li. Measurement-conditioned denoising diffusion probabilistic model for undersampled medical image reconstruction. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 655-664. Springer, 2022.

A Güngör, S U Dar, ¸ S Öztürk, Y Korkmaz, H A Bedel, G Elmas, M Ozbey, and T Çukur. Adaptive diffusion priors for accelerated MRI reconstruction. Medical Image Analysis , page 102872, 2023.

C Peng, P Guo, S K Zhou, V M Patel, and R Chellappa. Towards performant and reliable undersampled MR reconstruction via diffusion model sampling. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 623-633. Springer, 2022.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations , 2021.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation , 23(7):1661-1674, 2011.

J. I. Tamir, F. Ong, J. Y . Cheng, M. Uecker, and M. Lustig. Generalized magnetic resonance image reconstruction using the berkeley advanced reconstruction toolbox. In ISMRM Workshop on Data Sampling & Image Reconstruction, Sedona, AZ , 2016.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600612, 2004.

Gengsheng L Zeng. Fast filtered backprojection algorithm for low-dose computed tomography. Journal of radiology and imaging , 4(7):45, 2020.

Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5087-5096, 2021.

A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: convergence and generalization in neural networks. In Proceedings of the 32nd Neurips , NIPS'18, page 8580-8589. Curran Associates Inc., 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1026-1034, 2015.

Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, et al. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 , 2018.

Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated mri reconstruction. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part II 23 , pages 64-73. Springer, 2020.

Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations , 2023.

Appendix

In the Appendix, we first shed more light on the impact of the DIP network input by studying the training dynamics using the neural tangent kernel for CNNs with residual connections. Next, we show how trained autoencoders on clean images can be used as reconstructors at testing time. Lastly, we provide additional experimental results and visualizations.

A Case Study: Impact of the DIP Network Input through lens of Neural Tangent Kernel in Residual Networks

We show the impact of the DIP input through the lens of the Neural Tangent Kernel (NTK) [30, 44] for residual networks 6 . The NTK is a tool used to analyze the training dynamics of neural networks in the infinite width limit, where for CNNs the network width corresponds to the number of channels. In this limit, the change of any individual parameter during training becomes very small, which means that the change in the network's output during training can be accurately approximated by a first order Taylor expansion around its initialization. In the context of DIP, we consider training a neural network f with parameters θ and a fixed input z using gradient descent. At each training iteration, the network parameters are updated according to:

where L is the loss function, and β is the learning rate. We also consider the resulting change in the network's output due to this parameter update using the first order Taylor expansion:

Substituting (7) into (8) and applying the chain rule to write

yields the equation:

In the infinite width limit, NTK theory states that the matrix Θ ( t ) := ( ∇ θ f θ ( z ) ∣ ∣ θ = θ ( t ) )( ∇ θ f θ ( z ) ∣ ∣ θ = θ ( t ) ) T stays fixed throughout training, so that Θ ( t ) = Θ (0) for all t . This matrix is called the neural tangent kernel, and we denote it as Θ . Moreover, because the parameters θ (0) are initialized randomly, in the infinite width limit, the NTK Θ becomes deterministic (as a function of z ) due to the law of large numbers [30], and does not depend on the specific instantiation of θ (0) . In DIP, the loss function is the least squares loss given in (1). For simplicity, we consider the denoising case, where the forward operator A = I . Then, substituting the gradient of the loss into (10) shows explicitly how the output of deep image prior evolves during training:

Using this recursion relation, one can derive a closed form of the network output at iteration t in terms of the initial output and NTK [20, 30]. The reconstruction at iteration t is given by:

It is evident from (12) that the initial reconstruction of the network, f θ (0) ( z ) , has important effects on the training dynamics of DIP. Furthermore, networks used in DIP often feature skip connections from earlier layers to later ones, and it is natural to believe that these connections may cause the input z to have a large effect on f θ (0) ( z ) . In the following theorem, we analyze the training dynamics of CNNs with a very similar architectural modification: a residual connection that adds the input directly to the network output.

Theorem A.1 (Dynamics of DIP with Residual Connections) . Let g be a convolutional neural network with parameters θ . We consider the complementary residual network f defined by f θ ( z ) = z + g θ ( z ) . Suppose that f is trained using gradient descent with the loss L ( f θ ( z )) = 1 2 || f θ ( z ) -y || 2 2 . Then, in the infinite width limit (number of channels), in expectation over the initialization of the parameters θ , we have that the output at training iteration t is given by:

The proof of Theorem A.1 is provided in Appendix A.1, along with a precise statement of the assumptions on the network architecture and parameter initialization. Additionally, in Appendix A.2 we provide a simple experiment to validate that the training dynamics given in (13) hold for real networks.

Remark A.2. Theorem A.1 can be used to understand how the choice of network input affects the performance of DIP for image denoising. To gain intuition, we consider two special cases. First, we consider using the noisy image y as the input z . In this case, equation (13) simplifies to f θ ( t ) ( y ) = y for all iterations t . In this case absolutely no denoising occurs. On the other hand, we consider the oracle case where the clean image x is used as z . This gives us f θ ( t ) ( x ) = y -( I -β Θ ) t ( y -x ) . We see that at initialization ( t = 0 ), we already expect perfect denoising, since f θ (0) ( x ) = y -( I -β Θ ) 0 ( y -x ) = x . These two cases support intuition that using a network input closer to the true image could result in better performance in fewer training iterations.

A.1 Proof of Theorem A.1

Setting of Theorem A.1 . We first precisely state the conditions of Theorem A.1, in particular the network architectures considered and the corresponding parameter initializations. The present setting is very similar to the setting considered in [30], but we provide the details here for completeness. We consider an L layer CNN with c in input channels and c out output channels, with c hidden channels in all intermediate layers. We assume that c in , c out << c . We assume all convolutions have a filter size of r . For simplicity, the network input and output are vectorized, so convolutions of any dimension are treated identically. For example, for a 2D CNN with 5 × 5 kernels, r = 25 . Written explicitly, a network g with this architecture takes the form

where the operators C i represent convolutions with an additive bias, and φ is a pointwise activation function such as ReLU. In this section, we also consider the residual network architecture defined by f θ ( z ) = z + g θ ( z ) .

We assume that the parameters are initialized using the He initialization [45]. With this initialization, the first layer convolutional filter weights are drawn from N (0 , σ 2 w c in r ) , and the filter weights for all other layers are drawn from N (0 , σ 2 w cr ) , where the variance σ 2 w depends on the non-linearity used in the network. For ReLU networks, σ 2 w = 2 [45]. All biases are initialized to 0.

Proof. In the setting described above, the NTK emerges in the limit c → ∞ . A body of existing theory [30, 44, 46] establishes that in this limit the NTK is a deterministic matrix as a function of the network input z . This theory does not consider residual connections, but applies immediately to both g and f . For g , the NTK is given by

However, we can see that ∇ θ g θ ( z ) ∣ ∣ θ = θ (0) = ∇ θ f θ ( z ) ∣ ∣ θ = θ (0) .

Therefore, the linearization given in equation (10) holds for f using the same kernel Θ , and equation (12) describes the training dynamics of f .

Using equation (12), we can write:

To prove Theorem A.1, we consider the output f θ ( t ) ( z ) in expectation over the initialization θ (0) . Since all parameters θ (0) are drawn from mean 0 gaussian distributions, we find that E θ (0) [ g θ (0) ( z )] = 0 for any input z . Since Θ is deterministic in the limit c →∞ , in expectation over θ (0) equation (16) reduces to f θ ( t ) ( z ) = y -( I -β Θ ) t ( y -z ) , which proves Theorem A.1.

A.2 Example to support the results of Theorem A.1

We now provide a simple example using real networks to support the validity of Theorem A.1. This experiment substantiates both of the special cases considered in Remark A.2. Additionally, it shows that using the ground truth as the network input greatly inhibits overfitting. Indeed, in this experiment, we find that a residual network trained with the ground truth as input takes approximately 25 times more training iterations to completely learn the noisy signal than a residual network trained using a random noise input.

We use DIP for denoising a 1D sinusoidal signal. We denote the clean signal x . The noisy signal is y = x + n , where n ∼ N (0 , I ) . The network used is a five layer ReLU CNN with a residual connection. The full architecture can be written as f ( z ) = z + C 5 ( ReLU ( C 4 ( · · · ReLU ( C 2 ( ReLU ( C 1 ( z ))))))) , where each C i represents a convolution (with bias). The signal has a size of 100 , and the convolutions each have a filter size of 3 , with 64 hidden channels. In all cases, the network is trained using gradient descent with a learning rate of 5 × 10 -4 . The same seed was used to initialize the network in all cases.

We trained this network using three different inputs: the true signal x , the noisy signal y , and noise z ∼ N (0 , I ) . The results of training the network using these three inputs are shown in Figure 6. We find that the results for training this real, reasonably sized network show the behavior predicted by Theorem A.1, which was obtained in the infinite width limit. Indeed, equation (13) predicts that when x is used as the input, the initial error will be small, with eventual overfitting to the noisy signal. This is observed in Figure 6, where the error is lowest at initialization, and it steadily increases throughout training. With this input, the network is highly resistant to overfitting, requiring approximately 10 5 training iterations to completely fit the noisy signal. We also see that when y is used as the input, the error curve obtained is essentially flat and converges quickly to the error of the noisy signal. This agrees with the expectation that the network output will be y for all iterations t when y is used as the input. Finally, when random noise z is used as the input, the typical DIP behavior emerges.

0

20

40

60

80

100

-3

-2

-1

0

1

2

3

Signals Used in Denoising Experiment

Ground Truth x

Noisy Signal y

10 0

10 1

10 2

10 3

10 4

10 5

Training Iteration

0

2

4

6

8

10

12

Error of Networ  Output (ℓ2 Norm)

DIP Denoising Performance Using Differen( Inp)(s

Inp)( Gro)nd Tr)(h x

Inp)( Noisy Signal y

Inp)( Noise z ∼  (0, I)

Noisy Signal Error ||y - x--

2

B Trained Autoencoders as Reconstructors

Here, we investigate how the autoencoder term in aSeqDIP is improving the reconstruction quality while mitigating the impact of noise overfitting. In particular, we try to answer the question: Can an autoencoder trained on clean images operate as a reconstructor at testing time?

To address this question, we perform the following steps: ( i ) train an autoencoder on fully sampled measurements or clean data images and ( ii ) utilize the trained autoencoder with unseen subsampled or corrupted measurements, optimizing over the input using the DIP objective with the autoencoder term. This enables the autoencoder to function as an image reconstructor. Specifically, given a training dataset, D , comprising unperturbed images or fully sampled MRI/CT data, denoted by x , we train an autoencoder U-Net g : R n → R n with parameters ψ . The training process seeks to obtain ˆ ψ as

0.5

1.0

1.5

2.0

2.5

3.0

a

34

35

36

PSNR (dB)

Average of 8 MRI testing scans

Subsequently, given unseen measurements y and the learned autoencoder's parameters ˆ ψ , we test the reconstruction of

where λ a ∈ R + is a regularization parameter. We perform this experiment by training ψ using 3000 fully sampled scans from the fastMRI dataset [47]. We then evaluate the reconstruction quality of the trained encoder using 8 scans from the fastMRI testing set. The average PSNR results for different values of λ a are depicted in Figure 7. As observed, a trained autoencoder effectively serves as a reconstructor as evidenced by the achieved PSNR. Thus, we deduce that the autoencoder term in aSeqDIP not only mitigates noise overfitting but also enhances the reconstruction quality as an important prior.

C Additional Experiments

C.1 Robustness to Noise Overfitting for the Denoising Task

0

2000

4000

6000

8000

10000

Iterations

15.0

17.5

20.0

22.5

25.0

27.5

30.0

32.5

PSNR

Average PSNR for Denoising

aSeqDIP (Ours)

Self-Guided DIP

SGLD DIP

Vanilla DIP

In this subsection, we illustrate aSeqDIP's robustness to the noise overfitting issue for the denoising task. The average PSNR for 20 images from the CBSD68 dataset for denoising using aSeqDIP and other DIP-based methods are given in Figure 8. We observe two key points. First, in addition to higher PSNR, aSeqDIP shows higher robustness against noise overfitting compared to other DIP-based methods, consistent with MRI and CT results in Figure 4. Second, unlike MRI and CT, the onset of noise overfitting occurs earlier, but the subsequent decay is very small.

C.2 Comparison with VarNet: An End-to-End MRI Supervised Method

Here, we compare aSeqDIP with the End-to-End (E2E) MRI supervised model [48] that uses the variational network (VarNet). Results are given in Table 3. As observed, we slightly under-perform when compared to VarNet (trained on 8000 data points) for the task of MRI reconstruction all without

requiring any labeled training data. It is important to note that, at inference, E2E models only require a few unrolling steps, whereas aSeqDIP is an optimization method that requires to train the network parameters for each new set of measurements.

C.3 Comparison with DM-based Methods on the FFHQ Dataset

Here, we present average PSNR results (averaged over 20 images) for the tasks of denoising , random inpainting (97% missing pixels), box-in-painting (with HIAR of 0.25), and non-linear deblurring of our method versus Denoising Diffusion Null-Space Model (DDNM) [49] and DPS [16] on the FFHQ testing dataset. For DPS and DDNM, we used a pre-trained model that was trained on the training set of FFHQ. As observed, our training-data-free method achieves competitive or slightly improved results when compared to data-intensive methods on all tasks other than box-inpainting (for which we under-perform by less than 1 dB), all without requiring a pre-trained model.

C.4 Ablation Study on the Regularization Parameter in aSeqDIP

0

2000

4000

6000

8000

10000

20

25

30

35

PSNR (dB)

Average of 20 MRI measurements with Regularization Parameter

=0.5

Average of 20 MRI measurements with Regularization Parameter  =1

Average of 20 MRI measurements with Regularization Parameter  =2

Iteration

i

In this section, we conduct an ablation study on the choice of the autoencoding regularization parameter, λ , in (3). Specifically, we conduct an experiment using 20 MRI scans to examine the impact of λ on the reconstruction quality and noise overfitting in aSeqDIP. We note that while the main results in Section 4 use NK = 4000 , in these experiments, we run our algorithm for an extended number of iterations to investigate the onset of noise overfitting. We set K = 5000 and N = 2 for this purpose.

In this experiment, we run aSeqDIP with values of λ ∈ { 0 . 5 , 1 , 2 } . Average PSNR results are given in Figure 9. It is evident that, on average, using λ = 1 yields the most favorable results in terms of

PSNR values, which is our selected choice. Furthermore, we observe that for λ = 0 . 5 (red), the start of the PSNR decay (the onset of noise overfitting) precedes that of λ = 1 and λ = 2 (blue and black).

C.5 Ablation Study on N and K in aSeqDIP

In this section, we conduct an ablation study to investigate the impact of the number of gradient updates ( N ) per one set of parameter ( K ) in aSeqDIP. Specifically, we report the PSNR results across the tasks of MRI, CT, denoising, and in-painting for the case of NK = 4000 , considering combinations of ( N,K ) as (1 , 4000) , (2 , 2000) , and (4 , 1000) . The results, presented in Figure 10, reveal that across all tasks considered, the combination of N = 2 and K = 2000 consistently yields the most favorable results in terms of PSNR.

MRI4x

CT30view

Od=25

HAIR = 0.25

34.50

35.00

25.35

34.25

34.95

32.2

25.30

(aP)

34.00

(ap)

34.90

(ap)

(aP)

25.25

33.75

34.85

32.1

NR

P

33.25

33.50

PSNR

34.80

34.75

PSNR

P

32.0

SNR

P

25.15

25.10

25.20

33.00

32.75

34.70

31.9

25.05

34.65

(1,4000)(2,2000)(4,1000)

(1,4000)(2,2000)(4,1000)

(1,4000)(2,2000)(4,1000)

(1,4000)(2,2000)(4,1000)

(N,K)

(N,K)

(N,K)

(N,K)

C.6 Additional Implementation Details

For denoising, each noisy RGB image is 512 × 512 and is generated by adding an additive Gaussian white noise with two noise levels as descired in Table 1. For in-painting, we consider a central region mask, and we evaluate two hole-to-image area ratios (HIAR) with image size 512 × 512 .

For Vanilla DIP, Self-Guided DIP, Reference-guided DIP, TV-DIP, Rethinking DIP, and SGLD DIP, we use 4000 iterations. For TV-DIP, we set the regularization parameter to 1 . For ES-DIP, we use the default configuration provided by the authors for the three considered tasks.

The reference image in Reference-guided DIP is chosen as (using a distance metric such as Euclidean distance or other metric) that which is most similar to an estimated test reconstruction from undersampled data or sparse-view data.

For DM-based approaches, we use the codes attached to the authors' papers. Specifically, Score-MRI [14], MCG [15], and DPS [16]. For our experiments with natural images (denoising, in-painting, and non-linear deblurring) in Table 2, we used the CBSD68 dataset. As such, for DPS (the DM-based method), we utilized a pre-trained model that was trained on a very large and diverse dataset which is ImageNet 128 × 128 , 256 × 256 , and 512 × 512 . This pre-trained model is much more generalizable when compared to the other option which was trained on FFHQ (a dataset of faces). According to [17], the ImageNet pre-trained model has high generalizability. For the FFHQ comparison results in Appendix C.3, we used an FFHQ-pre-trained DM for DPS and DDNM.

For MRI, the pre-trained model used in Score-MRI was originally trained on natural images then fine-tuned using the training set of fastMRI. Similar approach was used for the CT pre-trained model used in MCG.

C.7 Additional Visualizations

Figures 11 and 12 present additional MRI visualisations, whereas Figures 13 and 14 present CT visualizations. Samples from the natural image restoration tasks are given in Figure 15, Figure 16, and Figure 17 for box-inpainting, denoising, and non-linear deblurring, respectively.

Ground Truth

PSNR = 31.2 dB

Input

PSNR = 34.68 dB

PSNR = 30.26 dB

Ground Truth

PSNR = 30.5 dB

Input

PSNR = 33.18 dB

PSNR = 28.96 dB

PSNR = 32.31 dB

PSNR = 32.45 dB

Ground Truth

Input

Self-Guided

Reference Guided

PSNR =

∞

dB

PSNR = 17.12 dB

PSNR = 33.05 dB

PSNR = 32.31 dB

MCG

aSeqDIP

Vanilla DIP

PSNR = 32.47 dB

PSNR = 33.48 dB

PSNR = 29.77 dB

Ground Truth

Input

Self-Guided

Reference Guided

PSNR =

∞

dB

PSNR = 22.67 dB

PSNR = 34.45 dB

PSNR = 33.31 dB

MCG

aSeqDIP

Vanilla DIP

PSNR = 34.15 dB

PSNR = 35.18 dB

PSNR = 30.16 dB

Ground Truth

Input

Self-Guided

SGLD

PSNR =

∞

dB

PSNR = 14.56 dB

PSNR = 24.21 dB

PSNR = 23.02 dB

DPS

aSeqDIP

Vanilla DIP

PSNR = 23.12 dB

PSNR = 24.35 dB

PSNR = 22.16 dB

Ground Truth

Input

Self-Guided

SGLD

PSNR =

∞

dB

PSNR =19.68 dB

PSNR = 30.19 dB

PSNR = 29.62 dB

DPS

aSeqDIP

Vanilla DIP

PSNR = 29.92 dB

PSNR = 30.35 dB

PSNR = 28.78 dB

F16

+

16

F6

Ground Truth

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The Abstract and Introduction highlight the challenges of previous works, leading to the motivation of the our proposed approach, followed by the contributions of the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 3, we show that while we use the autoencoding regularization term to mitigate the issue of noise overfitting, the problem still exists. Nevertheless, it is highly mitigated when compared to previous works. This was supported by experiments.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: For the first remark, we provide the assumptions needed to state the results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the detailed settings, baselines, and forward models for each imaging task we consider (Table 1). Furthermore, the implementation details for baselines are provided in Appendix C.6.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our code that contains all the experiments of our method and most baselines are included in the repo link we provide.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 4, we provide all the details needed to understand the results.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report average PSNR and SSIM scores as well as standard deviation in Section 3 for the main experiments.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the beginning of Section 4, we exactly specify the GPU machine used to run our method and all other baselines. Furthermore, in Table 2, we report run time results.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Authors have reviewed the code of ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The paper addresses the problem of image reconstruction, and our method can not be used for image manipulation. Therefore, there is no societal impact of the work performed.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: We cite the code used for baselines, and provide the links used for each dataset.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the code used for baselines, and provide the links used for each dataset.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We only release code for our algorithm.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Autoencoding Sequential Deep Image Prior (aSeqDIP) an extension to the family of untrained (no training data required) DIP methods. \nAs in previous work a combination of data consistency loss and autoencoding regularization (a loss between network input and output) is used. However, instead of further regularizing the optimization by adding noise to the network input in every iteration as proposed in previous work, this paper varies the network input by setting it to the network output after every two iterations. \nExperimental evidence is provided that demonstrates superior robustness to overfitting and overall reconstruction accuracy compared to other DIP approaches and trained diffusion models.\n\nSTRENGTHS:\nSignificance: Untrained neural networks are an important technique used in many fields, where training data is scarce. Mitigating the problem of overfitting is an important contribution. \n\nOriginality: Despite the relatively small difference compared to previous work (regularize network input by setting it equal the output instead of adding noise to it as in the self-guided DIP) the importance and effectiveness of this idea seems to be justified by the experimental results in terms of reconstruction accuracy and robustness to overfitting.\n\nClarity: Overall the paper is written in a clear way and related work in terms of DIP approaches and diffusion models are discussed.\n\nQuality: The experiments towards evaluating the reconstruction performance and overfitting robustness of the proposed DIP approach are well designed and ablation studies for the most important hyperparameters are provided. \n\nOverall an interesting paper. However, some questions remain and some important points remain unclear (see weeknesses). If those can be addressed adequately, I would consider raising the score.\n\nWEAKNESSES:\n**1. No results are provided for the most interesting case: Overfitting robustness for the task of denoising.**\nTo my understanding Prop. 3.1 says that in the limit the proposed approach converges to a network that has learned the identity and to a network output (which is identical to the network input) that perfectly fulfills data consistency under the forward operator and the given measurement. \nFor the tasks of MRI and inpainting (in the absence of noise) solutions that fulfill Prop. 3.1 include the ground truth image and the empirical evidence provided in the paper shows that indeed the solution found by the proposed method is closer to the ground truth image than that of other DIP based approaches (why it is closer remains unclear up to intuitions/speculations, but answering this question is non-trivial and does not have to be the scope of the paper). \n\nAlso, the robustness to overfitting seems reasonable for the aforementioned tasks. My intuition would be that repeatedly setting the network input to the output combined with the autoencoding regularization loss lets the network converge to identity relatively quickly. If at the same time data consistency is reached, there is no incentive for the network to change its weights anymore and a stable solution is reached. \n\n**However, things change in the presence of noise or in general for the task of denoising.** \nTo my understanding, for denoising Prop. 3.1 is fulfilled, when the network perfectly reconstructs the noisy measurement $\\mathbf{y}$, which is the very definition of overfitting. So, here it would be interesting to see curves as presented in Figure 4 for MRI and CT also for denoising to demonstrate to what extend overfitting is still prevented.\n \n**2. In general, I am not sure about the usefulness of Prop. 3.1.** \nAs discussed above for the tasks of MRI and inpainting reconstructions that fulfill Prop 3.1 comprise good and bad solutions however without making a statement why a good solution should be preferably reached over a bad solution, whereas for the task of denoising it implies convergence to a bad solution. \n\n**3. The learning based \"SOTA\" baselines perform very badly, which is not discussed in the paper and no information regarding the used training sets is provided.** \nDiffusion model-based methods are introduced as the state-of-the-art in the paper (line 113) and indeed the works referenced there report impressive results in terms of quantitative and qualitative reconstruction performance. \nYet, in this work the reconstruction scores and especially the reconstructed images (Figure 5, 12, 13) contain severe artifacts. I understand that training and inference of those diffusion models is sensitive to hyperparameters, but then potential reasons for this bad performance should at least be discussed in the paper. Also, no information is provided regarding the training data used for those baseline models.\n\nIf there is no comparison to state-of-the-art end-to-end approaches (which definitely should give better results than DIP based reconstruction if trained properly on enough data) like the end-to-end VarNet (https://arxiv.org/abs/2004.06688) then at least the diffusion model based baseline should show reasonable results or if not it should be discussed why. \n\n**4. Code is only provided for the task of MRI reconstruction and the implemented method seems to differ from the description in the paper.** Checking the code provided via a link (https://anonymous.4open.science/r/Aseq_DIP-E728/README) in the paper I can only find a notebook that performs MRI reconstruction *Aseq_DIP.ipynb*. \nIn the code the step where the network output is set to be the network input for the following steps is implemented as follows:\n```\n    pred_ksp = mps_and_gt_to_ksp(mps1.to(device), net_output)\n\n    new_pred_ksp = (1 - mask_from_file).to(device) * pred_ksp.detach() / scale_factor + mask_from_file * ksp1\n\n    new_ref = ksp_and_mps_to_gt(new_pred_ksp, mps1)\n    random_smoothing_temp = torch.zeros_like(new_ref).to(device)\n\n    for jj in range(1):\n        random_smoothing = new_ref + 1e-5 * torch.rand((640, 372), dtype=torch.complex64).to(device)\n        random_smoothing_temp += random_smoothing\n\n    random_smoothing_final = random_smoothing_temp \n    ref[:, 0, :, :] = random_smoothing_final.real\n    ref[:, 1, :, :] = random_smoothing_final.imag\n```\nIf I understand it correctly the new input (ref) is first set to fulfill perfect data consistency and then processed with random smoothing, two things not mentioned anywhere in the paper. Also, the new input (ref) is updated before the computation of the loss, which means that in fact the autoencoding regularizer is not really computed between the network input and the network applied to this same input.\nIt is unclear how essential these two steps (hard data consistency and smoothing) are to the proposed approach.\n\nFurther, \n```\n    loss.backward()\n    for i in range(2):\n        optimizer.step()\n```\nimplies that the network weights are updated twice without updating the loss functions, which is not what is outlined in Algorithm 1 of the paper.\n\nIf that is really how the mehtod is implemented that corresponds to the results in the paper the ablation study regarding the values for the number of iterations $N$ per network in Appendix C.4 seems meaningless as the loss is not updated as $N$ increases.\n\nQUESTIONS:\n1. In the Figures (like Fig. 5) showing qualitative reconstruction results it would be helpful to show the measurement, i.e. noise image for denoising, mask fr in-painting and zero-filled reconstruction for MRI.\n\n2. I personally find the formulation in line 10 \"sequential optimization of multiple network architectures\" a bit missleading as the networks are not changed or re-initialized. Only the network input is adapted. \n\n3. Regarding Section 3.1 and Figure 2 it makes sense to me why the blue curve is decaying, but do you know why the red curve is decaying? Does the red curve decaying imply that a constant input (like all zeros) to the DIP would work best?\n\nLIMITATIONS:\nLimitations are not discussed in the paper. This could be for example the limitted interpretability of the comparison to trained baseline methods as current results in the paper significantly fall behind the results reported in the baseline papers or the long reconstruction times compared to end-to-end deep learning based methods.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript describes a variant of the seminar Deep Image Prior (DIP) work that incorporates aspects and mindset of the iterative prediction workflow in diffusion models.\nIn particular, the proposed procedure (method?) aSeqDIP is training a network to predict a single and fixed but distorted (e.g. noisy) image when given a pixel-wise random input image (just as DIP has initially proposed). The key novelty is to only train for a few training steps before switching to feeding the current prediction as an updated input (instead of the initial noisy input image). The similarity to predictions with diffusion models is apparent.\nThe approach is to the best of my knowledge novel and the idea, in my point of view, the biggest contribution of the paper.\n\nSTRENGTHS:\n* The idea is fantastic and thought provoking. \n* Introduction of the regularization term (autoencoding term) during iterative training and showing that it is useful and that tuning it is important. (Figure 7 from the supplement could make it into the main paper.)\n* Application to 4 different tasks (MRI and CT (important real-world use-cases) and denoising and in-painting (to make the CV community happy… ;)\n* Partially informative appendix.\n\nWEAKNESSES:\n* It surprises me that the manuscript does not argue more about WHY this approach leads to better results even when compared to data-dependent baselines. How strong are these baselines? Do better pre-trained methods exist? If so, why not show results with them as well?\n   * An answer to the above “WHY?” question would go right to the heart of why diffusion works well and would therefore be very interesting.\n   * Since the presented method/procedure is not dependent on any amount of available training data (but the single compromised target image) I wonder how this manuscript can avoid talking about inductive biases of the used network architecture (a UNet, hence, a CNN).\n* The manuscript is overall rather compact in content, the text certainly not too dense but maybe even a bit too repetitive and wordy, and the figures not very legible (adequately sized). I also much regret not to see the compromised target image used by aSeqDIP, but only GT and final predictions.\n* Figure 1: when seeing it first it really did not help me understand the paper any better. After reading the entire manuscript and coming back to Figure 1 I can confirm the figure makes sense, but one would hope that Figure 1 is more educational as it currently is.\n* Figure captions are in general not terrible but also not in all cases making the figures self-contained. It is necessary to find some potentially distant place in the manuscript to fully grasp the visuals (e.g. Fig. 2).\n\nQUESTIONS:\n* I cannot judge the quality or sufficiency of the used baseline methods. If for any of the 4 tasks the current SOTA (or industry standard) method would also be given and compared against, I would find that very useful.\n* When does the proposed approach work (when is it applicable) and what are known limitations of its applicability? (When does the inductive bias of the setup not suit the desired task?)\n\nLIMITATIONS:\n* I would expect that the inductive biases that comes with the used network and training procedure dictate for what problems the presented method/procedure can produce good results. If the authors have any thoughts, I think the manuscript would much benefit from a short discussion in that vein.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this paper propose an Autoencoding Sequential DIP (aSeqDIP) which aims to address the overfitting issue of DIP while without introducing extra parameters. The idea is very simple, the authors simply feed the output of the DIP into DIP model after each N updates. The authors validate the efficiency of their methods on several different image restoration tasks.\n\nSTRENGTHS:\n1) The presentation of the paper is good and it is very easy to understand.\n2) The authors have conducted experiments on for different image restoration tasks.\n3) The authors provide numerical comparisons as well as visual comparisons.\n\nWEAKNESSES:\nI have several concerns about this paper.\n\n1) The novelty is significantly limited. It is almost the original DIP. The only difference is: the original DIP each iteration uses a random noise as its input; while here, the output of DIP is fed into the DIP for the next N iteration's updates.\n\n2) The current experiments are all linear image restoration tasks. It would be interesting to see how this model works for non-linear image restoration tasks such as image delurring.\n\n3) The authors should compare their method with more advanced DIPs such as Ref1.\n\n4) How does this model compare with SOTA diffusion models Ref2?\n\n5) The current results (Table2), the improvement is very small. And I am wondering what if we run the experiments many rounds and report the mean and std. \n\n\nRef1: Jo, Yeonsik, Se Young Chun, and Jonghyun Choi. \"Rethinking deep image prior for denoising.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nRef2: Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\nQUESTIONS:\nI have several questions which have been listed in [Weaknesses].\n\nLIMITATIONS:\nI have listed the limitations in [Weaknesses].", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates how to prevent deep image prior (DIP) from overfitting to the noise or compressed measurements, which is a classic problem of DIP. To address the problem, the authors proposed Autoencoding Sequential DIP (aSeqDIP). The general idea of aSeqDIP is to cuts the overall training process into K sequential blocks, in each of which the input ($\\mathbf{z_k}$) of the network is the previous output ($f(\\mathbf{z_{k-1}})$), and the loss is defined as $\\mathcal{L} = ||Af(\\mathbf{z}_k) - y||_2^2 + || z_k - f(\\mathbf{z}_k)||_2^2$, where the second term forces the output to be alike the input. As k increases, the network aims to find the network $f$ such that $z = f(z)$ and $||Af(\\mathbf{z}_k) - y||_2^2$ is minimized. In a word, aSeqDIP forces the output to be a fixed point of the network that fits the measurements as much as possible.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow\n\n2. The proposed approach for enforcing DIP to find its fixed point is interesting and novel\n\n3. Superior performance has been demonstrated again current diffusion-model-based approaches for image reconstruction tasks.\n\nWEAKNESSES:\n1. Although the experiment is thorough in terms of the variety of image reconstruction tasks, I still think it misses an important baseline that uses explicit regularizer (e.g. TV) to stablize the learning process. It would strengthen the argument if the authors could include a comparison like that.\n\n2. The current introduction is concise, but it does not convey the message that the key benefit of aSeqDIP is to address the problem of overfitting.\n\n3. The authors do not discuss the limitations of the proposed method.\n\nQUESTIONS:\n1. Please consider including a baseline like DIP+TV in the experiment.\n\n2. What is the runtime of aSeqDIP? A comparison between aSeqDIP and other baselines is preferred.\n\n3. [Subjective] The proposition seems redundant: 1) it assumes very strong conditions (the convergence of the network), which, in my opinion distracts the reader from the key message; 2) it kinda shows how aSeqDIP prevent itself from overfitting, but it is not very clear without text. I suggest to write the proposition in the form of plain text with equations.\n\nLIMITATIONS:\nPlease discuss the limitations such as runtime, practical convergence, etc.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}