UGoDIT: Unsupervised Group Deep Image Prior Via Transferable Weights
Authors: Shijun Liang
Ismail Alkhouri
Siddhant Gautam
Qing Qu
Saiprasad Ravishankar
Venue: NeurIPS 2025 poster ()
Keywords: Inverse Imaging Problems
Deep Image Prior
Unsupervised Learning
Magnetic Resonance Imaging
Primary area: applications

Abstract: Recent advances in data-centric deep generative models have led to significant progress in solving inverse imaging problems. However, these models (e.g., diffusion models (DMs)) typically require large amounts of fully sampled (clean) training data, which is often impractical in medical and scientific settings such as dynamic imaging. On the other hand, training-data-free approaches like the Deep Image Prior (DIP) do not require clean ground-truth images but suffer from noise overfitting and can be computationally expensive as the network parameters need to be optimized for each measurement vector independently. Moreover, DIP-based methods often overlook the potential of learning a prior using a small number of sub-sampled measurements (or degraded images) available during training. In this paper, we propose **UGoDIT**—an **U**nsupervised **G**r**o**up **DI**P with **T**ransferable weights—designed for the low-data regime where only a very small number, $M$, of sub-sampled measurement vectors are available during training. Our method learns a set of transferable weights by optimizing a shared encoder and $M$ disentangled decoders. At test time, we reconstruct the unseen degraded image using a DIP network, where part of the parameters are fixed to the learned weights, while the remaining are optimized to enforce measurement consistency. We evaluate \our on both medical (multi-coil MRI) and natural (super resolution and non-linear deblurring) image recovery tasks under various settings. Compared to recent standalone DIP methods, \our provides accelerated convergence and notable improvement in reconstruction quality. Furthermore, our method achieves performance competitive with SOTA DM-based and supervised approaches, despite not requiring large amounts of clean training data. Our code is available at: https://github.com/sjames40/UGoDIT.

UGoDIT: Unsupervised Group Deep Image Prior Via Transferable Weights
Shijun Liang" 1.2 Ismail R. Alkhouri*
Siddhant Gautum". Qing Qu', Saiprasad Ravishankar
1 Department of Electrical Engineering & Computer Science, University of Michigan Ann Arbor
Department of Computational Mathematics, Science, & Engineering, Michigan State University 'Department of Biomedical Engineering, Michigan State University {ahijunli, iamailal, gingqu)@umich. Sumich. edu {liangs16, alkhour3, gautamsi. raaiisha@@@mau.eee edu
Abstract
Recent advances in data-centric deep generative models have led to significant progress in solving inverse imaging problems. However, these models (e.g.. diffu- sion models) typically require large amounts of fully sampled (clean) training data, which is often impractical in medical and scientific settings. Training-data-ll approaches like Deep Image Prior (DIP) not require clean images but suffer from noise overfitting and can be computationally expensive as the network parameters need to be optimized for each measurement vector independently. Moreover, DIP- based methods often overlook the potential of learning prior using small number of sub-sampled measurements (or degraded images) available during training. this paper, we propose UCoDIT an Unsupervised Group DIP via Transferable weights-designng for the low-data regime where only a very small number, M, sub-sampled measurement vectors are available during training. Our method learns a set of transferable weights by optimizing a shared encoder and M disentangled decoders. At test time, we reconstruct the unseen degraded image using DIP network, where part of the parameters are fixed to the learned weights, while the remaining optimized to enforce measurement consistency. We evaluate UGoDIT on both medical (multi-coil MRI) and natural (super resolution and non- linear deblurring) image recovery tasks under various settings. Compared to recent standalone DIP methods, UGoDIT provides accelerated convergence and notable improvement in reconstruction quality. Furthermore, UGoDIT achieves perfor- mance competitive with SOTA DM-based and supervised approaches, despite not requiring large amounts of clean training data. Our code available at: UGoDIT.
Introduction
Solving inverse problems is important across wide range of practical applications, such those in robotics [1], forensics [2], remote sensing [3], and geophysics [4]. One class of such problems Inverse Imaging Problems (IIPs), including super-resolution [5], image denoising [6], image deblurring [7], recovering missing portions of images [8] (in-painting), and magnetic resonance
imaging (MRI) reconstruction [9].
In recent years, an abundance of deep learning (DL) methods have been developed to solve IIPs [10, 11, 12, 13, 14]. Based on the availability of training data points, DL-based IIP methods can be categorized into (i) data-less DL approaches, such as the ones based on the Deep Image Prior (DIP)
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[15]. Implicit Neural Representation (INR) [16], and zero-shot learning with self-supervision [17]; and (ii) data-centric DL approaches, such as the ones based on supervised networks [18. 19], unsupervised [20], or generative models such as those based on diffusion models (DMs) [13, 14].
Data-centric methods rely pre-trained models that typically require large training sets. In the context of scientific or medical IIPs, in addition to being large-scale, this translates to the need for fully sampled measurements--iee. clean images (serveing as ground truth). Unlike natural images, acquiring large datasets of scientific or medical data such as fully sampled MRI scans is particularly challenging [21. 22, 23]. The underlying data may be inherently undersampled or corrupted with unknown noise. Similar issues arise in other domains such as medical computed tomography (CT) [24]. To mitigate this, previous DM-based methods for MRI and CT often fine-tune DMs pre-trained on natural images using a limited number of fully sampled medical images [13, 12, 14]. This underscores the need for approaches that reduce reliance on large, clean (or fully sampled) datasets.
Training-dan degg-ddt--feeee methods, such as DIP [15], which rely on untrained networks (e.g., U-Net [25]). avoid the need for training data altogether- sampled or otherwise. However, they often face two main challenges: (i) vulnerability to noise overfitting (though recent variants, such as auto-encoding sequential DIP (aSeqDIP) [26], have shown improved robustness). and (ii) high computational cost at inference time (as a full set of network parameters need to be optimized for each measurement independently) especially compared to the inference time of supervised methods. Although recent work, such as MetaDIP [27]. has explored transferring learned DIP weights to speed up inference, it still requires supervised training with fully sampled data points, and does not achieve competitive results when compared to standalone DIP methods as was demonstrated in [28].
The scarcity of fully- sally-s measurements (i.e., clean images as ground truth) in certain applications, and the inability of most DIP methods to transfer learned weights serve as our motivation to explore the following question:
possible to train a fully unsupervised model using only a few degraded images, such that the learned weights can later be used to reconstruct unseen measurements during inference? We term the regime when only very few training images are available the "low-data regime", and towards answering the above question, we make the following contributions:
Decoder-diseetanggle architecture for training: Given few measurement vectors (or degraded images), we propose the use of a shared encoder and multiple decoders for each measurement vector during training such that the inputs, shared encoder, and multiple decoders optimized jointly using a measuremeet-consiitte
and auto-encoded objective function.
Transferable weights testing: Given an unseen measurement vector, we transfer the pre-trained weights of the shared encoder to a single-encoder, single-decoder DIP network. Reconstruction is then performed by fixing the encoder and optimizing the decoder parameters using an input-adaptive, www.urrmmmen--oosssttet objective function. Therefore, we term our method as Unsupervised Group DIP with Transferable weights (UGoDIT).
Extensive evaluation: We conduct extensive evaluations using multi-coil MRI, super resolution, and non-linear deblurring. We show that we outperform most recent advanced DIP methods while performing comparably with DMMbased and supervised data-intensive methods all while requiring only a very few degraded images. Furthermore, our ablation studies show the impact of each component such as the proposed disentangled decoders.
2 Preliminaries & related work
Inverse Imaging Problems (IIPs) are defined as recovering an image x' Rnd from measurement vector (or degraded image) y E TRm (with m & n)) governed by the forward operator A. For multi- coil MRI, the forward operator is given A MFS, for which M denotes coil-wise undersampling, F is the coil-by-coil Fourier transform, and S represents the sensitivity encoding with multiple coils". For natural image restoration tasks, the forward operator in super resolution (SR) is a down sampler. For non-linear deblurring (NDB), the non-linear forward operator is the neural network approximated kernel given in [7] and adopted in multiple works such as [14, 29].
Next, we provide a brief overview of closely related DL-based image reconstruction methods based on the availability of training data.
In MRI. the entries of x" and y contain both real and imaginary numbers. However, to generalize the detinition of the problem, use real numbers.
2

---

High-data regime methods: Training data-intensive methods-then is, methods that rely on large number of training data points-ce can be categorized into generative, supervised, and unsupervised approaches. Among generative methods, recently, numerous IIP solvers have employed pre-trained diffusion models (DMs) as priors (e.g.. [13, 14]; see [30] for a recent survey). These methods typically modify the reverse sampling steps to enable sampling from a measurement-condiionn
distribution. In supervised end-to-end (E2E) approaches, a variety of techniques have been explored, including unrolled networks such as the variational network (VarNet) in [18]. A common limitation of both generative and supervised methods is their reliance not only on large amounts of training data, but also on clean (i.e., fully sampled) ground truth images an assumption that may not hold in many medical and scientific applications such as MRI reconstruction' Moreover, DM- DM-based generative methods are generally slower at inference compared to supervised counterparts. Unsupervised (or self-supervised) methods have also been explored [32, 33, 34]. While they do not require clean images, they still depend on large amount of training data and often fall short of the performance achieved by supervised or generative models [35]. In this paper, we introduce a method that requires neither large-scale training data nor clean images (or fully sampled measurements).
Training-data- DIP-based methods: The pioncering work of [15] introduced the Deep Image Prior (DIP), a fully training-data-fre approach for solving IIPs, demonstrating that an untrained convolutional neural network (CNN) can effectively recover low-frequency image structures using the implicit bias of the network. However, DIP is prone to noise overfitting due to the network's s over-parameetrrizaton.. As a result, numerous techniques have been proposed to mitigate this issue, including the use of regularization [36, 26), early stopping [37], and network re- ee-parameterization [28, 38]. We refer the reader to the recent tutorial in [39] for a comprehensive overview of DIP and its variants. Most DIP methods require optimizing full set of weights for each mcasurement vector independently. In this paper, we present a DIP-based method that Iearns a subset of transferable weights from only few degraded images, addressing this gap.
Low-data regime methods: Low-data regime refers to the case where the number of training images is extremely small for training standard supervised, generative, and/or unsupervised methods. Here, we discuss two methods that operate in the low-data regime and are closely related to our work. The first is MetaDIP [27], where the authors leverage meta-learning to learn transferable weights for accelerated DIP reconstruction at test time. MetaDIP adopts standard single-encoder, single-decoder U-Net architecture. There are three key differences between UGoDIT and MetaDIP: (i) MetaDIP requires clean images during training, while our method uses only degraded images (or sub- sub-sampled measurement vectors); (ii) MetaDIP does not include architectural changes during training; (ii) their evaluation is limited to natural image restoration tasks, whereas our method also considers more complex, large-scale MRI reconstruction. The second method is STRAINER [40], which extends the Implicit Neural Representation (INR) framework [41, 42] to the low-data regime. Specifically, the authors propose using a fully connected network (or multi-layer perceptron, MLP) with multiple output sub-networks to learn transferable weights. At test time, a subset of these weights is used to initialize a subset of the MLP-based INR model for image recovery. The main differences between UGoDIT and STRAINER are: (0) STRAINER uses MLPs with spatial coordinate input embedding, while method is based on CNNs and DIP; (ii) STRAINER uses clean (fully sampled) training data, whereas UGoDIT uses degraded images (or sub-sampled measurements); (iti) the learned weights in STRAINER are used only for initializing part of the test-time INR network, whereas in our method, we keep the parameters of the pre-trained shared encoder unchanged during testing and only adapt the parameters of the test-time decoder.
3 Proposed method
Assume that we are given a set Y, consisting of M under-sampled measurement vectors (or degraded images), where each y; E Y, for i E [M] := (I,..., M). is an m-dimensional vector. The degraded images in set Y are obtained using the following forward models:
...,,,,,
y, Axx + 77
(1)
'W note that there have been recent efforts to train DMs on degraded images, such as the study in [31]. However, these approaches still rely large amounts of training data.
3

---

where 77 represents the noise in the measurement domain, e.g., assumed sampled from a Gaussian distribution (000 oz I), with yy > 0 representing the noise level of the i-th vector. this setting. we assume access only to y; and the corresponding A, for each E [M]. We note that while we adopt a linear forward model for notational simplicity, our approach can be applied to nonlinear IIPs, as we will demonstrate in our experimental results.
Here, the number of available training images, M, is extremely small (e.g., M < 10), making it infeasible to learn data distribution be used as a prior during inference. Under this under-sampled (or degraded), low-data regime, we consider the following question:
Given a CNN architecture and set of M mcasurement vectors, y: E Y, with corresponding forward operators A,, can we learn set of weights such that, for a unseen test-time measurement vector y $ Y and its operator A, we can reconstruct the image both robustly and efficiently?
Here, robust reconstruction refers to achieving high-quality recovery while being resilient to noise overfitting- an issue commonly encountered in standalone DIP methods [39, 36]*. Efficient recon- struction refers to faster convergence compared to standalone DIP.
Among existing DIP methods, autoencoding Sequential DIP (aSeqDIP) [26] has demonstrated superior performance in terms of robustness and reconstruction quality. Meanwhile, Farly Stopping DIP (ES-DIP) [37] and Deep Random Projector (DRP) [28] have shown improved efficiency in terms of faster run-time.
To enable the learning of transferable weights- weights-which which is the central goal of this paper- we propose the use of a modified network architecture: a shared encoder jointly learned across all M degraded images, along with separate decoder for each individual measurement vector y,. resulting in M disentangled decoders. To mitigate noise overfitting, we adopt an input-adaptive autoencoding objective, which describe in the next subsection. At test time, given unseen vector y. a DIP network, consisting of the learned shared encoder and a randomly initialized decoder, and perform image reconstruction by optimizing only the decoder parameters. In other words, given y test time, the pre- pre-trained encoder is used to extract latent features and the decoder parameters are optimized adapt the unseen measurement vector.
We note that using a single encoder with multiple decoders has also been explored in other areas of machine learning, including multi-task and federated learning [43, 44]. Another example is [45], which proposed a multi-decoder architecture to accelerate the training of diffusion models.
3.1 Proposed training optimization and network architecture
Assume we are given a CNN network (e.g.. a U-Net [25]). Let h R" - RR, parameterized by é. denote the encoder function of the CNN-i.e., the downsampler, consisting of a stack of convolution and pooling layers that extracts and compresses features, with / < n. Similarly, let g: RR' R", parameterized by d. be the decoder function-i.e., the upsampler, comprising stack of transposed convolution and/or interpolation layers that reconstruct the nigh-resolution output from the compressed representation.
Given some input z E IK"" the network output expressed as Iv (zz()). Using this setting with some measurement vector y and its forward operator A, reconstruction via vanilla DIP [15] is formulated as in (2), where denotes the reconstructed image.
b L argmin |Age hz(z) 112
* g (z)
Given set of M degraded measurements of images, y; E Y, each associated with a forward operator A, for i [M], the proposed UGoDIT method utilizes a shared encoder h2 and distinct decoder 9o. for each training image. This results in a total of M disentangled decoders tailored to the individual measurement vector.
Given that we have M measurement vectors, choosing the input is not straightforward across a training set. Hence, we develop an input-adaptive approach. To this end, during training, we propose
Stannalooee DIP refers to the standard setting where single measurement vector is used, and the network is optimized from randomly initialized parameters.

---

Algorithm TTraining:
Unsupervised Group DIP Transferable via weights (UGoDIT)
Input: Number of training images M, measurement vectors y, and forward operators A, for i E [M]. number of input updates K, number of gradient updates N, regularization parameter X, and learning rate B.
Output: Trained encoder weights
Initialization: 2; + A y:: yi 0 Uss ~ (O0, omill for i E [M].
1: For K iterations, Do
2: For N iterations, Do (Shared encoder and M decoders parameters update) V.v.[IAA00 1 - - v Σie(M) |A,g, o ha(z,) ) yell 2 - Alloo. he he(2)) 2:11
3: 1/
v. |A o (z,))) yill2 1112+ + Allgv. (z)))) zz =
Vi E [M]
4:
5: Obtain gv, hz(z)) (Network input update for each training image E [M]
6: Trained weights: á +
optimizing the weights of the shared-encoder, decodeer-disentanlled CNN architecture using the following objective:
5 Sr 2: argmin [E(M RE[M]
Σ
|| o ho(zz) - villž s.t. Ie. o ho((z) Zi vi E [M]} (3)
To solve (3), we initialize o, th ~ (O, o and set z; + A for all i E [M], where Jini corresponds the standard deviation for initialization. We then perform the following alternating updates for K iterations (which corresponds to the number of network inputs updates):
④, 11 ..... thM argmin (.L.EE[M
Σ iE[M]
A ha(z)) y:112 + Allga. ho(2.) 2. 2.1
(4a)
2,
< 9v. o (2,))), Vi E [M]
(4b)
where A a regularization parameter. Similar to aSeqDIP [26], we alternate between two phases: (i) perform N < K gradient updates to optimize the network parameters & and d using the optimization in (4), and (ii) update the network inputs z, via forward passes. Specifically, each input z; is updated by passing it through the shared encoder hs and the corresponding decoder go.
In the training optimization of (4), the first term enforces the output of decoder to be consistent with the measurements vector y; governed by Aç. The second term is an autoencoding term that implicitly enforces the equality constraints (3). It is used mitigate noise overfitting by enforcing that the output of each decoder is approximately consistent with the updated input of that particular image z, as the optimization progresses.
In what follows, we provide a justification of how we relaxed (3) into the alternating optimization updates (4). The general intuition stems from prior studies that highlight the impact of network inputs in DIP-based reconstruction [46, 26, 36], where the study in [26] shows that reconstruction quality improves when the network input is closer to the ground truth.
Assume that the iterates in (4) converge to Φ", V. 2 Vi E [M]. Then, according to the second equation in (4), for continuous mappings Iv. o ha(-), Vi [M], have z lez h&- (z,)
Substituting this into the first equation of (4), in the limit, we get
argmin { Σ ||Agge. hhz(z;) yell2 s.t. 9v. ho(z))) 2,, Vi [M]]
(. aE[M]
which corresponds to the minimizer of (3) given a fixed z For each decoder, the limit points of (4) correspond to the solution of constrained version of vanilla DIP in (2). The constraint enforces additional prior that could alleviate overfitting. While is not straightforward to use first-order tradient-based algorithms for solving (3) given the hard equality constraints, the limit points in the updates of (4) nevertheless minimize (3). Furthermore, the network input is updated by a sequential feed forward process without needing expensive gradient-based updates.
The training procedure of UGoDIT given Algorithm 1. the initialization step, we assume the encoder and decoder weights are of equal size, though this is not necessary requirement. We note
5

---

Truining Phase: Optimization to gel encoder weights @i.
Testing Phase: Optimization to get reconstructed image
I hhpe (z,)
Training
21
y
I4 (+)
Testing Image
9v h2 (2)
htp())
Z A"y
Training image 2
gvp hg (Z2)
Zz
y2
y y
Trainable parameters
Training image
Fixed parameters
Zyz YM
Don ha (ZM)
Dum O
Training set: Y .Ym]
Figure I: Illustrative block diagram UGoDIT. During training (left), we Ieam the shared encoder by optimizing over 6, t.,,..., Par. inference (right), given some unseen measurement vector y and forward operator A. we reconstruct the image & by optimizing over tb
that while steps 2 to 4 describe updating the network parameters using a fixed-step gradient descent, we use the ADAM optimizer [47], initialized with learning rate B.
The computational requirements for training UGoDIT are determined by: (i) the K gradient-based parameter updates, and (ii) the number of function evaluations (network forward passes) necessary for updating 2,, which is K. See Figure (left) for an illustrative block diagram of the training phase.
3.2 Testing phase
Provided with the trained weights of the shared encoder, i.e., parameters Ø from Algorithm and unseen measurement vector y with its forward operator A, at inference, we propose to use new network mapping denoted by he 90 (.), where ø is set to the trained weights &. Then, ub is initialized using the standard way. More specifically, we set y + Φ. and tb (O0 oh I), and perform the following updates for K iterations.
v + argmin [| o hz(z) - y112 + Allgo hg(2) 2|2]
(5a)
= Iv hz(z).
(5b)
In (5), we only optimize over while the parameters of the pre-trained encoder are fixed (i.e., 8)s. Similar to the training algorithm, for every input update (i.e., second equation in (5)), we use N < K gradient steps for the optimization in (5).
The intuition of using the steps in (5) the same as in DIP [26]. However, we emphasize that our goal here is different in UGoDIT, the aim is learn transferable weights for better and accelerated reconstructions at testing time. Algorithm presents the procedure for solving (5). The training and testing phases of UGoDIT illustrated in Figure! 1.
3.3 Intuition behind using M decoders the training of UGoDIT
In this subsection, we describe the rationale behind using M disentangled decoders the UGoDIT training phase and how this design choice influences the quality of the learned encoder weights. Specifically, we aim to answer the following question: Why does learning with multiple decoders result in better shared encoder for test-time reconstruction?
To build this intuition, we first examine the standard architecture comprising a single encoder and single decoder (single denoted by he, 9v. (-)-trained across M degraded images. The parameters are initialized as Φx, v. A/(0, of I) and the inputs as Al Vi E [M]. Then,
"In Appendix F. we conduct study to show the difference between freezing the encoder (our method) and only initializing (as done in STRAINER [40]).

---

Algorithm 2 Testing: Unsupervised Group DIP Transferable via weights (UGoDIT-M)
Input:
Pre-trained encoder weights from Algorithm 1, measurement vector y, forward operator A, number input updates K, number gradient updates N, regularization parameter A, and learning rate 3. Output: Reconstructed image &.
Initialization: +- y (O, Jmll).
1: For K iterations, Do
2: For N iterations, Do (Decoder parameters update)
+ vV ||Ags hz(z) yllă + Allgo o h2(()) z11ž
4: Obtain z <- gv o hag (z). (Network input update)
5:
Reconstructed Image: 2+ hz(2))
we perform the following alternating optimization (which extends [26] to the M >> 1 case).
Φa, 1/e <- argmin 22.0
| o ho, (2:) villă Allgo, (zi) 2;
(6a)
M
2; + gv. o hos (z.), Vi [M]
(6b)
Here, a single decoder Iv, is shared across all training images. While the inputs z, are updated and initialized individually, they are all later optimized using the same encoder-decoder network. The gradient updates for this architecture are:
1/ V.
3V
A o ho, (2:) -yil Allgo, (2.) 22
(7a)
M
Ø, 6. BV
Σ M
A ho, (zi) yi112 Allgo. hog (2,) 2
(7b)
From these expressions, the encoder hg, is trained with using back-propagatee updates from single decoder. In contrast, in UGoDIT, the encoder receives supervision from M distinct decoders, each independently optimized reconstruct different training image (steps 3 and 4 in Algorithm 1). This over-parameeerization allows each decoder to specialize in reconstructing its corresponding input, thereby improving the enforcement of measurement consistency across the training
Moreover, because the encoder must support multiple decoders reconstructing different training samples, it is encouraged to extract more generalizable and robust features. This leads to the intuition: encoders that yield more accurate reconstructions during training tend to reconstruct better during testing. We empirically support this claim in Section 4 by comparing both training and test-time reconstructions using shared decoder versus disentangled decoders (our method).
Next, we offer justification from the literature of multi-task representation learning. In our setting, each degraded image corresponds to an image-specific inverse problem, which can be interpreted as separate task.
Our training setup aligns with the multi-task learning framework studied in [48], where a shared representation (encoder h&) is learned jointly across multiple tasks, each with its own function (i.e., task-speeific decoder gy: for i [MM). Specifically, Theorem in [48] demonstrates that this structure (i) reduces generalization error across tasks and (ii) promotes robust and transferable feature learning in the shared encoder. Importantly, these theoretical results rely on the structural assumption of a shared encoder with task-specifie decoders. In contrast, the single ED setup (as in Eq. (6)) violates this assumption, since the decoder receives gradients from multiple tasks simultaneously. This prevents specialization and introduces gradient interference learning [49]. Consequently, the theoretical benefits shown in [48] do not extend the single ED setup which serves as additional motivation of our proposed multi-decoder method.
known challenge in multi-task
7

---

Experimental results
4.1 Settings
We consider three tasks: MRI reconstruction from undersampled measurements, super resolution (SR), and non-linear deblurring (NDB). For MRI, we use the knee portion of the fastMRI dataset [22]. The forward model is y = Ax'. The multi-coil data is obtained using 15 coils and is cropped to a resolution of 320 320 pixels. To simulate undersampling of the MRI k-space, we use a Cartesian sampling pattern with 4x and 8x acceleration factors (AF). Sensitivity maps for the coils are obtained using the BART toolbox [50]. For the tasks of SR and NDB, we use the FFHQ dataset [51]. For all three tasks, we test UGoDIT and baselines with 20 randomly selected degraded images". Backbone architectures are adopted from [25] and [52] for MRI and SR (and NDB), respectively. For evaluation metrics, we use the Peak Signal to Noise Ratio (PSNR), the Structural SIMilarity (SSIM) index [53], and run-time, For natural images, we also report LPIPS [54]. All the experiments are run on a single RTX5000 GPU machine.
DIP--bsed methods, we use three regularization brtiiizaatio--adeddddde recent methods aSeqDIP [26], Self-Guided DIP [36], and Deep Random Projector [28] (as a DIP acceleration method), and one early stopping (ES) method, ES-DIP [37]. For data-driven MRI baselines, we use two recent DM-based methods, SITCOM-MRI [14] and DDS [12], one self-supervised method, Test Time Training (TTT) [55], and two supervised models, VarNet [18] and Recurrent VarNet [56]. For natural images, we also use SITCOM [14], in addition to DPS [29] and STRAINER [40]. STRAINER is a representative of low-data regime methods. The selection criteria of these baselines depend on methods' ability to achieve competitive reconstruction quality and high robustness to noise overfitting (for the case of DIP) for which several medical and natural image recovery tasks were considered (linear and non-linear). Further implementation details provided in Appendix O.
For training-datee
For UGoDIT, we present results with three models, trained with 4, 5, and 6 degraded images (see Appendix L for results with M E {2, 3, 10, 10)). We test UGoDIT and other data-driven methods with in-distribution test instances, i.e., images from the testing sets of fastMRI and FFHQ Out-of- distribution evaluation is provided in Appendix A. Ablation studies on the number of layers and the number of training images are given in Appendix J and Appendix I., respectively. Step size is set to 8 0.0001. The regularization parameter is set to À = 2, following the study in Appendix G (where we also show the robustness to noise overfitting). For (N, K), we use (2, 2000), (10, 2000), and (10, 2000) for MRI, SR, and NDB, respectively, following the study in Appendix M.
4.2 Impact of using M decoders in UGoDIT
Training and Testing Reconstructions
Here, we examine the effect of the multi-decoder archi- tecture proposed in the training of UGoDIT. Specifically, we assess how using M > 1 disentangled decoders influences the quality of the shared encoder's 's learned weights during both training and testing for the super resolution task. To this end, we compare our method against baseline where M 6 degraded are trained using a standard single- -nnllee single-decoder architec- ture, as described in (6). At inference, both approaches follow the procedure in (5) using their respective trained encoders.
32
30-
28
26
24
Train: Multi-decoder (Ours)
Test: Multi-decoder (Ours) Train: shared decoder
Test: shared decoder
1500
2000
Figure 2 shows the average PSNR curves during train- ing (dashed) and testing (solid with 20 test images) of UGoDIT and the shared-decoder case. As seen, employ- ing multiple decoders yields approximately over a 1 dB to 1.5 dB improvement in both training and test- ing. These findings empirically support the rationale discussed in Section 3.3.
Iterations
Figure 2: Averaged PSNR curves (6 for train- ing and 20 for testing) of UGoDIT vs. the -ecoddr case (i.e., single E-D (6)) on the task of SR. See Appendix H for an addi- tional study that highlights the benefit of the shared encoder at testing.
'In Appendix B, we show the impact of the selection of the training on the performance of UGoDIT.
8

---

MRI Reconstruction
Super Resolution
Non-linear Debluning
30-
22.5
25-
25
30.0
20
15
UGaBMi-S [0urs)
UGaBIIF [Ouns]
UGGbiTT4 (Duml
UG:DfT (0urs)
US:DIT (Ours)
UG:DIT (Ours)
SSeaDIP
aStDDIP
UG:DITT (Ours)
20.0
Self-Gudded DIP
Self Gurded
aSeoDIP
1000
3000
1000
1500
2000
500
1000
1500
2000
2500
3000
Perattons
Iterations
Iterations
Figure 3: PSNR curves of UGoDIT-AN M vs. training- daangg-ddt-feeee free DIP- based baselines for the tasks of MRI (leff,, SR (middle), and NDB (right) averaged over 20 test images. Iterations (X- (x-axis) correspond to NK.
Data-intensive
Data-less
Degraded Image
PDS
Self-Guided DIP
SSeDDIP
UGoDIT
Mult-cool
MRI
PSNR 14:33 de SITCOM
GT
Degraded Image
Self-Guided DIP
aSeqDIP
UGoDIT
Super Resolution
PSNR 32.15
Degraded
SelPGuided
UGoDIT
Non- hneer Deblurring
PSNR $2.45
Figure 4: Reconstructed/reooeeeed images using UGoDIT (last column with M = 6) and baselines (columns to 5). As observed, UGoDIT return comparable PSNR results with data-intensive methods all without the need of large amount of clean images (or fully sampled measurement vectors). See Appendix P for more visualizations.
we observe gap of approximately to 4 dB between the training and testing curves. We hypothesize that this arises because, during training, the network parameters are jointly optimized to reconstruct = images, whereas the testing curve reflects the average performance over 20 separate inference- time optimizations. Appendix provides additional study the learned features of UGoDIT.
Furthermore,
4.3 Convergence of UGoDIT compared to training-data-fre DIP-based methods
Here, we aim to answer the question: Given a set of degraded test images, does using UGoDIT offer any advantages
running DIP-based method independently each image?" Figure presents the average PSNR curves, comparing UGoDIT (trained with 4, 5, and 6 degraded images) with four recent SOTA DIP baseline methods.
In terms of PSNR improvements, UGoDIT achieves dB or slightly less-compared to aSeqDIP, but
a dB gain over aSeqDIP for MRI. For
SR, we observe marginal improvement-aroumee more than dB over DRP and Self-Guided DIP. For NDB, UGoDIT consistently outperforms the second-best method (aSeqDIP) by at least dB. Regarding acceleration, UGoDIT demonstrates clear advantage all three tasks, as measured by the number iterations required to reach target PSNR values. For instance, in MRI, UGoDIT reaches 32.5 dB in 500 iterations fewer, whereas aSeqDIP requires over 750 iterations to achieve the same PSNR. This trend also holds for SR and NDB. For example, in SR, UGoDIT-6 requires only about 125 steps to reach 30 dB, while aSe needs
comparison with training -ata- data-free measurement splitting methods given in Appendix D.

---

Method
Training imuges M
Type
PSNR (?)
Acceleration Factor (AF).
SSIM (?)
run-time (L)
PSNR (?)
Acceleration (AF): 8x run run-time (1)
SSIM (?)
VarNer [18]
Fully sampled
33,06
5.02a112
32.05
5.45.211
Recument [56]
8000
Pully sampled
35.88
48
12.33
a.879
4.55
TTT [55]
300
Fully sampled
33.81
=0
31.83
0.879amu
48,25
DDS [12]
19480
Fully sumpled
34.95-0m4
0.954
25.20-112
31.72=155
0.876
31.02+2mm
SITCOM-MRI [14]
19460
Pully sampled
36.33gem
0.962
26.24
32.78g441
0.892.o0t
82,45
UC (Ours)
Sub-sampled
35,45.004
p.957
45 20
32 32.04.0445
1.87
47.42.55
UGODIT (Ours)
Sub-sampled
35.74:041
0.959
48. 48.12::54
32.36
0.881 soo
48.12:1324
1CCOTTT (Ours)
Sub-sampled
36.05.g844
0.960
48 18.42.55 1
32.54 15
0.887.n
49.532774
Table 1: MRI reconstruction average results (with run-time in seconds) across two AFs of UGoDIT vs. data-
centric methods. In SITCOM-MRI and DDS, the DM is trained on 973 MRI volumes each with 20 images.
Column 3 corresponds to the "Type" of the training images. Values past + represent the standard deviation.
Method
Training images 3f
Type
(*)
Super Resolution (SR)
LPIPS (1)
time (4)
PSNR (1)
Non-lincar Deblurring (NDB)
(1)
LPIPS (L)
cun (L)
DPS [20]
4X000 40000
Clean
24.44g
2000 2000
0.26gaa 0.142gams
75.60g14 3o
23.42g215
0.757
0.279
93.00
SITCOM [14]
Clean
10
27.00
30.12g0ua
0.903ga
0. 0.145
33.45
STRAINER
Clean
29.034139
18189
32.12a5
27.
0.12a60
33,40
STRAINER [40]
Clean
29.244c89
0.846115
33.45
27.39
0.874624
0.1
02.0011>
STRAINER (40)
Clean
29.45
a.8
0.177
31.12
27.41
0.819
0.19
30.50
UGa01T (Ous)
4
Degaddd
50.55grre
20 20.25
29 29.89
0. 0.
28 28.88
VG.0IT (Ours)
Degraded
30.62
30.15
30.09
0.899
0. 05
29 29.29
UGoDIT (Ouns)
Degraded
30.75axes
0.1
30.41as
30.19
0.906
30.18
Table 2: Restoration results of UGoDIT vs. data-centric methods. Column 3 corresponds to the "Type" of the training images. Following [29, 14], we use Sy, = 0.05, V E [M] and M E 5,,5, UGoDIT uses the
degraded versions of the clean training images used for STRAINER. Values past + represent standard deviation.
more than 250, and DRP nearly 600. Another noteworthy observation is that increasing the number of training images used by UGoDIT generally leads to better reconstruction quality. Collectively, these results highlight that UGoDIT provides improvements in both reconstruction performance and convergence speed. We refer the reader Appendix E for more insights and additional results about convergence speed and training (and testing) run-times.
4.4 Comparison with data-centric methods
Here, we compare UGoDIT with data-driven methods. Table 1 (resp. Table 2) presents the results for MRI (resp. SR and NDB). For MRT, in terms of PSNR, all three variants of UGoDIT outperform DDS, TTT, and VarNet. Only UGoDIT-6 outperforms Recurrent VarNet (a more advanced supervised model). As expected, UGoDIT is slower than supervised methods (VarNet and Recurrent VarNet). Compared to SITCOM-MRI, UGoDIT achieves slightly lower PSNR but is faster in terms of run-time.
For SR and NDB, UGoDIT achieves comparable PSNR performance to SITCOM. Compared to STRAINER, UGoDIT provides improvements of over 1 dB for SR and 2 dB for NDB, given similar run- times iimes-importantlyy without requiring clean images for training. Overall, these results demonstrate that UGoDIT can, under the considered settings, achieve competitive performance with SOTA data- driven methods, all without relying on large quantities of fully sampled (i.c., clean) training images. See also the visualizations in Figure
5 Conclusion
This paper introduced UGoDIT, a fully unsupervised framework for solving inverse imaging problems in the low-data regime, where only a very small number of degraded images are available during training. By learning a shared encoder with transferable weights across multiple decoders, UGoDIT effectively bridges the gap between the recent SOTA data-intensive generative models and training- data-fre free deep mage prior (DIP) methods. Unlike DIP approaches that optimize a separate network per image, UGoDIT leverages the shared encoder to accelerate convergence and improve reconstruction quality. Empirical evaluations across MRI, super-resolution, and non-linear deblurring tasks show that, under the considered settings, UGoDIT achieves results comparable to SOTA supervised and iiffussio-bbsed methods, all without requiring access to large number of clean ground-truth images. These results highlight the potential of UGoDIT as a data-efficient solution for inverse problems. Limitations and future works are discussed in Appendix N.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by the National Science Foundation (NSF) grants CCF-2212065, CCF- 2212066 and the CAREER Award CCF-2442240.
References
[1] Dinesh Manocha and John F Canny, Efficient inverse kinematics for general 6r manipulators. IEEE Transactions on robotics and automation, 10(5):648-65)) 2002.
[2] Shaofei Ren, Guorong Chen, Tiange Li, Qijun Chen, and Shaofan Li. A deep learning- based computational algorithm for identifying damage load condition: an artificial intelligence inverse problem solution for failure analysis. Computer Modeling Engineering & Sciences, 117(3):287-307, 2018.
[3] Aviad Levis, Pratul Srinivasan, Andrew A Chael, Ren Ng. and Katherine L Bouman, Gravita- tionally lensed black hole emission tomography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19841-19850, 2022.
[4] Noori BniLam and Rafid Al Al-Khoury. Parameter identification algorithm for ground source heat pump systems. Applied energy, 264:114712, 2020.
[5] Christian Ledig, Lucas Theis, Ferene Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. reato-rea single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages
2017.
[6] B. Goyal, A. Dogra, S. Agrawal, B.S. Sohi, and A. Sharma. Image denoising review: From classical to state-of-the-arr approaches. Information Fusion, 55:220-244, 2020.
[7] Phong Tran, Anh Tuan Tran, Quynh Phung, and Minh Hoai. Explore image deblurring via encoded blur kernel space. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 11956-11965, 2021.
[8] O. Elharrouss, N. Almaadeed, S. Al-Maadeed, and Y. Akbari. Image inpainting: A review. Neural Processing Letters, 51 11:000202222 2020.
[9] J. A. Fessler. Model-Based Image Reconstruction for MRL. IEEE Signal Processing Magazine, 27(4):81-89, 2010.
[10] Michael T McCann, Kyong Hwan Jin, and Michael Unser. Convolutional neural networks for inverse problems in imaging: review. IEEE Signal Processing Magazine, 34(6):85-95, 2017.
[11] Ismail Alkhouri, Shijun Liang, Rongrong Wang, Qing Qu, and Saiprasad Ravishankar. Diffusion- based adversarial purification for robust deep mri reconstruction. In IEEE International Con- ference Acoustics, Speech and Signal Processing (ICASSP), pages 12841-111 12845.. IEEE, 2024.
[12] Hyungjin Chung, Suhyeon Lee, and Jong Chul Ye. Decomposed diffusion sampler for acceler- ating large-scale inverse problems. In International Conference on Learning Representations, 2024.
[13] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated MRI. Medical image analysis, 80:102479, 2022.
[14] Ismail Alkhouri, Shijun Liang, Cheng- Han Huang, Jimmy Dai, Qing Qu, Saiprasad Ravishankar, and Rongrong Wang. Sitcom: Step-wise triple-consistent diffusion sampling for inverse problems. International Conference on Machine Learning, 2025.
[15] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 9446-9454, 2018.
11

---

[16] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraraghavan, and Richard Baraniuk. Wire: Wavelet implicit neural representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18507-18516. 2023.
[17] Burhaneddin Yaman. Zero-shot self-supervised learning for MRI reconstruction. In Interna- lional Conference on Learning Representations, 2022.
[18] Anuroop Sriram, Jure Zbontar, Tullie Murrell, Aaron Defazio, C Lawrence Zitnick, Nafissa Yakubova, Florian Knoll, and Patricia Johnson. End-to-end variational networks for accelerated MRI reconstruction. In Medical Image Computing and Computer Assisted Intervention-MICCA. 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part II 23, pages 64-73. Springer, 2020.
[19] H. K. Aggarwal, M. P. Mani, and M. Jacob. Modl: Model-based deep learning architecture for inverse problems. IEEE Transactions on Medical Imaging, 38(2):394-405. 2018.
[20] Mehmet Akçakaya, Burhaneddin Yaman, Hyungjin Chung, and Jong Chul Ye. Unsupervised deep learning methods for biological image reconstruction and enhancement: An overview from a signal processing perspective. IEEE Signal Processing Magazine, (922):2844,, 2022.
[21] Elisabeth R Garwood, Michael P Recht, and Lawrence M White. Advanced imaging techniques in the knee: benefits and limitations of new rapid acquisition strategies for routine knee MRI. American Journal of Roentgenology, 209(3):552-560, 2017.
[22] J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, al. fastMRI: An open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv: 11////////0. 1.08839, 2018.
[23] Akshay S Chaudhari, Feliks Kogan, Valentina Pedoia, Sharmila Majumdar, Garry E Gold, and Brian A Hargreaves. Rapid knee MRI acquisition and analysis techniques for imaging osteoarthritis. Journal of Magnetic Resonance Imaging, 121511333))333) 2020.
[24] Muhammad Shafique, Sizhuo Liu, Philip Schniter, and Rizwan Ahmad. MRI recovery with self-calibrated denoisers without fully-sampled data. Magnetic Resonance Materials Physics, Biology and Medicine, 38:53-66, 2025.
[25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted interventionnMMICC.. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[26] Ismail R. Alkhouri, Shijun Liang, Evan Bell, Qing Qu, Rongrong Wang, and Saiprasad Ravis- hankar. Image reconstruction via autoencoding sequential deep image prior. In Advances in Neural Information Processing Systems, volume 37, pages 18988-19012, 2024.
[27] Kevin Zhang. Mingyang Xie, Maharshi Gor, Yi-Ting Chen, Yvonne Zhou, and Christopher A. Metzler. Metadip: Accelerating deep image prior with meta learning, 2022.
[28] Taihui Li, Hengkang Wang, Zhong Zhuang, and Ju Sun. Deep random projector: Accelerated deep image prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18176-1 18176-181888 18185, 2023.
[29] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Mare Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh Interna- tional Conference on Learning Representations, 2023.
[30] Giannis Daras, Hyungjin Chung, Chieh-Hsin Lai, Yuki Mitsufuji, Peyman Milanfar, Alexan- dros G. Dimakis, Chul Ye, and Mauricio Delbracio. survey on diffusion models for inverse problems. 2024.
[31] Asad Aali, Giannis Daras, Brett Levac, Sidharth Kumar, Alexandros G Dimakis, and Jonathan I Tamir, Ambient diffusion posterior sampling: Solving inverse problems with diffusion models trained on corrupted data. International Conference Learning Representations, 2025.
12

---

[32] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Mika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference Machine Learning, pages 2965-2974. PMLR, 2018.
[33] Zhihao Xia and Ayan Chakrabarti. Training image estimators without image ground truth. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[34] Jiaming Liu, Yu Sun, Cihat Eldeniz, Weijie Gan, Hongyu An, and Ulugbek S. Kamilov. Rare: Image reconstruction using deep priors learned without groundtruth. IEEE Journal of Selected 2020.
Topics in Signal Processing, 14(6):1088-1000
[35] Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. RecorupteedAo--rcorppeedd Unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pages 2043-2052, 2021.
[36] Shijun Liang. Evan Bell, Qing Qu, Rongrong Wang, and Saiprasad Ravishankar. Analysis of deep image prior and exploiting self-guidance for image reconstruction. IEEE Transactions on Computational Imaging, 1 11:435-451, 2025.
[37] Hengkang Wang. Taihui Li, Zhong Zhuang, Tiancong Chen, Hengyue Liang, and Ju Sun. Early stopping for deep image prior. Transactions Machine Learning Research, 2023.
[38] R Heckel et al. Deep decoder: Concise image representations from untrained non-convolutional networks. International Conference on Learning Representations, 2019.
[39] Ismail Alkhouri, Evan Bell, Avrajit Ghosh, Shijun Liang, Rongrong Wang, and Saiprasad Ravishankar. Understanding untrained deep models for inverse problems: Algorithms and theory. JEEE Signal Processing Magazine, 2025.
[40] Kushal Kardam Vyas, Imtiaz Humayun, Aniket Dashpute, Richard Baraniuk, Ashok Veeraragha- van, and Guha Balakrishnan. Learning transferable features for implicit neural representations. Advances in Neural Information Processing Systems, 37:42268 42291, 2024.
[41] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
[42] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 193995948, 2019.
[43] Deblina Bhattacharjee, Tong Zhang, Sabine Süsstrrukk, and Mathieu Salzmann. Mult: An end-to-end multitask learning transformer. Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 12031-12041 2022.
[44] Jingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, and Yaohong Zhang. Multi-task federated learning with mncoder-decodee structure: Enabling collaborative learning across different tasks. arXiv preprint arXiv:2504. 250.0:000000000000 2025.
[45] Huijie Zhang, Yifu Lu, Ismail Alkhouri, Saiprasad Ravishankar, Dogyoon Song, and Qing Qu Improving training efficiency of diffusion models via multi-stage framework and tailored multi-decoder architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7372-7381, June 2024.
[46] Di Zhao, Feng Zhao, and Yongjin Gan. Reference-driven compressed sensing mr image recon- struction using deep convolutional neural networks without pre-training. Sensors, 20(1):308, 2020.
[47] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6940 2014.
[48] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning, Journal of Machine Learning Research, 17(11)1133,, 2016.
13

---

[49] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824-558 2020.
[50] J. I. Tamir, F. Ong, J. Y. Cheng, M. Uecker, and M. Lustig. Generalized magnetic resonance image reconstruction using the berkeley advanced reconstruction toolbox. In ISMRM Workshop on Data Sampling & Image Reconstruction, Sedona, AZ, 2016.
[51] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
[52] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 483 4999,99..0 Springer, 2016.
[53] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions Image Processing, (3):6600 612, 2004.
[54] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 586-595, 2018.
[55] Mohammad Zalbagi Darestani, Jiayu Liu, and Reinhard Heckel. Test-time training can close the natural distribution shift performance gap in deep learning based compressed sensing. In International Conference on Machine Learning, pages 4754-4776, PMLR, 2022.
[56) George Yiasemis, Jan-Jakob Sonke, Clarisa Sánchez, and Jonas Teuwen. Recurrent variational network: a deep learning inverse problem solver applied to the task of accelerated MRI re- construction. In Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, pages 732-741, 2022.. 2022.
[57] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on Computer Vision and Pattern Recognition, pages 248-255. Icce, 2009.
[58) Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kåmil Ugurbil, and Mehmet Akçakaya. Self-supervised learning of physics-guided reconstruction neu- ral networks without fully sampled reference data. Magnetic resonance in medicine, 84(6):1172- 3191, 2020.
[59] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2023.
14

---

Appendix
In the Appendix, first present experiments evaluate the out-of-distribttion performance of UGoDIT (Appendix A). Appendix presents a study that shows the impact of the selection of the training set. A study training UGoDIT with clean images is shown in Appendix C. Appendix D presents comparison results with measurement--plitting methods, followed by additional run-time experiments provided Appendix E. Appendix F examines the impact of freezing the encoder weights at test time, followed by study on robustness to noise overfitting (Appendix G). A study on reconstructing the test image during the training UGoDIT is given in Appendix H. Appendix I investigates the learned features in an MRI-trained UGoDIT model. This is followed by ablation studies on the number of convolutional layers (Appendix J), the number of frozen layers in the encoder of UGoDIT (Appendix K), the number of training images M (Appendix L). and the choice of N and K (Appendix M). Limitations and future directions are discussed in Appendix N. Appendix provides implementation details of the baselines, followed by set of figures for additional visualizations (Appendix P).
A Out-of-distributtio evaluation
A.1
MRI
In this section, investigate the -uttof-diitribution (OOD) capabilities of UGoDIT for the task of MRI. Specifically, we evaluate the encoder of UGoDIT-6 (which we will refer to as UGoDIT-OOD) trained knee scans with 20 brain MRI test scans randomly selected from the fastMRI brain dataset.
30
28
26
24
optimizing only
unee trained encoder with aptimizing only secoder IUGoDIT 000 with à ized aptimizing decoder, A encoder sptimbiing encoder eccoder,
20000
Berations
Figure 5: Average PSNR of 20 MRI brain test using knee-trained (with M UGODIT-OOD (blue) vs. case where test-time, we use the parameters the pre-trained encoder to only initialize the test-time encoder (red). Then, we optimize over both the encoder and decoder. Running aSeqDIP independently is also included (black). Iterations in the x x-axis correspond to K 30000.
Ground
31.51 de
27.50
30.54 dB
30.22 dB
32.94
Figure 6:
Brain reconstructed images for the OOD experiment. The titles of the third fifth reconstructed images correspond to the dashed blue (UGoDIT-QOD with > 0), solid blue (UGoDIT-OOD) with À 2). dashed red, and solid red curves, respectively, in the legend Figure PSNR values are given on the top of each reconstructed image.
The average PSNR curves shown in Figure 5. As observed, when compared to aSeqDDP-whiin run independently from scratch on each test sean (black curve) the encoder trained on knee images under-performs and converges very slowly brain scans.
hypothesize that the autoencoding term and the input updates in UGoDIT which implicitly enforce consistency between encoder-decoder input/output pairs-lcad lcad to the learning of features
15

---

that are overly tailored to the training distribution. As result, when the test distribution differs significantly, these learned features make the convergence slower.
More specifically, when training and test images come from the same distribution (i.e., anatomy), the input updates and autoencoding term help reconstruction, they promote alignment of shared low-frequency structures. However, under distribution shift (i.e., when the training and testing anatomies different), these mechanisms degrade performance, since training and test images no longer share similar low-frequency content.
To support this hypothesis, Figure 5 includes two additional cases: (i) using the knee-trained encoder while optimizing only the decoder and setting À = 0 (i.e., no autoencoding regularization the first term of Eq. (5); shown the dashed blue curve); and (it) initializing the encoder with knee-trained weights and optimizing both the encoder and decoder, with À E 0, (shown in red curves).
As observed, when À 0 (i.e., dashed curves), the results could be better than when À & 0 as the input will have less impact. Furthermore, when we compare UGoDIT-OOI with only initializing the encoder (the red curves), the results are further improved which indicate that pre-trained weights knees have less impact when only initialization is used. This even more evident in the case when A = with initialization only (i.e., red dashed curve).
The visualizations in Figure (which are recorded at iteration 30000) show that the all the considered cases would converge to between 27.58 dB to 31.51 dB.
A.2 Super Resolution
In this section, investigate the OOD capabilities of UGoDIT natural images for the task of super resolution (SR). To this end, we test the encoder of UGoDIT (trained with images from the FFHQ dataset) with 20 degraded images randomly selected from the ImageNet dataset [57].
25
PSNR
15
aSeqDIP
UGoDIT
0
250
500
750
1000
1250
1500
1750
2000
Epoch
Figure 7: Average PSNR of 20 test images from the ImageNet dataset of UGoDIT trained with M 6 FFHQ degraded images. Iterations the x-axis correspond to NK 20000.
For SR, we observe that UGoDIT achieves slightly better PSNR than aSeqDIP and eventually converges similar PSNR values. However, unlike the MRI case, UGoDIT-OOD converges faster than aSeqDIP. This can also be observed in the restored images of Figure and Figure 9, We attribute UGoDIT ability to generalize well to learning multi-frequency features under OOD scenario for the natural image.
In Table 3. report test-time average PSNR 20 CBSD68 images using two UGoDIT-6 models. The first model is trained using images from the FFHQ dataset, and the second is trained using images from the CBSD68 dataset. The results demonstrate that UGoDIT maintains strong performance across different datasets for the task of SR.
B Impact of the selection of the training set in UGoDIT
In this section, we provide an ablation study demonstrate the impact of the selection of the M training images for the task of SR.
brtpo:/ /ttp///..tt..///////// wttp://.w....c..ccc........
16

---

Ground Truth
Input
aSeq DIP
UGoDIT
PSNR dB
PSNR dB
PSNR 29.78 dB
PSNR 29.95 dB
Figure 8: Ground-truth, degraded, and restored images for the OOD experiment on the task of super resolution. where UGoDIT was trained on 6 images of faces from the FFHQ datasct and tested on the ImageNet degraded image in the second column. As observed, the testing set is not human face as is the case for the training images FFHQ
Ground Truth
Input
aSeq DIP
UGoDIT
PSNR dB
PSNR 19.32 dB
PSNR 24.38 dB
PSNR 24.89 dB
Figure 9: ground-truth. degraded, and restored images for the OOD experiment on the task of super resolution. where UGoDIT trained images of faces from the FFHQ dataset and tested on the ImageNet degraded image the second column. As observed, the testing set human face as is the case for the training images FFHQ.
To this end, we evaluate six models of UGoDIT-6, each trained with distinct set of randomly selected training images Y,, i.e., for any two i and j, we have Y; O Y 0. In Table 4, we report average test-time PSNR over 100 test images.
As observed, the averaged PSNR of our method ranges between 30.58 dB to 30.95 JB, i.c. the variability using different small training sets is very small (less than 0.4 dB). This indicates that the in-distribution results of UGoDIT are highly sensitive to the selection of training images for the task of SR.
C Comparison of UGoDIT-M vs. training on M clean images
Here, we attempt to answer the question: Does the test-time performance change if M clean images are used train the shared encoder? Put differently, would training with M clean images represent UGoDIT's best-case learning?
If clean images available, we first need modify the training objective since there would no longer be forward operators A; or any inverse problems during training. To this end, the optimization reduces pure multi-decoder autoencoding loss, where the minimization in (4) becomes:
Φ, V Pi..... tMM
argmin
Σ M
l1gs, ho(2)) 2
where z, x,, for i E [M]. are the clean images.
To test this idea, we compare UGoDIT trained on six degraded images vs. the multi-decoder autoen- coder trained (i.e., (8)) on clean versions of UGoDIT's degraded images for the SR task.
In Table 5, we report average test-time PSNR over 100 test images.
As observed, test-time PSNR for the auto-encoder with clean data is slightly better than UGoDIT, This means that training clean images could also serve prior for test-time reconstructions, The
17

---

Method
Evaluation Type
Average PSNR over 20 CBSD68 test images
FFHQ-trained UGoDIT-6
OOD
26.98
CBSD68-trained UCoDIT-6
In-distribution (InD)
27.13
Table 3: Average lest-lime PSNR 20 CBSD68 test images using UGoDIT-6 models. The first (top row) model trained using 6 images from FFHQ dataset (which represents OOD case), second (bottom row) is trained using images from the CBSD68 dataset (which represents the InD case).
Training set for UGoDIT-6
Average PSNR over 100 FFHQ test images
30,64 0.24
30.57.233
Y;
20.65=027
Y4
30.58-022
Y/
30.74=0.66
Ye
10.95
Table 4: Average test-time PSNR over FFHQ test images task SR (with 4x) using UGoDIT-6 models, each trained with distinct training set Y,, E [6].
decoder clean training is initialized at random and then optimized to reconstruct the test image. However, we emphasize that UGoDIT is designed specifically for the low-data regime without clean images.
D Comparison with Wraining-dtta-fe self-supervised measurement--slittngg methods
Here, we compare UGoDIT-6 with measurement-slitee training-dataa free self-supervised meth- ods: Self-supervised learning via data under-sampling (SSDU) [58] and Zero-Shot Self- Solf--Sueeviied Learning (ZS-SSL) [17], for the task of MRI reconstruction.
Table 6 reports the average test-time PSNR over 20 test images using two knee datasets: fastMRI [22] and the Cor-PD dataset [59].
As observed, under the considered settings, our method returns the best results as compared to these baselines.
E Additional insights and results on convergence speed and run-time
In this section, we provide two sets of results in regard to the convergence speed and run-time (training and testing) of UGoDIT as compared to the SOTA DIP-based method, aScqDIP [26].
First, order to highlight the convergence speed advantage UGoDIT (relative to aSeqDIP), in Table 7, report the number of optimization iterations needed achieve 30 dB across three tasks, obtained from Figure 3. We note that, at inference, the number of parameters to optimize in UGoDIT is lower than aIPqDIP the encoder in our case is fixed.
As observed, UGoDIT requires significantly lower number of iterations to achieve 30 dB MRI, while requiring less than half of the number of iterations for SR and NDB.
In the second part of this section, Table the SR task, and compare it with aSeqDIP. All methods were run for 4000 iterations.
report the training time, testing time, and FLOPs for
As observed, the more training images, the higher run-time and FLOPS for UGoDIT decoders to train when M increases. As for testing, we only require less than half the run- run-time as aSeqDIP since the optimization in UGoDIT takes place w.r.t. only the decoder, whereas DII optimizes the entire network.
have more
Moreover, our run-time and FLOPs are significantly lower than training diffusion models or supervised methods (data-intensive methods). For example, as per Figure of [45], diffusion models require more than 17 GFLOPs to train.
18

---

Method
Training images
Average PSNR over 100 FFHQ test images
UGcDIT
6 degraded images
30.75
Multi-decoderr AB (clean training) (y)
5 clean images
31 31.02200r
Table 5: Average test-time PSNR over 100 FFHQ test images of UGoDIT-6 model multi-decoder AE trained using (8) with the clean versions of training images of UGGDIT.
Dataset
Method
Average PSNR 20 test slices
SSDU
33.25
fastMRI
ZS-SSL
33.47
UGoDIT-6
36.07
ZS-SSL
39.55
Cor-PD
UGoDIT-6
39.98
Table 6: Average test-time PSNR over 20 MRT test slices of UGoDIT-6 vs. measurement slitring methods using the fastMRI and Cor-PD datasets (both with ID uniform mask).
F Impact of freezing the test-time encoder in UGoDIT
Here, we show the impact of freezing the weights of the pre-trained encoder at test time. More specifically, given pre-trained encoder with weight é, we compare UGoDIT at test time with an alternative approach that initializes the test-time encoder with á and then optimizes both the encoder and decoder. We note that this is a similar setting to how the pre-trained MLP weights were used in STRAINER [40].
37
36
35
(UGoD1))
34
2000
8000
Figure 10: Average PSNR of 20 MRI test scans of UGoDIT-6 vs. the case where we use the parameters of the pre-trained encoder only initialize the test-time encoder. Then, the optimization takes place both the encoder decoder. Iterations x-axis correspond to N K 10000. both cases, we use
Figure 10 presents the average PSNR over 20 MRI scans. As shown, freezing the encoder weights yields better results in terms of both PSNR (around 1dB difference) and convergence speed. Specifi- cally, UGoDIT approaches the peak performance in approximately 2000 iterations. Moreover, UGoDIT requires optimizing fewer test-time parameters compared the approach that initializes the encoder and optimizes both the encoder and decoder.
G Robustness to noise overfitting and the impact of the regularization parameter X
In this section, we demonstrate how UGoDIT is robust to noise overfitting, typically encountered in DiP-based methods. To this end, for the task of MRI reconstruction, we train UGoDIT with M knee scans across different values of the regularization parameter: À E (0.1, 1, 10). During testing, we use 20 scans.
Figure 11 presents the average PSNR during testing. As observed, even when we run our method for 10000 iterations, the PSNR drops by approximately less than 1.5 dB. This indicates that noise overfitting takes place but minimal given the large number of iterations.
Additionally, the case of À 2 returns the highest PSNR and has the least noise overfitting (as indicated by the drop PSNR). We also observe that the difference in PSNR across different X's is approximately 1 dB. This indicates that the performance of UGoDIT is not highly sensitive to the choice of the regularization parameter. In other words, if different values of À are used (between 0.1 and 10), the PSNR value does not change significantly.
19

---

Number of iterations needed to achieve 30 dB
Method
MRI
SR
NDB
aSeqDIP
450
325
1750
UGoDIT-6
10
120
650
Table 7: Number iterations needed to achieve 30 dB using UGoDIT-6 and aScqDIP across the three considered tasks for the results in Figure 3.
Method
Run-time (seconds)
GFLOPs
aSeqDIP
142
0.01004
UGoDIT-4 (training)
384
0,0143
UGoDIT-5 (training)
415
0.0142
UGoDIT-6 (training)
472
0.0174
UGoDIT-4 (testing)
62
0.0062
UGoDIT-5 (testing)
64
0.0078
UGoDIT-6 (testing)
66
0.0084
Table 8: Training and testing run-time and FLOPs of different UGoDIT models.
H A study on reconstructing the test image during the training of UGoDIT
To further highlight the benefit of the fixed encoder (and decoder-only optimization) testing, in this section, we report the results Table 9 (with M = 5) where, given the same M 1 FFHQ images, consider:
1. The PSNR of image M using pre-trained UGoDIT-M encoder trained image 1 to image M (first row). In other words, the PSNR of image M + using our test-time optimization in (5) (i.e., Algorithm 2).
2. The PSNR of image M + when training UGoDIT-M + 1 jointly on all + images (second row). In other words, the PSNR at decoder M + 1 using training optimization in (4) (i.e., Algorithm 1).
Method
PSNR for image M + 1
Pre-trained UGoDIT-M encoder with test-time decoder
29. 12
UGoDIT-M + training
25.41
Table 9: PSNR for image M for the two in Appendix H.
As observed, training on images to M and then testing on image M + returns a better PSNR. Our interpretation of the result is follows. When the encoder trained M images with multiple image-specific decoders, the optimization encoder having fit over all images, leading to unfavorable convergence trade-off compared to learning the shared encoder and fixing it for the (M + 1)-th image and optimizing the decoder freely from random initialization. In this case, the decoder adapts more freely for the (M + 1)-th image without having to account for other images leading to better performance.
the decoder for the ( (M + 1)-th image is affected by the
A study on the learned features in UGoDIT
To better understand the representations learned by our shared encoder, we examine the feature maps from its first few convolutional layers across two difffernnt approaches one with the proposed UGoDIT framework and the other with the shared decoder (termed single E-D in Section 3.3) framework.
To analyze the frequency content of the learned features, for the task of MRI, we applied 2D Fourier transform to each channel the encoder's layer and visualized the corresponding magnitude spectra. We computed low-frequency (LF) magnitude ratio, which is the proportion of spectral energy contained within central region of the frequency domain. This helps us to quantify, to how much extent, low frequency features dominate the learned representation. The LF magnitude ratio can be
20

---

39.0
2=2
38.5
0.1
37.5
$7.0
56.5
2000
4000
6000
8000
10000
Iteratiors
Figure 11: Average PSNR of 20 MRI test scans of UGoDIT-6 using different values of the regularization
parameter À at training and testing. Iterations in the x-axis correspond to K 10000.
defined as
Ee E Eu.)ee Σ. (u.v) [ vil
LF ratio
where de e] the 2D Fourier transform of the feature map at channel and C a centered square region in frequency space corresponding to low frequencies.
Slice
Slice 2
Metric
UGoDIT
Single E-D
UGoDIT
Single
LF magnitude ratio: First Layer
0.463
0.525
0.363
0.417
LF magnitude ratio: Second Layer
0.341
0,318
0.302
0.325
LF magnitude ratio: Third Layer
0,235
0.337
0,233
0.357
magnitude ratio: Fourth Layer
0.166
0.232
0.207
0.303
I.F magnitude ratio: Decoder Output
0.475
0.500
0.503
0.534
PSNR (dB)
36.61
33.98
35.92
34.82
Table 10: Low-frequency (LF) magnitude ratios at the output of four encoder layers (first four rows) and the decoder output (fifth row), and the overall PSNR for UGoDIT and the Single E-D approach (i.e., (6)). The task is MRI reconstruction 4x acceleration factor. For reference, the LF magnitude ratios for the ground truth images are 0.412 (Slice and 0.439 (Slice 2).
Table 10 summarizes the low-frequency (LF) magnitude ratios computed for two images across four encoder layers and the decoder output, along with PSNR (in dB). These results are reported for both the UGoDIT and Single E-D (one encoder, one decoder) approaches where we used an acceleration factor of 4x.
Across both test slices, Single E-D exhibits higher LF magnitude ratios in most layers, including the decoder output, indicating stronger low-frequency bias. UGoDIT achieves higher PSNR values for both slices that have only moderate low-frequee low-frequency content. Its feature representations better capture high-frequents details in the image and yield better final reconstructed images than the single E-D case, which shows a more pronounced low-frequency bias from shared training.
Impact of the number of convolutional layers in the architecture of UGoDIT
In this section, analyze the impact of varying the depth of the convolutional blocks in UGoDIT. Specifically, we evaluate five different configurations of the Skip-net architecture [52] on the super- resolution task performance, each with a different number of upsampling and downsampling convolu- tional layers, and report the resulting PSNR values.
As shown in Figure 12, the configuration with five convolutional layers in both the encoder and decoder produces the highest and most stable PSNR. Increasing the number of layers beyond five yields no further improvement, while reducing the layer count leads to a noticeable degradation in PSNR.
K Impact of the number of frozen layers in the encoder of UGoDIT
In this subsection, we attempt to answer the question: How much of the encoder arm needs to be trained over the M images?
To this end, in Table 11, we report the results UGoDIT-6 (MRI task) with freezing different number of encoder layers. As observed, the default setting (5 frozen layers) returns the best results.
21

---

31.5
30.0
29.5
25.0
Number of Convolution Layers
Figure 12:
Ablation study in terms of PSNR as a function of the number of Convolution layers in the Skip Net for the Super Resolution task, averaged over 20 test images. The x-axis indicates the number of convolutional layers in each of the upsampling and downsampling parts of the network architecture.
Number encoder frozen layers test time
Average PSNR over 20 test slices
5 (default)
35.89
35.70=241
35.37-1051
35.23=2044
34.89
Table 11: Average test-time PSNR 20 fastMRI test slices for UGoDIT-6 with freezing different numbers of layers in the 5-layer encoder.
L Impact of the number of training images (M) in UGoDIT
In this section, we investigate how the size of the training set influences UGoDIT' 's performance. To this end, we consider the non-linear deblurring task and train the shared encoder using five different FFHQ-derived subsets [51], and evaluate each on the same deblurring benchmark. As shown in Figure 13, performance steadily improves as we increase the number training images, peaking at six images beyond which additional data yields negligible gains.
30.5
PSNR (dB)
30.0
29.0
20
Number of Training Ima gas
Figure 13:
Ablation study on the number of training images, M, as given in Algorithm and Algorithm 2. Results are averaged over 20 imaaes for the task of NDB.
M Impact of the selection of N and K in UGoDIT
Here, we conduct an ablation study on the number of gradient updates, N, for each inputs updates (i.e., K) in the training (Algorithm and testing (Algorithm 2) UGoDIT.
The results are given presented in Figure 14, where we report the average PSNR values across the tasks of MRI (considering combinations of (N, K) as (1, 4000), (2, 2000), and (4. 1000)). SP, and NDB (considering combinations (N, K) as (4, 5000), (10, 2000), and (20, 1000)).
The results show that the combination we selected (for each task), average, returns the best results.
22

---

MRI
Task
NDB
36.00
31.00
32.0
35.75
35.50
31.5
30.90
PSNR (49)
35.25
35.00
31.0
34.75
30.80
34.50
30.5
34.25
30.70
34.00
(1,4000)
(2,2000) (4,1000)
(20,1000)
(10,)000) (20,1
K)
Figure 14: Ablation study the number of gradient updates, N. for every inputs updates, K, given in Algorithm and Algorithm 2. Results are given as average 20 images for tasks of MRI (left), (middle), and NDB (right).
N Limitations and future work
In this paper, we assume full access to the forward model and focus on 2D image recovery problems. In future work, aim extend UGoDIT 3D image reconstruction and the blind setting, where the goal to jointly reconstruct the image and estimate the forward operator.
Additionally, our out-of-distribution (OOD) performance varies across tasks, discussed in Ap- pendix A. Specifically, for MRI, our OOD performance is lower than that of independently running aSeqDIP. In contrast, for super-resolution (SR), observe improved acceleration while slightly outperforming DIP in terms of PSNR. As such, for future work, we will explore combining method with test-time adaptation approaches enhance performance OOD test cases. Fur- thermore, we will consider cases where the training and/or testing data come from different forward models, i.e., train-test distribution mismatch.
O Implementation details for baselines
Natural image restoration baselines: For Diffusion- approaches, we use the authors' official code for each method: DPS with the DDPM solver using 1, 000 diffusion steps; SITCOM with N = 20 diffusion steps where K = 20 optimization steps are used for each. These methods use the pre-trained DM (trained on FFHQ) from DPS. For STRAINER, the Shared INR architecture consists of single shared encoder composed of six sine-activated layers-ane and io decoder heads, where is the number of images to be learned. During training, we jointly train the encoder and all decoders by minimizing the sum of per-decoder mean squared enror (MSE) losses using ADAM [47] with a learning rate 0.001. test time, we instantiate new single-decoder INR initialized with pre-trained encoder weights and train only both using data-consistency loss tailored to the relevant forward operator.
MRI reconstruction baselines: For DM-based methods, the pre-trained model is from DDS [12]. In DDS, we set 100 diffusion steps, whereas SITCOM-MRI uses N = 50 diffusion steps, 10 optimization steps, and À = 0 (no regularization). For the supervised learning baselines (i.e., VarNet and Recurrent VarNet), we adopt the default VarNet configuration, comprising 12 cascades of unrolled reconstruction steps. Optimization is performed with Adam (learning rate 3 x 10 10-4) on the FastMRI knee dataset [22]. which contains 973 volumes. From each volume, we discard the first and last five slices, then randomly sample 8,000 images from the remaining slices form the training set.
P Visualizations
Figure 15 shows the clean versions of 6 training images (top row) and the 20 testing images sampled from the FFHQ dataset that we used in UGoDIT. Figures 16 and 17 present additional MRI visualiza- tions. Samples from the natural image restoration tasks are given in Figure 18 and Figure 19 super resolution and non-line non-linear deblurring, respectively.
23

---

Figure 15: Top: the training images. Bottom: the 20 test images.
Ground Truth
Input
Self-Guided
DDS
PSNR oo dB SITCOM MRI
PSNR dB
PSNR 32.78 dB UGoDIT (Ours)
PSNR 34,55 dB
DIP
PSNR 35.67 dB
PSNR 34.48 dB
PSNR 35.56 dB
Figure 16: Visualization of ground-truth and reconstructed images using different methods of knee image from the fastMRI dataset with 4x k-space undersampling. A region of interest is shown with green box and its (magnitude) shown in the panel the bottom left.
24

---

Ground Truth
Input
Self-Guided
DDS
dB
16.45 DIP
PSNR 31.65
PSNR 32.15
SITCOM MRI
UGoDIT (Ours)
Figure 17: Visualization of ground-truth and reconstructed images using different methods of knec image from the fastMRI dataset with 8x k-space undersampling. region of interest is shown with green box and its error (magnitude) shown panel on the bottom left.
Ground Truth
Input
STRAINER
DPS
SITCOM
DSeqDIP
UGoDIT (Ours)
PSNR 31.78 dB
PSNR 33.244 dB
PSNR 31.96 dB
Figure 18:
Super Resolution example from the FFHQ dataset.
25

---

Ground Truth
Input
STRAINER
DPS
PSNR dB SITCOM
PSNR 23.24 dB
PSNR 24.82 dB UGoDIT (Ours)
PSNR 25.52 dB
DIP
PSNR 30.82
PSNR 30.17
PSNR 31.06
Figure 19: Non-linear deblurring example from the FFHQ datasct.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Section 2 for how our method is related prior arts, Section 3 for the proposed method.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix N.
Guidelines:
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
Justification: We provide code base with instructions on how to re-produce our results. See Settings in Section
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: We provide a code base with instructions on how to re-produce our results. See Settings in Section
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide code base with instructions on how to re-produce our results. See Settings in Section 4.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper provide experimental results.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We specify the GPU used in Section 4.
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the httppalic/eeiccccieeeeee......ee
NeurIPS Code of Ethics https://neurt
Answer: [Yes]
Justification: We follow NeurlPS Code Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents method for unsupervised learning for image recon- struction in the low data regime.
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The paper does not existing assets.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
27

---

Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
28


Reviews:
[{"review_id": "lX4CYxa5ZQ", "reviewer": "Reviewer_2CSh", "summary": "The paper introduces UGoDIT, which uses a shared convolutional encoder ( encoding set of layers of a DIP U-Net) and disentangled convolutional decoders (decoding set of layers of a DIP U-Net). During the training phase, UGoDIT reconstructs $M$ measurements, thereby capturing a data-driven prior and learning transferable weights beneficial at test-time. At test time, the pre-trained encoder is kept frozen while optimizing the decoder parameters in order to recover a clean / noise-free image. The approach is evaluated on accelerated MRI reconstruction, super-resolution, and non-linear deblurring tasks and shows superior performance over existing baselines.", "questions": "1. Can the authors provide more analysis on the out-of-distribution capabilities of UGoDIT? \n2. Can the authors also provide visualizations or insights on the quality of features learned in the shared encoder?\n3. How does the set of training images impact reconstruction quality? For example, Figure 15 (supplementary) exhibits a clear bias towards skin tone in training and evaluation sets.\n4. Can the authors comment on the stability and impedance to overfitting noise over a long number of iterations of UGoDIT?", "limitations": "Yes. However, please include Limitations in the main paper and not in the supplementary.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "__Strengths__:\n\n1. The paper addresses overfitting to noise and high computational cost at inference for deep image prior-based methods by learning generalizable features in the shared encoder layers, resulting in better reconstruction quality.\n2. The paper provides a thorough and extensive evaluation. It includes comparisons with relevant baselines and provides rigorous ablation studies.  A clear empirical effect of design decisions is presented in the paper.\n3. The papers include experiments to highlight the method’s effectiveness and demonstrate competitive, if not better, performance while using subsampled/degraded to baseline methods, which use fully sampled / clean data.\n\n__Weakness__:\n1. Lack of theoretical justification: While the paper shows empirical evaluations, it fails to mention any theoretical insights or analysis as to why sharing the initial encoder layers would learn better feature learning. The authors should detail a more rigorous theoretical intuition beyond what's currently mentioned in section 3.2. \n2. Bias in training set: From Figure 15 (supplementary), it seems that the training set and test set exhibit a bias towards properties such as skin tone in face images, which may lead the shared encoder to learn favorable features. Further, MRI examples also include similar structure and intensity patterns. The paper does not discuss the performance of UGoDIT when the training set differs substantially from the test set, while comprising a high variance in structure and texture of samples within the same class/task.\n3. The paper is overly verbose and lacks brevity. The paper can perhaps tidy up the repeating equations across the main text and further refine the overall process of UGoDIT and its underlying intuition.\n4. Small test sets: The paper tests the approach on 20 images in the test set, which does not convey sufficient information to assess the approach's generalizability and superior performance over baselines.", "was_revised": true, "final_justification": "The idea presented in the paper is interesting, with theoretical insights from multi-task learning and recent literature from neural fields.  I would argue that the paper provides limited empirical evidence, making it difficult to thoroughly understand how the proposed method performs in a wide range of inverse problems and compares to competitive baseline methods. Increased my score.", "rebuttal": "We thank the reviewer for their comments and feedback. We are glad that the reviewer finds that our paper addresses overfitting to noise and high computational cost at inference for DIP-based methods, and acknowledges that we provide a thorough and extensive evaluations. As suggested by the reviewer, we will include the the limitations in the main body of the paper. \n\n## (W1) Theoretical justification to why sharing the initial encoder layers would learn better feature learning beyond what's currently mentioned in section 3.2.\n\nWe thank the reviewer for this valuable comment. While our discussion in Section 3.2 provided intuitive motivation, we now offer a justification from the literature of **multi-task representation learning**. In our setting, each degraded image corresponds to an image-specific inverse problem, which can be interpreted as a separate task.\n\nOur setup aligns with the multi-task learning framework studied in [A, B], where a shared representation (encoder $h_\\phi$) is learned jointly across multiple tasks, each with its own function (i.e., task-specific decoder $g_{\\psi_i}$ for $i \\in [M]$). Specifically, Theorem 3 in [A] and Theorem 2 in [B] demonstrate that this structure (i) reduces generalization error across tasks and (ii) promotes robust and transferable feature learning in the shared encoder. Importantly, these theoretical results rely on the structural assumption of a shared encoder with task-specific heads.\n\nIn contrast, the single-encoder, single-decoder setup (as in Eq. (5)) violates this assumption, since the decoder receives gradients from multiple tasks simultaneously. This prevents specialization and introduces **gradient interference** — a known challenge in multi-task learning [C]. Consequently, the theoretical benefits shown in [A, B] do not extend to the single-decoder case which motivates our proposed multi-decoder method. We will include this discussion in the revised paper. \n\n\n[A] A Model of Inductive Bias Learning, JAIR 2000  \n[B] The Benefit of Multitask Representation Learning, JMLR 2016  \n[C] Gradient Surgery for Multi-Task Learning, NeurIPS 2020  \n\n## (W2+W4+Q2) Is there bias in training set? Impact of the selection of training set. Evaluation on more than 20 test images. Performance of UGoDIT when the training set differs substantially from the test set.\n\n**We would like to emphasize that we selected the training and testing images completely at random**. This includes the set of images in Figure 15, our in-distribution (Section 4), and out-of-distribution (Appendix A) evaluations. This means that there is no intentional, skin-tone, or favorable bias. \n\nWe note that the selection of images in our OOD experiments is done such that the training and testing sets are substantially different. For example, the faces training images are the degraded versions of the ones in the first row of Figure 15 while the testing images are the ones in Figures 8 (a bird) and 9 (a front view of a cinema building). \n\nTo fully address the comment, we evaluate six models of UGoDIT-6, each trained with a distinct set of randomly selected training images $Y_i$, and baselines using the same **100 test images** for the SR task. Distinct training sets mean that for any $i$ and $j$, $Y_i \\cap Y_j = \\emptyset$. \n\n| Method       | Test-time Avg. PSNR over 100 test images |\n|--------------|------------------|\n| UGoDIT-6 with $Y_1$       | 30.64 $\\pm$ 0.24         | \n| UGoDIT-6 with $Y_2$       | 30.57 $\\pm$ 0.31         | \n| UGoDIT-6 with $Y_3$       | 30.65 $\\pm$ 0.27         | \n| UGoDIT-6 with $Y_4$       | 30.58 $\\pm$ 0.25         | \n| UGoDIT-6 with $Y_5$       | 30.74 $\\pm$ 0.16         | \n| UGoDIT-6 with $Y_6$       | 30.95 $\\pm$ 0.13         | \n| STRAINER-6       | 29.72      $\\pm$ 0.65         | \n| DPS       | 25.63       $\\pm$ 0.67         | \n| SITCOM       | 30.81       $\\pm$ 0.82         | \n\n\nAs observed, the averaged PSNR of our method ranges between 30.58 dB to 30.95 dB, i.e., the variability using different small training sets is very small (less than 0.4 dB). This indicates that the in-distribution results of UGoDIT is not highly sensitive to the selection of training images. \n\nIn our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using **100 ImageNet test images** and **20 CBSD65 test images** for the SR task (additional OOD evaluation).\n\nFor MRI, we refer the reviewer to our additional results using additional sampling patterns in our response to Reviewer oaHC (W4+Q3). \n\nWe will include both results to the revised paper. \n\n## (W3) The paper is verbose; equations are repeated, and the description of UGoDIT and its intuition should be streamlined.\n\nWe thank the reviewer for this helpful comment. In the revision, we will refine the writing near the optimization formulation of UGoDIT and the alternating updates of Eq.(3) by consolidating related derivations, and move secondary details to the appendix.\n\nWe agree with the reviewer that equations (3) to (5) have similarities, especially in terms of using a measurement consistency term and an autoencoding term. We will update the writing to reduce redundancy. That said, we respectfully clarify that the equations serve distinct purposes: Eq. (3) introduces the alternating updates to solve Eq. (2). Eq. (4) defines the test-time optimization. Eq. (5) introduces the baseline with the shared decoder.\n\n## (Q1) Can the authors provide more analysis on the OOD capabilities of UGoDIT?\n\nWe thank the reviewer for this question. We include a detailed out-of-distribution (OOD) evaluation in **Appendix A**, where we assess UGoDIT’s performance when tested on distributions that differ from the training set. Specifically:\n- In **Appendix A.1**, we train UGoDIT on knee MRI scans and test it on brain scans. We observe that performance degrades compared to in-distribution cases. See Figure 6. \n- In **Appendix A.2**, for the super-resolution task, we train on FFHQ (dataset of faces) and test on ImageNet (see Figures 8 and 9). UGoDIT shows better convergence and improved final PSNR over standalone DIP (aSeqDIP), demonstrating robustness under natural image distribution shifts.\n\nWe will clarify and emphasize these findings more clearly in the revised version.\n \n\n## (Q2) Can the authors provide visualizations or insights on the quality of features learned in the shared encoder?\n\nWe thank the reviewer for this question. It is indeed important to shed more light on the learned features. \n\nIn **Appendix D**, we include a study analyzing the features of a UGoDIT encoder trained on MRI data. We analyze the encoder outputs at various layers and compare them with those from a single-decoder baseline.\n\nOur analysis shows that the shared encoder in UGoDIT tends to learn smoother and more structured intermediate representations that are more robust to input variations. \n\nAs for visualizations, we note that features at the output of the trained encoder are of much lower dimension and do not typically hold semantic meanings. We will add more context to further clarify this point.\n\n\n## (Q4) Can the authors comment on the stability to overfitting noise over a long number of iterations of UGoDIT?\n\n\nWe thank the reviewer for this question. We partially address this in **Appendix C**, where we evaluate UGoDIT’s robustness to noise overfitting across different values of the regularization parameter $\\lambda$, using 10000 iterations. Additionally, in **Appendix A.1**, we run UGoDIT (and other variants) for 30000 iterations in the OOD setting.\n\nWe would like to clarify that the autoencoding term $\\|\\|g_{\\psi_i} \\circ h_\\phi(z_i) - z_i\\|\\|_2^2$ acts as an implicit constraint that encourages the output of the NN (i.e., the reconstruction) to remain close to the input. This promotes stability during optimization and mitigates overfitting to noise in the measurements. For this reason, we hypothesize that even with substantially more iterations, PSNR curves will plateau rather than degrade, as the network converges to a solution that satisfies both the data fidelity and autoencoding terms.\n\nDue to NeurIPS restrictions on including external links, we were not able to share plots for running UGoDIT for more than 30000 iterations. However, we promise the reviewer that we will include experiments with more iterations in the revised version to further empirically support this claim."}, {"review_id": "FSe1DsxxAA", "reviewer": "Reviewer_eMg2", "summary": "This work proposes UGoDIT: a DIP-based method that uses a small number of corrupted measurements for training. The method uses  M corrupted images at train time to learn one shared-encoder + M independent decoders, with the additional autoencoder constraint (this is different from DIP). At test time, a new sample is optimized by solving for a new deoder (and keeping the original encoder as fixed).\n\nThe method is compared to other variants of DIP, as well as to end-to-end and generative methods trained on large sets of clean images across compressed sensing MRI, super-resolution, and non-linear deblurring. Results indicate that UGoDIT can match or outperform performance of all other approaches.\n\nStrengths and Weaknesses\nStrengths:\nThe paper is easy to follow and the idea is presented in a conceptually simple way. The work contextualizes the different options depending on the amount/quality of available training data, and compares to a number of these combinations. Results are presented over several different imaging inverse problems tasks, including out-of-distribution experiments. Several ablation experiments are performed and discussed.", "questions": "Please refer to the Strenghts and Weaknesses section for a list of questions and concerns. Specifically, my main questions are regarding training on clean vs. corrupt, comparison to measurement splitting, and more training details.", "limitations": "Please refer to the Strenghts and Weaknesses section for additional limitations that should be discussed. Specifically, training memory scales with number of reference images at test time.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe paper is easy to follow and the idea is presented in a conceptually simple way. The work contextualizes the different options depending on the amount/quality of available training data, and compares to a number of these combinations. Results are presented over several different imaging inverse problems tasks, including out-of-distribution experiments. Several ablation experiments are performed and discussed. \n\nWeaknesses:\n- It seems like the main contribution of the work is to use a shared-encoder with M decoders at test time. The auto-encoding self-consistency was previously proposed by aSeqDIPin Ref 24.\n\n- The justification on page 5 for the self-consistency is not really clear. Unless I am misunderstanding, the minimum of Eq 2. is z_i = A^T y_i, i.e. do nothing. It is not really clear how this formulation avoids the need for early stopping, which is the original issue with DIP.\n\n- The key idea of the method seems to be unrelated to the fact that the training is over corrupt images. For example, and as another form of ablation, what would happen if M clean images were used to train the shared encoder? Wouldn't this represent the best-case learning? In addition, what is the distinction at test time of the new sample? What would happen if it were included as the (M+1)'th sample at training time instead? Would performance improve at the cost of compute?\n\n- There is no mention or comparison to any of the works that use measurement splitting as a form of self-supervision, even though some of these have been directly applied to DIP. Like my comment above, it seems to me like measurement splitting could be used here as well.\n\n[1] Noise2Self: Blind Denoising by Self-Supervision  Joshua Batson, Loic Royer \n\n[2] Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data\nBurhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kâmil Uğurbil, Mehmet Akçakaya\n\n[3] Zero-Shot Self-Supervised Learning for MRI Reconstruction  Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Mehmet Akçakaya\n\n\n- There is no discussion regarding test-time training methods, such as the following reference:\n[4] Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing Mohammad Zalbagi Darestani, Jiayu Liu, Reinhard Heckel Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4754-4776, 2022. \n\n- It seems like the training requirements scale with the number of images used. This is not discussed as a limitation.\n\n- It is unclear why the method performs better than methods trained on order of magnitude clean images. If this is indeed the case, then this should be emphasized and discussed.\n\n\n- no details are shared on how the M images were picked. this seems important given the message of appendix  A\n\n- Appendix F figure 13 does not seem to indicate that increasing M helps. While the mean increases, the variance does not always shrink, there are outliers, and none of the results would likely be significant. Furthermore, the difference is less than 0.5 dB. The plot also starts at M=4 (what does it look like for M=1,2,3 ?)\n\nAppendix I: Very few details are given regarding the training setup. The paper does not describe how the train/test sets were formed (are only 20 images used for validation?) The code does not indicate what should be run to replicate the different results, and does not appear to include any of the MRI experiments. Regarding the MRI experiments, how were the slices chosen for training and testing? Appendix I makes it sound like different slices from the same volume were used in training and in testing? Is that the case?\n\n- Almost no images are shown except in the appendix (and all MRI images in the paper are upside-down). The few that are shown are very small; Are the ROIs of Fig 16 the same central image just copied onto the top-left? It would be more helpful to make the ROI larger and increase the scale of the error image.", "was_revised": true, "final_justification": "I have updated my score from borderline reject to borderline accept. I have also increased my confidence from 4 to 5. I have left detailed comments in response to the authors.\n\nUpdate Aug 6: I have increased my score to 4 to 5 and increased significance from fair to good.", "rebuttal": "We are glad that the reviewer finds our paper ''easy to follow'' and ''presented in a conceptually simple way'', and acknowledges the several imaging tasks, OOD experiments, and the several ablation studies.\n\n## (W1) Main contribution is the shared-encoder with $M$ decoders. Self-consistency was proposed by aSeqDIP.\n\nIndeed, our main contribution is the shared-encoder architecture jointly trained with $M$ decoders on degraded images. This design learns transferable encoder weights that significantly accelerate and improve DIP-based reconstruction at test time.\n\nWhile we extend the “auto-encoding self-consistency” formulation in aSeqDIP [24] to our multi-decoder setting, UGoDIT differs in three key ways:\n- aSeqDIP operates on per-image and learns no transferable prior; UGoDIT learns a shared encoder across $M$ degraded inputs.\n- UGoDIT’s encoder is fixed at test time, enabling faster reconstructions and better generalization (Figures 2-3, and Appendix B). See also our response to Reviewer usw1 (D2). \n- The multi-decoder structure enables task-specific learning (where each task is an inverse problem), accelerating convergence at test time.\n\nThus, we think that extending self-consistency to a transferable, low-data regime with shared encoder weights is non-trivial and impactful, especially when clean data is unavailable, and it has led to significant improvements in PSNR and convergence.\n\n\n## (W2) Clarity on Eq. (2) minimizer and early stopping. \n\nThis is a two-part concern, and we clarify both points below:\n\n**Regarding Eq. (2):** The objective in Eq. (2) is jointly minimized over encoder $\\phi$, the decoders $\\psi_i$, and the inputs $z_i$. Eq. (2) is the same as Eq. (1) but with the autoencoding hard constraint over the training set that serves to mitigate overfitting. Since CNNs (like a U-Net) are known to denoise or autoencode images, they serve as an explicit constraint to restrict the solution space. The reviewer’s suggestion that $z_i = A^T y_i$ is the minimizer assumes a fixed network or identity mapping. In contrast, $g_{\\psi_i} \\circ h_\\phi$ is a deep network trained to balance measurement consistency and autoencoding regularization. The input $z_i$ is updated based on the network’s current parameters, and its convergence is determined by the evolving encoder–decoder mapping. \n\n**Regarding early stopping:** The need for early stopping in DIP arises from the risk of overfitting to noise or undesired points in the null space of $A$ when optimizing over all network weights per image. In UGoDIT, the autoencoding term $\\|\\|g_{\\psi_i} \\circ h_\\phi(z_i) - z_i\\|\\|_2^2$ helps enforce input–output consistency, acting as a regularizer that stabilizes PSNR curves over longer iterations. As shown in **Appendix C** (Figure 11), this reduces the need for early stopping.\n\n## (W3) [MAIN] (1) Training with $M$ clean images; (2) Including test image as the (M+1)'th sample at training time.\n\nWe appreciate the reviewer's important suggestions. These will help improve our work. \n\nUGoDIT is designed specifically for the **low-data regime without clean images**. However, we agree that training the shared encoder on clean images is a valid comparison and may represent an upper bound as we show next. \n\nIf clean images are available, we first need to modify the training objective since there would no longer be forward operators $A_i$ or inverse problems during training. The optimization reduces to a pure multi-decoder autoencoder, where Eq.(3a) becomes:\n\n$\\phi, \\psi_i \\leftarrow \\arg \\min_{\\phi,\\psi_i} [\\sum_{i\\in[M]} \\|\\|g_{\\psi_i}\\circ h_\\phi(z_i) - z_i\\|\\|^2_2 ],$ ... (\\*)\n\nwhere $z_i = x_i$ are the clean ground truth images.\n\nTo test this idea, we compare UGoDIT trained on degraded images vs. a multi-decoder autoencoder trained on clean versions of UGoDIT's degraded images for the SR task.\n\n| Method                                     | Training  | Test-time PSNR  |\n|--------------------------------------------|---------------------|-------------------------------|\n| UGoDIT                                   | 6 degraded images   | 30.75 $\\pm$ 0.58                  |\n| Multi-decoder AE (clean training) Eq. (\\*) | 6 clean images      | 31.02 $\\pm$ 0.67       |\n\nAs observed, test-time PSNR for the auto-encoder with clean data is *slightly* better that UGoDIT. This means that training on $M$ clean images could also serve as a prior for test-time reconstructions. The decoder for clean training is initialized at random and then optimized to reconstruct the test image. \n\nAlthough this represents a different setup from UGoDIT, we will include this ablation study in the revised paper.\n\nAs for the second part of the question, we hypothesize that including the test image as the ($M+1$)-th sample during training will not improve performance compared to our approach of learning a prior and testing on the image separately. This is supported by the results of Figure 1, where the average training curve is lower than the testing curve. The reason is that during training, the network parameters are optimized to solve multiple inverse problems, while at test time the decoder parameters are optimized (with the help of the frozen pre-trained encoder) to solve a single image-specific problem. Furthermore, including the test image as the ($M+1$)-th changes the problem from generalizing to unseen measurements to directly fitting all available data, which—as the reviewer correctly notes—comes at a higher computational cost. UGoDIT, in contrast, is designed for inference on unseen data using a fixed encoder. That said, we believe that the reviewer’s suggestion is insightful, for which we leave a deeper investigation to future work.\n\n\n## (W4+W5) [MAIN] Comparison with measurement splitting and test-time training. \n\nWe thank the reviewer for pointing us to these unsupervised learning baselines. Please see below comparison for the task of MRI. \n\n|Method (MRI task)|Test-time PSNR|\n|-|-|\n| UGoDIT                |   36.07 $\\pm$ 0.34   |\n| SSDU [A]  (measurement splitting)                             |  33.25 $\\pm$ 0.45   |\n| ZS-SSL [B] (measurement splitting)                                   |  33.47 $\\pm$ 0.66   |\n| TTT [C] (test-time with default 300 training images)     | 33.81 $\\pm$ 0.45   |\n\nAs observed, our method returns the best results as compared to the suggested baselines. We did not have sufficient time to compare to [D], but we promise to include it in a revised version.\n\n[A] (SSDU) Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data, MRM 2020\n\n[B] (ZS-SSL) Zero-Shot Self-Supervised Learning for MRI Reconstruction, ICLR 2022\n\n[C] (TTT) Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing, ICML 2022\n\n[D] Noise2Self: Blind Denoising by Self-Supervision, ICML 2019\n \n\n## (W6) Why does UGoDIT outperform high-data methods?\n\nWe thank the reviewer for this question. We believe there are two factors:\n\n**Strong inductive bias of DIP-like architectures:**  \nUntrained convolutional networks already capture useful low-frequency image structures (specifically when overfitting is mitigated), UGoDIT leverages this property while learning a transferable encoder that further enhances these representations. For example, see the visualizations in Figure 5 of aSeqDIP [24].\n\n**Regularization through shared encoder and multi-decoder training:** Our training encourages the encoder to learn features that generalize across tasks, while the multi-decoder setup reduces gradient interference and overfitting, resulting in representations that are robust even without large datasets.\n\nWe note that we do not claim that UGoDIT will always outperform high-data methods. In settings where large-scale clean datasets are available and well-matched to the test domain, supervised or generative models can achieve higher performance (such as SITCOM vs. our method in Table 2). \n\nOur key message is that UGoDIT offers **competitive performance in the low and degraded data regime**, where such large clean datasets are not available.\n\n## (W7) [MAIN] Selection of the $M$ training images.\n\nWe'd like to thank the reviewer for this comment. **The training and testing images were selected at random** including for the MRI experiments. We refer the reviewer to our response to Reviewer 2CSh (W2+Q3) for more details and experiments about the performance of UGoDIT using different training sets over the same 100 test images.\n\nIn our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using **100 ImageNet test images** and **20 CBSD65 test images** for the SR task (additional OOD evaluation).\n\n## (W8) Figure 13 results for $M\\in\\\\{1,2,3\\\\}$.\n\nWe extend the results of Figure 13 to the following table with $M\\in \\\\{1,2,3\\\\}$. We note that $M=1$ trains a one-decoder autoencoder and then fixes the weights, so the performance might highly depend on the selected training image. \n\n\n|$M$ in UGoDIT (NDB)|PSNR|\n|-|-|\n|1|29.07 $\\pm$ 0.52|\n|2|29.40 $\\pm$ 0.56|\n|3|29.80 $\\pm$ 0.45|\n|4 (from Figure 13)|30.02 $\\pm$ 0.40|\n\nAs observed, the performance slightly degrades when $M$ becomes smaller. We will include this result in the revised version.  \n\n## (W9) Training/testing code and setup. Evaluation with $>20$ test images. \n\nThank you for the comment. We added more details in the anonymous readme file and included the MRI code.  See our response to Reviewer 2CSh (W2+Q3) for evaluation over **100 test images**. \n\nIn our response to Reviewer usw1 (Q3), we report the results of an FFHQ-trained-UGoDIT-6 using **100 ImageNet test images** and **20 CBSD65 test images** for the SR task (additional OOD evaluation).\n\n## (W10) Visualizations in the main text. ROIs. MRI orientation. \n\nWe will include visual examples in the main text and fix the orientation of the MRI images. We will also enlarge the ROI (central region) crops and the error maps."}, {"review_id": "VICiSDVuv4", "reviewer": "Reviewer_usw1", "summary": "UGoDIT is a single-image reconstruction method designed for image recovery and enhancement from compressed signals. This approach combines unsupervised learning and adaptive feature extraction techniques, addressing specific challenges in image restoration by designing a training mechanism and incorporating multiple decoders and a universal encoder.", "questions": "1. As a single-image reconstruction method, it is essential to compare the required FLOPs and training time.\n\n2. The paper should include comparisons with the latest image reconstruction methods based on Implicit Neural Representations (INR), such as:\n\nCai Z. Conv-INR: Convolutional Implicit Neural Representation for Multimodal Visual Signals [J]. arXiv preprint arXiv:2406.04249, 2024.\n\n3. The paper should include generalization validation on larger datasets beyond facial or specialized medical regions, such as for natural images in tasks like super-resolution and compressed sensing. It demonstrates both the efficiency and generalization ability of the proposed method across more diverse and realistic scenarios.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Advantages of UGoDIT:\n\nSingle-image reconstruction: UGoDIT performs image reconstruction and enhancement from compressed signals, making it particularly effective in scenarios where data is limited or compressed.\n\nUnsupervised learning: The method leverages unsupervised learning, which allows it to work without relying on large, labeled datasets for training, reducing the need for ground-truth data.\n\nImproved generalization: The encoder is encouraged to extract more generalizable and robust features due to the multiple decoders, ensuring better reconstruction performance during both training and testing.\n\nDisadvantages of UGoDIT:\n\nOut-of-distribution (OOD) performance variability: UGoDIT's performance on OOD test cases varies across tasks. Specifically, it shows lower OOD performance for tasks like MRI compared to methods like aSeqDIP, limiting its application for certain medical imaging tasks.\n\nTask-dependent performance: While UGoDIT performs well on certain tasks like super-resolution (SR), it does not consistently outperform other methods like aSeqDIP on all tasks, such as MRI, indicating that its strengths may vary based on the specific image recovery task.\n\nThe experimental results of the paper show that when the number of images exceeds six, there is no significant improvement in model performance. This indicates that the enhancement from this training approach is limited and lacks strong generalization capability.", "was_revised": true, "final_justification": "All of my concerns have been addressed. The paper explores an direction for unsupervised image reconstruction, but there is still room for improvement. Therefore, I will maintain my original score. However, The proposed method suffers from two significant limitations:\n\nLimit generalization capability: As noted by the reviewers, the introduction of additional reference images improves reconstruction quality in specific scenarios such as super-resolution of human faces, outperforming baselines like UGoDIT. However, this improvement comes at the cost of reduced performance on the primary evaluation task, MRI reconstruction. This suggests that the method may trade off generalization for performance gains on narrowly distributed data.\n\nReduce effectiveness of the encoder: The fact that the encoder achieves competitive results even when trained on a minimal amount of data raises concerns about its generalization ability. Ideally, the encoder should benefit from increased data volume by learning more robust and expressive features, leading to better reconstruction. However, the proposed design appears to limit this capacity, indicating a potential flaw in its scalability. I hope the AC will take these limitations into consideration when making the final decision.", "rebuttal": "We'd like to first thank the reviewer for their feedback. We are glad that the reviewer acknowledges the limited-data setting of our work and its fully unsupervised nature. \n\n## (D1) OOD performance varies across tasks: Lower OOD performance for MRI compared to methods like aSeqDIP.\n\nWe thank the reviewer for this comment. We agree that, as shown in Appendix A, UGoDIT performs well on some OOD tasks (e.g., super-resolution) but shows reduced OOD performance for MRI.\n\nWe believe this variability stems from two factors:\n\n1. **Encoder specialization:** UGoDIT learns a shared encoder that captures features from the training degradations. When the test distribution deviates substantially (e.g., training on knees, testing on brains), the learned encoder may not fully generalize.\n\n2. **Role of autoencoding regularization:** The autoencoding term stabilizes training but also biases the encoder toward the statistics of the training images. This can limit adaptability under severe distribution shifts, whereas per-image methods like aSeqDIP can adjust their weights to the new domain.\n\nWhile this is a limitation, we see it as an avenue for future work — for example, exploring encoder fine-tuning at test time or hybrid strategies that combine UGoDIT’s transferable features with adaptive refinement. We will discuss this point explicitly in the revised manuscript.\n\n## (D2) While UGoDIT performs well on certain tasks like super-resolution (SR), it does not consistently outperform other methods like aSeqDIP on all tasks.\n\nWe agree with the reviewer that UGoDIT’s performance varies across tasks. However, compared to aSeqDIP and other dataless methods, Figure 3 shows that UGoDIT achieves **significant improvements in convergence speed** while also providing higher reconstruction quality: approximately **+3 dB** PSNR for MRI, **+1 dB** for SR, and **+2 dB or more** for NDB. \n\nTo support this claim, we report the number of optimization iterations needed to achieve 30dB across the three task, comparing UGoDIT-6 and aSeqDIP. We note that, at inference, the number of parameters to optimize in UGoDIT (with $M=6$) is lower than aSeqDIP as the encoder in our case is fixed. \n\n| Method                                     | iterations to achieve 30dB (MRI)  | iterations to achieve 30dB (SR)  | iterations to achieve 30dB (NDB) |\n|--------------------------------------------|---------------------|-------------------------------|-|\n| UGoDIT                                   | 10   | 120                  | 650 |\n| aSeqDIP |   450  | 325       | 1750  |\n\nWhen compared to data-centric baselines such as SITCOM, UGoDIT returns slightly lower PSNR but does so **without requiring any pre-trained model** or access to large datasets of clean images, making it more applicable in low-data settings.\n\n\n## (D3) The experimental results of the paper show that when the number of images exceeds six, there is no significant improvement in model performance. This indicates that the enhancement from this training approach is limited.\n\nWe agree that our experiments show that performance tends to saturate when the number of training images exceeds six. This suggests that UGoDIT is particularly well-suited for the **very low-data regime**, where only a handful of degraded images are available — a setting where many existing methods struggle.\n\nWe hypothesize that the saturation effect arises because the encoder quickly captures the shared structure present in the training images, and adding more examples beyond this point yields diminishing returns. This is consistent with our goal of designing a method that is **effective even under severe data scarcity**.\n\nThis property also makes UGoDIT appealing for domains where only a few measurements are accessible (e.g., specialized scientific imaging tasks such as black hole observations). We plan to explore such applications in future work.\n\n\n## (Q1) As a single-image reconstruction method, it is essential to compare the required FLOPs and training time.\n\nWe thank the reviewer for this comment. At test-time, UGoDIT is indeed a single-image reconstruction method. \n\nWe report the training time and FLOPs at training for the SR task, and compare it with aSeqDIP. All methods were run for 4000 iterations. \n\n| Method            | time (sec) | GFLOPs|\n|-------------------|---------------------|--------|\n| UGoDIT-4 (training)        | 384  | .0143        |\n| UGoDIT-5 (training)        | 415  | .0152        |\n| UGoDIT-6 (training)        | 472  | .0174        |\n| UGoDIT-4 (testing)         | 62  |  .0062        |\n| UGoDIT-5 (testing)         | 64  |  .0064        |\n| UGoDIT-6 (testing)         | 66  |  .0066        |\n| aSeqDIP                    | 142  | .01004       |\n\n\nAs observed, the more training images, the higher run-time and FLOPS for UGoDIT as we have more decoders to train when $M$ increases. We are at most less than four times the run-time when compared to aSeqDIP. As for testing, we only require less than half the run-time as aSeqDIP since the optimization in UGoDIT takes place w.r.t. only the decoder, whereas aSeqDIP optimizes the entire network. \n\nMoreover, our run-time and FLOPs are significantly lower than training diffusion models or supervised methods. For example, as per Figure 1 of [A], diffusion models require more than 17 GFLOPs.\n\n[A] Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture, CVPR 2024\n\n## (Q2) Comparison and discussion with latest INR methods such as [B].\n\nWe thank the reviewer for pointing us to this baseline. **We note that the INR-based method we compare with (i.e., STRAINER [37]) is from NeurIPS 2024**. \n\nAdditionally, in this rebuttal, we include comparisons with three more baselines (see our response to Reviewer eMg2 (W4+W5)). \n\nWe could not find a code base for [B], and due to the limited time of the rebuttal, we couldn't code it from scratch. However, we promise the reviewer to include discussion (and/or results) of [B] in our revised paper. \n\n[B] Conv-INR: Convolutional Implicit Neural Representation for Multimodal Visual Signals, ArXiv 2024\n\n## (Q3) Generalization validation on larger datasets. \n\nWe thank the reviewer for this suggestion. \n\nFor more diverse OOD evaluation, we report the results of an FFHQ-trained-UGoDIT-6 using **100 ImageNet test images** and **20 CBSD65 test images** for the SR task. \n\n| Method            | Test-time Avg. PSNR on 100 ImageNet images |\n|-------------------|------|\n| FFHQ-trained-UGoDIT-6     | 25.67  | \n| ImageNet-trained-UGoDIT-6          | 25.89  | \n\n| Method            | Test-time Avg. PSNR on 20 CBSD65 images |\n|-------------------|------|\n| FFHQ-trained-UGoDIT-6     | 26.98  | \n| CBSD65-trained-UGoDIT-6          | 27.13 | \n\n\nWe would like to clarify that we include an evaluation in **Appendix A.2**, where we test an FFHQ-trained-UGoDIT-6 on **20 natural test images from ImageNet** for the super-resolution task in Figure 7 (with visual examples in Figure 10). \n\nThe results demonstrate that UGoDIT maintains strong performance across different datasets. We will include these in the revised paper."}, {"review_id": "ewU11V1mD6", "reviewer": "Reviewer_oaHC", "summary": "This paper proposes a new training approach for DIP methods in solving inverse imaging problems. Instead of the standard DIP approach, which trains a full network for a dataset, this paper concentrates on CNNs that have encoder-decoder structures, and shares the encoder for training over few datasets, while tuning the decoder individually for each problem. Additionally, this training is built into an alternating optimization algorithm that also updates the input to the network. The results show competitive results. However, there are concerns about the justification of the method, the limited evaluations and implementation details.", "questions": "- Why does the encoder trained only on 4-5 images generalize to unseen data well? If it does not, as eluded to in Appendix A, then what is the advantage over standard \"dataless\" methods\n- Why are the methods not evaluated over the full test datasets?\n- What are the details of the forward operators?\n- What are the details of how the M images for the proposed method are selected with respect to the test cases?", "limitations": "There is a limitation section in the appendix, but this is more about future work than actual limitations, e.g. the limited evaluation and other issues are not discussed.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The idea of sharing the encoder of a U-Net is reasonable, which has been explored in other scenarios before.\n- The reported results show improvement over existing datasets.\n\nWeaknesses:\n- Unclear why this setup is valuable, since the run time is still quite high. There are also other non-DIP methods for test-time training that are not even discussed.\n- While the idea of the shared encoder is intuitive, it is unclear why this would work in the limited data regime in the first place. This is somewhat discussed in Sec. 3.2, where it is claims that \"encoders that yield more accurate reconstructions during training tend to reconstruct better during testing\". However, the \"more accurate\" nature of the encoder is never evaluated, except via the reconstruction. Furthermore, it is not clear why the encoder trained on 5-6 images is not overfitted, and can provide reasonable representations for new test data.\n- 20 test cases is not enough to evaluate these methods. \n- Forward operators are not clearly defined for any of the experiments. For instance, the MRI data is most likely undersampled randomly, but this is not mentioned. It is also unclear if it will work well for uniform undersampling that typically arises in brain imaging, since 4b inherently relies on denoising of the input via projections, which applies to random undersampling but not uniform.\n- It is not clear how the additional M images for the proposed methods are selected with respect to the test cases. For instance, if all MRI datasets are chosen from similar cases for this (e.g. same field strength, same contrast/SNR), this is an unfair advantage compared to the database-trained DM and supervised methods that are trained across varying conditions.\n- There is something wrong with the VarNet implementation (which incidentally is outdated). The metrics are below previously published results. The run time is also excessively long, since these type of networks usually take on the order of hundred ms, not 5 seconds. It also looks like sensitivity maps were pre-generated via BART, which should further cut down on inference time. No images are shown for VarNet either.\n- Similarly, PSNR/SSIM calculations on FFHQ does not seem to align with the images, e.g. in Fig. 18. In this figure, STRAINER and aSEQDIP are highly noisy/pixellated, yet outperform DPS by 1 and 4 dB respectively.\n- No ablation studies on the impact of the neural network architecture choice (Appendix E does not address this) or how much of the encoder arm needs to be trained over the M images.\n\n- Some claims in the introduction are not aligned with the literature. For instance, there are many work that train DMs for MRI/CT from scratch without adapting from natural images, or the claim that unsupervised methods fall short of generative models need citation.\n- Unlike what is stated in Sec. 3.1, not all CNNs include a downsampler and an unsampler. Please rewrite this for the U-Net architecture, which is the only one considered.\n- Impact of using M decoders is shown with eq. (5), why not (1)?", "was_revised": true, "final_justification": "The rebuttal has answered some of the questions raised. There are still some fundamental issues, such as the lengthy run time making these impractical or the setup where this is useful, which are inherent to the study design. I have increased my initial score accordingly.", "rebuttal": "We'd like to thank the reviewer for their comments. We are glad that the reviewer acknowledges our use of a multi-decoder UNet and the improvements over baselines. \n\n## (W1) Setup value. Run-time. Comparison with test-time training methods.\n\n**Value of the setup** is in **applications where only a few degraded measurements are available** (e.g., in some scientific and medical applications). In these low-data regimes, data-centric methods requiring large clean datasets are not applicable, and our approach provides a strong case under such scenario.\n\n**Our Run time** when compared to standalone DIP-based shows that UGoDIT achieves **significantly faster convergence** because the learned encoder provides a strong initialization at test time. **See our response to Reviewer usw1 (D2) for the number of iterations needed to achieve 30dB at inference across the three tasks**.\n\n**Comparison with test-time training methods:** See our response to Reviewer eMg2 (W4+W5) for comparison with a **TTT method from ICML 2022** in addition to 2 other baselines. **We would also appreciate any specific TTT papers that the reviewer considers relevant.**\n\n## (W2) Why the shared encoder works in the limited-data regime? The \"more accurate\" nature of the encoder is only evaluated by reconstruction. Why the encoder trained on 5-6 images is not overfitted?\n\nWe thank the reviewer for raising these important questions.\n\n**Shared encoder in the limited-data regime:** UGoDIT jointly optimizes the encoder with multiple decoders across a small set of degraded images, which encourages it to capture **task-common features**. This inductive bias, supported by multi-task learning (see our response to Reviewer 2CSh (W1)), enables the encoder to generalize even when trained with as few as 5–6 images.\n\n**Evaluation of \"encoder accuracy\":** We use reconstruction quality as the primary evaluation metric because the encoder is trained solely to produce features that aid reconstruction. Thus, improved reconstruction accuracy on unseen test images directly supports the claim that the encoder has learned meaningful representations. **We would welcome suggestions from the reviewer on alternative evaluation criteria/metrics they would find more convincing**.\n\n**Encoder's features not over-fitting:** The shared encoder saves parameters and hence prevents over-fitting. In **Appendix D**, we provide a feature analysis showing that the encoder trained with multiple decoders learns smoother and more structured intermediate representations compared to a single-decoder baseline. These features transfer well to new test data, which explains the strong generalization despite the small training set.\n\n## (W3+Q2) Evaluation on $>20$ test images.\n\nPlease see our response to Reviewer 2CSh (W2+W4+Q2), where we conducted experiments **using 100 test images**. \n\nWe report the results of an FFHQ-trained-UGoDIT-6 using **100 ImageNet test images** and **20 CBSD65 test images** for the SR task in response to Reviewer usw1 (Q3).\n\n## (W4+Q3) UGoDIT MRI with uniform sampling. \n\nIn our experiments, we used **variable density random sampling masks commonly used in compressed sensing MRI** [A]. \n\nIn response to the reviewer's suggestion, **we report the performance of UGoDIT-6 using uniform (equispaced) undersampling patterns on fastMRI brain data**. For each acceleration factor (AF), one-third of the k-space lines are fixed at the center, while the remaining lines were uniformly distributed across the high-frequency regions.\n\n| Method|PSNR at AF = 4x|PSNR at AF = 8x|\n|-|-|-|\n|UGoDIT-6|35.88 ± 0.58|32.78 ± 0.68|\n|SITCOM|36.33 ± 0.37|32.92± 0.64|\n\nAs observed, UGoDIT results are still competitive using a different undersampling patterns even when compared to SITCOIM which was trained on thousands of fully sampled images. \n\n[A] Sparse MRI: The application of compressed sensing for rapid MR imaging, MRM 2007\n\n## (W5+Q4) Selection of $M$ images. Unfair advantage (compared to high-data methods that are trained across varying conditions) if all MRI data are chosen from similar cases.\n\nWe thank the reviewer for this comment. **The training and testing images were selected at random** including MRI, i.e., they are not selected to be close to the test images. \n\nIn our evaluation, DMs are trained on fully sampled measurements, whereas supervised methods are trained separately for each setting. For example, in Table 1, we have two pre-trained VarNets, one for 4x and one for 8x, where 8000 pairs of sub-sampled and full-sampled images are used. In UGoDIT's main results, at most, we train on six sub-sampled images.\n\nTo show the impact of the selected training images, **see to our response to Reviewer 2CSh (W2+W4+Q2)**. \n\n## (W6) VarNet (being outdated) selection, results, run-time, and visualizations.\n\nWe thank the reviewer for this comment.\n\n**On VarNet being \"outdated\":** We respectfully disagree with the characterization of VarNet as outdated. VarNet [17] is a paper from MICCAI 2020 (a well-known venue for MRI) that remains a widely used baseline in MRI reconstruction. However, to address the reviewer's concern, we compare with **Recurrent VarNet (CVPR 2022)** [B].\n\n| Method|PSNR| \n|-|-|\n|UGoDIT-6| 36.05 $\\pm$ 0.78 | \n|Recurrent VarNet [B]|35.88 $\\pm$ 0.46|\n\nOur VarNet implementation follows the official codebase but was trained on **8,000 images** (as mentioned in Table 1), which is fewer than what typically used to report SOTA results. This difference likely explains the lower metrics we observed. However, the performance of our approach trained in very low data regime is still competitive with methods trained on orders of magnitude more images indicating the advantages of expertly exploiting shared architectures, regularization, and implicit biases of neural networks to direct data-consistent reconstructions.\n\nThe ~5 seconds reflects **end-to-end inference** including BART-based sensitivity map generation and data loading. This also depends on the compute power. \n\nWe will include VarNet and Recurrent VarNet visuals with error maps in the revised paper. **The key message for our comparison with supervised methods is to show that UGoDIT can achieve competitive results without needing large number of paired training images.**\n\n[B] Recurrent Variational Network: A Deep Learning Inverse Problem Solver applied to the task of Accelerated MRI Reconstruction, CVPR 2022\n\n## (W7) STRAINER and aSeqDIP are better than DPS but noisier.\n\nThe reviewer is indeed correct. DPS is known to produce artifact-free but more blurred images that could often violate measurement consistency, leading to lower PSNR/SSIM despite perceptually cleaner results (see also the visualizations in Figure 5 of aSeqDIP). At every reverse step of DPS, one gradient is taken w.r.t. $y$ and $A$. In contrast, STRAINER and aSeqDIP enforce data consistency more explicitly.\n\n\n## (W8) Ablations: (1) Impact of the NN architecture choice beyond Appendix E; (2) How much of the encoder arm needs to be trained over the $M$ images.\n\nSince our method builds on DIP, we use a standard U-Net with residual connections. If the reviewer refers to non-CNN architectures, we note that Transformers and MLPs, unlike CNNs, lack spectral bias [15]. **We kindly ask the reviewer to indicate if they refer to other CNNs such as U-Nets that don't use residual connections.** \n\nTo address the second part, we report the results of UGoDIT-6 (MRI task) with freezing different number of encoder layers. \n\n|Number of encoder frozen layers at testing |PSNR| \n|-|-|\n|5(default)|35.89 $\\pm$ 0.28| \n|4|35.70 $\\pm$ 0.43| \n|3|35.37 $\\pm$ 0.51| \n|2|35.23 $\\pm$ 0.44| \n|1|34.89 $\\pm$ 0.38| \n\nAs observed, the default setting (5 frozen layers) return the best results. We thank the reviewer for this suggestion. We will include this study in the revised paper. \n\n## (W9) Works that train DMs from scratch without adapting from natural images; Unsupervised methods fall short of generative models need citation.\n\nWe say that DM-based methods **often** use fine-tuning in MRI/CT which is the case in several recent methods [12,13,17]. **Our sentence does not rule out the existence of other methods that train MRI/CT DMs from scratch. We would greatly appreciate it if the reviewer could point us to such works.** \n\nThanks for suggesting to cite paper(s) that supports the claim: unsupervised methods **often** fall behind DM methods. This claim is supported in works such as [C] as well as the results of baseline SITCOM in Tables 1 and 2, where a DM method is shown to be better than our proposed unsupervised method.  \n\n[C] Sequential Diffusion-Guided Deep Image Prior for Medical Image Reconstruction, ICASSP 2025\n\n## (W10) Description of UNet in Sec. 3.1.\n\nWe will re-write according to the reviewer's suggestion.\n\n## (W11) Impact of using $M$ decoders is shown with (5), why not (1)?\n\nSince Eq. (1) corresponds to vanilla DIP, overfitting is expected. Eq. (5) describes the single-encoder/single-decoder case, which—similar to UGoDIT—uses the autoencoding term with input updates that help mitigate overfitting.\n\nMoreover, there is no distinction between training and testing for Eq. (1), as vanilla DIP is a training-data-free method and does not involve $M$ training images.\n\n## (Q1) Advantage over dataless methods if UGoDIT does not generalize well on OOD. \n\nIn Appendix A.1, our method with MRI does indeed require a large number of iterations to converge. However, SR results in Appendix A.2 show that UGoDIT generalizes very well to OOD data in terms convergence speed and PSNR. Therefore, for MRI, if faced with unseen data that is from a different anatomy, using UGoDIT or aSeqDIP would return similar PSNRs (see Figure 5), but for fast convergence, it would be better to use a dataless method, such as aSeqDIP.\nWe thank the reviewer for their comment. **See our response to Reviewer 2CSh (W1) for generalization. For an analysis on the learned features, see Appendix D.**"}]

Author rebuttal:

Meta-review: In this paper, the authors present a variation of a Deep Image Prior designed to work in the setting where there are noisy measurements available for a small number of sample images.  In this approach, the model contains a shared encoder and multiple image-specific decoders.  During training time, the shared encoder and image-specific decoders are jointly optimized.  During test time, the shared encoder is held fixed and only an image-specific decoder is optimized to fit the measurements.  The method is shown to outperform baselines on a variety of linear and nonlinear image restoration tasks, along with thorough ablation studies.  The proposed approach has an elegance to it and makes significant progress in addressing challenges of generative models (high data requirements upon training) and deep image priors (overfitting to noise and non-specificity for specific signal classes) when applied to inverse problems.  The work is likely to be of interest to the community.
Decision: Accept (poster)