Title: UGoDIT: Unsupervised Group Deep Image Prior Via Transferable Weights
Venue: NeurIPS 2025
Accepted: yes
Keywords: Inverse Imaging Problems

Deep Image Prior

Unsupervised Learning

Magnetic Resonance Imaging
Abstract: Recent advances in data-centric deep generative models have led to significant progress in solving inverse imaging problems. However, these models (e.g., diffusion models (DMs)) typically require large amounts of fully sampled (clean) training data, which is often impractical in medical and scientific settings such as dynamic imaging. On the other hand, training-data-free approaches like the Deep Image Prior (DIP) do not require clean ground-truth images but suffer from noise overfitting and can be computationally expensive as the network parameters need to be optimized for each measurement vector independently. Moreover, DIP-based methods often overlook the potential of learning a prior using a small number of sub-sampled measurements (or degraded images) available during training. In this paper, we propose **UGoDIT**—an **U**nsupervised **G**r**o**up **DI**P with **T**ransferable weights—designed for the low-data regime where only a very small number, $M$, of sub-sampled measurement vectors are available during training. Our method learns a set of transferable weights by optimizing a shared encoder and $M$ disentangled decoders. At test time, we reconstruct the unseen degraded image using a DIP network, where part of the parameters are fixed to the learned weights, while the remaining are optimized to enforce measurement consistency. We evaluate \our on both medical (multi-coil MRI) and natural (super resolution and non-linear deblurring) image recovery tasks under various settings. Compared to recent standalone DIP methods, \our provides accelerated convergence and notable improvement in reconstruction quality. Furthermore, our method achieves performance competitive with SOTA DM-based and supervised approaches, despite not requiring large amounts of clean training data. Our code is available at: https://github.com/sjames40/UGoDIT.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new training approach for DIP methods in solving inverse imaging problems. Instead of the standard DIP approach, which trains a full network for a dataset, this paper concentrates on CNNs that have encoder-decoder structures, and shares the encoder for training over few datasets, while tuning the decoder individually for each problem. Additionally, this training is built into an alternating optimization algorithm that also updates the input to the network. The results show competitive results. However, there are concerns about the justification of the method, the limited evaluations and implementation details.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The idea of sharing the encoder of a U-Net is reasonable, which has been explored in other scenarios before.\n- The reported results show improvement over existing datasets.\n\nWeaknesses:\n- Unclear why this setup is valuable, since the run time is still quite high. There are also other non-DIP methods for test-time training that are not even discussed.\n- While the idea of the shared encoder is intuitive, it is unclear why this would work in the limited data regime in the first place. This is somewhat discussed in Sec. 3.2, where it is claims that \"encoders that yield more accurate reconstructions during training tend to reconstruct better during testing\". However, the \"more accurate\" nature of the encoder is never evaluated, except via the reconstruction. Furthermore, it is not clear why the encoder trained on 5-6 images is not overfitted, and can provide reasonable representations for new test data.\n- 20 test cases is not enough to evaluate these methods. \n- Forward operators are not clearly defined for any of the experiments. For instance, the MRI data is most likely undersampled randomly, but this is not mentioned. It is also unclear if it will work well for uniform undersampling that typically arises in brain imaging, since 4b inherently relies on denoising of the input via projections, which applies to random undersampling but not uniform.\n- It is not clear how the additional M images for the proposed methods are selected with respect to the test cases. For instance, if all MRI datasets are chosen from similar cases for this (e.g. same field strength, same contrast/SNR), this is an unfair advantage compared to the database-trained DM and supervised methods that are trained across varying conditions.\n- There is something wrong with the VarNet implementation (which incidentally is outdated). The metrics are below previously published results. The run time is also excessively long, since these type of networks usually take on the order of hundred ms, not 5 seconds. It also looks like sensitivity maps were pre-generated via BART, which should further cut down on inference time. No images are shown for VarNet either.\n- Similarly, PSNR/SSIM calculations on FFHQ does not seem to align with the images, e.g. in Fig. 18. In this figure, STRAINER and aSEQDIP are highly noisy/pixellated, yet outperform DPS by 1 and 4 dB respectively.\n- No ablation studies on the impact of the neural network architecture choice (Appendix E does not address this) or how much of the encoder arm needs to be trained over the M images.\n\n- Some claims in the introduction are not aligned with the literature. For instance, there are many work that train DMs for MRI/CT from scratch without adapting from natural images, or the claim that unsupervised methods fall short of generative models need citation.\n- Unlike what is stated in Sec. 3.1, not all CNNs include a downsampler and an unsampler. Please rewrite this for the U-Net architecture, which is the only one considered.\n- Impact of using M decoders is shown with eq. (5), why not (1)?\n\nQUESTIONS:\n- Why does the encoder trained only on 4-5 images generalize to unseen data well? If it does not, as eluded to in Appendix A, then what is the advantage over standard \"dataless\" methods\n- Why are the methods not evaluated over the full test datasets?\n- What are the details of the forward operators?\n- What are the details of how the M images for the proposed method are selected with respect to the test cases?\n\nLIMITATIONS:\nThere is a limitation section in the appendix, but this is more about future work than actual limitations, e.g. the limited evaluation and other issues are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUGoDIT is a single-image reconstruction method designed for image recovery and enhancement from compressed signals. This approach combines unsupervised learning and adaptive feature extraction techniques, addressing specific challenges in image restoration by designing a training mechanism and incorporating multiple decoders and a universal encoder.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages of UGoDIT:\n\nSingle-image reconstruction: UGoDIT performs image reconstruction and enhancement from compressed signals, making it particularly effective in scenarios where data is limited or compressed.\n\nUnsupervised learning: The method leverages unsupervised learning, which allows it to work without relying on large, labeled datasets for training, reducing the need for ground-truth data.\n\nImproved generalization: The encoder is encouraged to extract more generalizable and robust features due to the multiple decoders, ensuring better reconstruction performance during both training and testing.\n\nDisadvantages of UGoDIT:\n\nOut-of-distribution (OOD) performance variability: UGoDIT's performance on OOD test cases varies across tasks. Specifically, it shows lower OOD performance for tasks like MRI compared to methods like aSeqDIP, limiting its application for certain medical imaging tasks.\n\nTask-dependent performance: While UGoDIT performs well on certain tasks like super-resolution (SR), it does not consistently outperform other methods like aSeqDIP on all tasks, such as MRI, indicating that its strengths may vary based on the specific image recovery task.\n\nThe experimental results of the paper show that when the number of images exceeds six, there is no significant improvement in model performance. This indicates that the enhancement from this training approach is limited and lacks strong generalization capability.\n\nQUESTIONS:\n1. As a single-image reconstruction method, it is essential to compare the required FLOPs and training time.\n\n2. The paper should include comparisons with the latest image reconstruction methods based on Implicit Neural Representations (INR), such as:\n\nCai Z. Conv-INR: Convolutional Implicit Neural Representation for Multimodal Visual Signals [J]. arXiv preprint arXiv:2406.04249, 2024.\n\n3. The paper should include generalization validation on larger datasets beyond facial or specialized medical regions, such as for natural images in tasks like super-resolution and compressed sensing. It demonstrates both the efficiency and generalization ability of the proposed method across more diverse and realistic scenarios.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes UGoDIT: a DIP-based method that uses a small number of corrupted measurements for training. The method uses  M corrupted images at train time to learn one shared-encoder + M independent decoders, with the additional autoencoder constraint (this is different from DIP). At test time, a new sample is optimized by solving for a new deoder (and keeping the original encoder as fixed).\n\nThe method is compared to other variants of DIP, as well as to end-to-end and generative methods trained on large sets of clean images across compressed sensing MRI, super-resolution, and non-linear deblurring. Results indicate that UGoDIT can match or outperform performance of all other approaches.\n\nStrengths and Weaknesses\nStrengths:\nThe paper is easy to follow and the idea is presented in a conceptually simple way. The work contextualizes the different options depending on the amount/quality of available training data, and compares to a number of these combinations. Results are presented over several different imaging inverse problems tasks, including out-of-distribution experiments. Several ablation experiments are performed and discussed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is easy to follow and the idea is presented in a conceptually simple way. The work contextualizes the different options depending on the amount/quality of available training data, and compares to a number of these combinations. Results are presented over several different imaging inverse problems tasks, including out-of-distribution experiments. Several ablation experiments are performed and discussed. \n\nWeaknesses:\n- It seems like the main contribution of the work is to use a shared-encoder with M decoders at test time. The auto-encoding self-consistency was previously proposed by aSeqDIPin Ref 24.\n\n- The justification on page 5 for the self-consistency is not really clear. Unless I am misunderstanding, the minimum of Eq 2. is z_i = A^T y_i, i.e. do nothing. It is not really clear how this formulation avoids the need for early stopping, which is the original issue with DIP.\n\n- The key idea of the method seems to be unrelated to the fact that the training is over corrupt images. For example, and as another form of ablation, what would happen if M clean images were used to train the shared encoder? Wouldn't this represent the best-case learning? In addition, what is the distinction at test time of the new sample? What would happen if it were included as the (M+1)'th sample at training time instead? Would performance improve at the cost of compute?\n\n- There is no mention or comparison to any of the works that use measurement splitting as a form of self-supervision, even though some of these have been directly applied to DIP. Like my comment above, it seems to me like measurement splitting could be used here as well.\n\n[1] Noise2Self: Blind Denoising by Self-Supervision  Joshua Batson, Loic Royer \n\n[2] Self-supervised learning of physics-guided reconstruction neural networks without fully sampled reference data\nBurhaneddin Yaman, Seyed Amir Hossein Hosseini, Steen Moeller, Jutta Ellermann, Kâmil Uğurbil, Mehmet Akçakaya\n\n[3] Zero-Shot Self-Supervised Learning for MRI Reconstruction  Burhaneddin Yaman, Seyed Amir Hossein Hosseini, Mehmet Akçakaya\n\n\n- There is no discussion regarding test-time training methods, such as the following reference:\n[4] Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing Mohammad Zalbagi Darestani, Jiayu Liu, Reinhard Heckel Proceedings of the 39th International Conference on Machine Learning, PMLR 162:4754-4776, 2022. \n\n- It seems like the training requirements scale with the number of images used. This is not discussed as a limitation.\n\n- It is unclear why the method performs better than methods trained on order of magnitude clean images. If this is indeed the case, then this should be emphasized and discussed.\n\n\n- no details are shared on how the M images were picked. this seems important given the message of appendix  A\n\n- Appendix F figure 13 does not seem to indicate that increasing M helps. While the mean increases, the variance does not always shrink, there are outliers, and none of the results would likely be significant. Furthermore, the difference is less than 0.5 dB. The plot also starts at M=4 (what does it look like for M=1,2,3 ?)\n\nAppendix I: Very few details are given regarding the training setup. The paper does not describe how the train/test sets were formed (are only 20 images used for validation?) The code does not indicate what should be run to replicate the different results, and does not appear to include any of the MRI experiments. Regarding the MRI experiments, how were the slices chosen for training and testing? Appendix I makes it sound like different slices from the same volume were used in training and in testing? Is that the case?\n\n- Almost no images are shown except in the appendix (and all MRI images in the paper are upside-down). The few that are shown are very small; Are the ROIs of Fig 16 the same central image just copied onto the top-left? It would be more helpful to make the ROI larger and increase the scale of the error image.\n\nQUESTIONS:\nPlease refer to the Strenghts and Weaknesses section for a list of questions and concerns. Specifically, my main questions are regarding training on clean vs. corrupt, comparison to measurement splitting, and more training details.\n\nLIMITATIONS:\nPlease refer to the Strenghts and Weaknesses section for additional limitations that should be discussed. Specifically, training memory scales with number of reference images at test time.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces UGoDIT, which uses a shared convolutional encoder ( encoding set of layers of a DIP U-Net) and disentangled convolutional decoders (decoding set of layers of a DIP U-Net). During the training phase, UGoDIT reconstructs $M$ measurements, thereby capturing a data-driven prior and learning transferable weights beneficial at test-time. At test time, the pre-trained encoder is kept frozen while optimizing the decoder parameters in order to recover a clean / noise-free image. The approach is evaluated on accelerated MRI reconstruction, super-resolution, and non-linear deblurring tasks and shows superior performance over existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__:\n\n1. The paper addresses overfitting to noise and high computational cost at inference for deep image prior-based methods by learning generalizable features in the shared encoder layers, resulting in better reconstruction quality.\n2. The paper provides a thorough and extensive evaluation. It includes comparisons with relevant baselines and provides rigorous ablation studies.  A clear empirical effect of design decisions is presented in the paper.\n3. The papers include experiments to highlight the method’s effectiveness and demonstrate competitive, if not better, performance while using subsampled/degraded to baseline methods, which use fully sampled / clean data.\n\n__Weakness__:\n1. Lack of theoretical justification: While the paper shows empirical evaluations, it fails to mention any theoretical insights or analysis as to why sharing the initial encoder layers would learn better feature learning. The authors should detail a more rigorous theoretical intuition beyond what's currently mentioned in section 3.2. \n2. Bias in training set: From Figure 15 (supplementary), it seems that the training set and test set exhibit a bias towards properties such as skin tone in face images, which may lead the shared encoder to learn favorable features. Further, MRI examples also include similar structure and intensity patterns. The paper does not discuss the performance of UGoDIT when the training set differs substantially from the test set, while comprising a high variance in structure and texture of samples within the same class/task.\n3. The paper is overly verbose and lacks brevity. The paper can perhaps tidy up the repeating equations across the main text and further refine the overall process of UGoDIT and its underlying intuition.\n4. Small test sets: The paper tests the approach on 20 images in the test set, which does not convey sufficient information to assess the approach's generalizability and superior performance over baselines.\n\nQUESTIONS:\n1. Can the authors provide more analysis on the out-of-distribution capabilities of UGoDIT? \n2. Can the authors also provide visualizations or insights on the quality of features learned in the shared encoder?\n3. How does the set of training images impact reconstruction quality? For example, Figure 15 (supplementary) exhibits a clear bias towards skin tone in training and evaluation sets.\n4. Can the authors comment on the stability and impedance to overfitting noise over a long number of iterations of UGoDIT?\n\nLIMITATIONS:\nYes. However, please include Limitations in the main paper and not in the supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}