Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an early stopping strategy for deep image prior.  This is achieved by using an efficient ES strategy that consistently detects near-peak performance across several CI tasks and DIP variants. The paper provides high level intuition, theoretical proof and multiple use cases to demonstrate the proposed method. Also, the limitation is well discussed. 
strength_and_weaknesses: - Strength
1. The paper is well-written with enough details 
2. High-level intuition is great and easy to follow 
3. Theoretical analysis is provided
4. Multiple use cases in DIP are discussed
5. The limitation is shown in different scenarios. 

- Weakness 
1. The theoretical analysis is not strongly related to the strategy, can you explain why the theoretical helps the early stopping criteria? 
2. The experiments focus on image denoising, MRI reconstruction, and blind deblurring, why choose these three? is that possible to solve other general inverse problems, e.g., superresolution, implanting? 
3. Is that early stopping DIP competitive with the SOTA approaches, e.g., diffusion-based models in these tasks? 
clarity,_quality,_novelty_and_reproducibility: Clarity, quality nd reproducibility are great. 
novelty is not very strong but it is an interesting topic 
summary_of_the_review: Overall, the paper is good for me but I just want to know the answer to the points listed in weakness.  Other than that, the paper is a good contribution to improving the DIP if it is competitive with the SOTA inverse problem solutions
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes an early stopping mechanism for DIP based on the windowed moving variance of the output images. An early stop is detected when the variance does not decrease after a certain number of training steps. The proposed method is observed to have small PSNR and SSIM gaps compared to other early stopping methods and is applicable to different DIP variants.
strength_and_weaknesses: Strength: The proposed method is simple enough and to some extent aligned with the reconstruction quality. Extensive experiments do show the superiority of the the method. And the proposed method is generally applicable to various types of noises.

Weaknesses:

1. In some cases, the PSNR gaps are still very high, up to 5dB. And according to the plots of variance curves, the proposed metho still detects early stopping time that is way off the correct time (with highest PSNR).

2. I strongly recommend the authors to provide some output images to be included in the manuscript, since PSNR is not perfectly aligned with visual quality some times (usually have some smoothening effect). Moreover, the NF-IMQ reference baselines are naturally biased towards better "quality" in their metrics instead of PSNR. Therefore, solely comparing PSNR-gaps might not be quite fair.
clarity,_quality,_novelty_and_reproducibility: This work is clear and easy to follow with obvious novelty because it proposes a new, effective most of time, and easy enough early stopping mechanism for DIP.
summary_of_the_review: I hold an overall positive view on this work for its technical novelty and empirical contribution.

===================================

Update after the rebuttal discussion:

After reading comments from other reviewers and the feedback from the authors, I can relate to the concerns of other reviewers that the ad-hoc manner of the proposed method, especially on the hyperparameter configuration of the patience time, creates another "early-stopping" problem. This issue is more severe considering the lack of theoretical analysis and essentially undermines the practicality of the proposed method. Co-considering this and the underwhelming performances from case to case, I decided to adjust my evaluation negatively towards this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper solves the early-stopping problem for deep image prior by tracking the variance of reconstructed images within a moving window during training. It shows that early stopping when the variance reaches the minimum results in the best deep image prior. 
strength_and_weaknesses: Strength: 
- The paper demonstrates the early-learning pheromone of the deep image prior both empirically and theoretically. 
- A broad range of experiments are conducted, including different applications (image denoising, deblurring, MRI reconstruction)

Weakness:
- A sensitivity analysis on hyperparameters (patience number and window size) is conducted, but it's unclear which dataset it that. Figure 3  shows the pattern of training curves looks really different across different noise levels or different DIP methods/variants. Are they using the same hyperparameter?  Will the performance still be stable in that case?
- The experiments are not systematically conducted. The paper lists a lot of related works, but fails to organize the experiments in a logical way. Some of them are benchmarked on one dataset, while others are on another. What DF-STE is? Why has it only experimented on the nine-image dataset, not CBSD68? 
- The improvement compared to SB is marginal. The gaps are all within one standard deviation.
- Lack of other denoising baselines, like noise2self, self2self and etc.
- How does the training loss look compared to the other training curves? Does the low variance part look flat? Can you tell anything about early-learning from the loss curve?
clarity,_quality,_novelty_and_reproducibility: The motivation and methodology of the paper are clear. However, the experiment section is hard to follow. The method is novel, but the experiments don't show a significant improvement with the method. 
summary_of_the_review: The paper explores an interesting early-learning phenomenon in DIP and proposes a novel early stopping method to optimize DIP. However, the experiments are not well-organized to support the claims. The organization and writing of the paper need to be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an Early Stopping (ES) method for Deep Image Prior (DIP) based on the running variance of the intermediate outputs. For the task of denoising, a broad set of experiments is provided to study it under different noise types and intensities and against other approaches for early stopping. Besides, some experiments showing how it can be utilized in blind image deblurring and MRI reconstruction are provided.
strength_and_weaknesses: - S1: The motivation and justification of using the proposed metric for ES is discussed in details.
    
- S2: Providing extensive experiments on denoising and showing robustness to different noise types and higher intensities of noise.
    
- S3: Outperforming non-reference based image quality metrics and other methods of ES for denoising specifically designed for DIP either in terms of average quality or wall-clock time per iteration
    
- S4: Proposing a lighter variant of the algorithm to save on computations per iteration
    
- W1: The method does not perform well when an image consists of too much high-ferequency components.
    
- W2: Based on Figures 3, 8a-b, and 17, it seems like in practice the method is prone to higher error or failure when the changes in the PSNR curve are relatively smooth or when the noise level is low.
    
- W3: The experiments are imbalanced and mainly focused on denoising. For example, for blind image deblurring a limited subset of the dataset and/or kernels are considered and the the results in case of the default noise levels used in the original Ren et al. (2020) solution are not provided.
clarity,_quality,_novelty_and_reproducibility: - Concern regarding the significance of the results:
    
    1) One aspect of measuring the significance of a work is how it performs against the SOTA methods. However, this paper has only focused on the performance gain over the solutions for the family of DIP and ignored considering SOTA methods in general, either supervised or unsupervised.
        
- Clarification questions:
    
    2) For Table 2, it is not clear if the wall-clock time of the three ES methods should be added to DIP’s to achieve the total wall-clock time per iteration? Or that ES-WMV has some optimization resulting in lower wall-clock time per iteration when used with DIP, compared to only using DIP?
        
    - For the experiment on image deblurring:
        
        3) How are the overfit iteration numbers chosen?
        
        4) Why only a subset of images/kernels are chosen, and what was the criterion to choose them?
- Minor point:
    
    5) The references are not written in a consistent manner. For example, few times the full name of a conference is written and at other times its abbreviation such as NeurIPS. Also, sometimes the URL and/or DOI are provided and sometimes they are not. 
summary_of_the_review: The empirical results seems to be well-aligned with the justification of the proposed method. The performance is generally improved on denoising compared to previous ES for DIP methods and the experiments on this task are extensive. There are some situations such as too much high-frequency components or too much smoothness in the optimization curve that could affect the result negatively. While the idea of tracking the running variance of the outputs seems to be simple, I have not seen it used for ES in DIP. The significance cannot be properly measured when only looking at family of DIP methods and not including other SOTA methods for denoising. The situation is worse on the other tasks as the comparisons are far more limited and does not even include latest methods based on DIP, for example for deblurring, not to mention the rest of SOTA methods.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Authors propose a technique to detect peak performance without a reference for Deep Image Prior (DIP) based computational imaging algorithms.  The proposed strategy utilizes the running variance of past reconstruction iterates to determine when to terminate training. Authors provide theoretical results to justify the proposed early stopping criterion. Through numerical experiments, they demonstrate that the proposed method can detect peak performance reasonably well across a wide range of inverse problems and types of measurement noise.
strength_and_weaknesses: Strengths: 

-  The paper is very well-written and organized. It is easy to follow and the structure is logical. 

- The problem the paper is investigating is a very relevant one and therefore this paper is well-motivated.  

- The proposed technique can be combined with any DIP-based method, is simple to implement and the overhead in terms of computation is minimal.  Therefore, it has potential in practice.

Weaknesses:

- I am not completely satisfied with the justification of tracking running variance to detect peak performance. The theoretical justification doesn't seem to be too convincing, as the provided bounds are very loose, and having a U-shaped upper bound does not necessarily mean that the running variance will also be U-shaped. Even though the experiments show that the metric works in some cases, it is not clear to me why variance should be minimized close to the solution and increase during overfitting. 

- Based on the experimental results, the proposed technique is somewhat inaccurate in low-level noise scenarios (Figure 3 top row) and often surpassed by other methods (Table 3, Figure 6, Figure 7). A more in-depth study on the noise regime where the proposed method is reliable would be very useful.

- The proposed method has a significant memory overhead. Storing W=100 iterates in some applications may be prohibitive. The proposed EMV direction is interesting and can potentially address this issue, however currently there are not enough experiments on this variant to fully support its merit. 

- Most experiments are on very small datasets (1-9 images). Evaluation on larger datasets would help filtering out statistical variations in the presented results. 
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is easy to follow and clear in most parts.
Quality: I have some concerns with respect to the empirical and theoretical support of the proposed method (see Weaknesses above).
Novelty: to the best of my knowledge the proposed early stopping scheme is novel for Deep Image Prior.
Reproducibility: code is provided to reproduce the experimental results.

Minor suggestions:

- Typo in 'Prior work addressing the overfitting': 'noisy type and level'

- Typo in 'Intuition for our method': 'more details...'

- Possible typo in Theorem 2.1: what does index m denote in C_{m, \eta, \sigma_i}? It should probably be W. 

- In Algorithm 2 in the appendix VAR is not defined.
summary_of_the_review: Overall, in its current form I would recommend borderline rejecting the paper. The proposed method shows promise and works reasonably well across different types of additive noise. However, in my opinion the justification for the particular early stopping metric is somewhat lacking. The provided theory doesn't provide a satisfying answer either. Could the authors provide more explanation and further backing why running variance would show the U-shape phenomenon close to the ground truth signal? Furthermore,  the experimental results seem to show that the proposed technique is mostly reliable in high-noise scenario and a detailed investigation with respect to noise level limitations is missing. Could the authors provide more discussion on the aforementioned limitations?
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed an empirical technique to detect peak DIP performance without a clean reference image. The proposed strategy utilizes the running variance of past reconstruction iterates to determine when to terminate training. The observations are certainly interesting. The paper is very well-written and easy to follow.

Two factors hold this work from being more positively assessed. On one hand, the method's performance is underwhelming sometimes (the gap can be really big, even not compared to the strongest SOTAs). The authors explained they "have not vigorously optimized the patience number, or at a higher level, the valley-detection method itself". But that brought up further concerns about how ad-hoc and sensitive their method (detection part) could perform in the presence of real and varying noise. On the other hand, as the authors also agreed in their rebuttal, the theoretical contribution of this paper is rather thin and incomplete. It is also not strongly related to the algorithm.


justification_for_why_not_higher_score: While the theoretical contribution isn't major, the empirical benefit is sometimes unobvious either.
Decision: Reject