Title: Masked Pre-training Enables Universal Zero-shot Denoiser
Venue: NeurIPS 2024
Accepted: yes
Keywords: image restoration

image denoising，self-supervised learning
Abstract: In this work, we observe that model trained on vast general images via masking strategy, has been naturally embedded with their distribution knowledge, thus spontaneously attains the underlying potential for strong image denoising.
Based on this observation, we propose a novel zero-shot denoising paradigm, i.e., $\textbf{M}$asked $\textbf{P}$re-train then $\textbf{I}$terative fill ($\textbf{MPI}$).
MPI first trains model via masking and then employs pre-trained weight for high-quality zero-shot image denoising on a single noisy image.
Concretely, MPI comprises two key procedures:
$\textbf{1) Masked Pre-training}$ involves training model to reconstruct massive natural images with random masking for generalizable representations, gathering the potential for valid zero-shot denoising on images with varying noise degradation and even in distinct image types.
$\textbf{2) Iterative filling}$ exploits pre-trained knowledge for effective zero-shot denoising. It iteratively optimizes the image by leveraging pre-trained weights, focusing on alternate reconstruction of different image parts, and gradually assembles fully denoised image within limited number of iterations.
Comprehensive experiments across various noisy scenarios underscore the notable advances of MPI over previous approaches with a marked reduction in inference time.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel zero-shot image denoising method named Masked Pre-train then Iterative fill (MPI). This method leverages a pre-trained model on vast natural images using a masking strategy to learn generalized image distributions, enabling effective denoising without prior knowledge of the specific noise type.\n\nSTRENGTHS:\n- Novel and sound idea, with clear benefits.\n- Good generalization performance for unseen noise types.\n- Thorough analysis of their method and results.\n\nWEAKNESSES:\n- Regarding real noise removal, it is well known that spatial correlation of the real noise makes pixelwise masking based methods like N2V or N2S to fail at denoising, making them inappropriate comparatives. Several self-supervised denoising methods have been designed to remove real world noise (e.g., AP-BSN [1], MM-BSN [2], etc.). While these are not zero-shot methods, zero-shot modifications of these blind spot networks would serve as better comparative methods, especially since the authors have already modified N2N and N2S into zero-shot versions.\n\n- Understanding and discussion of masking ratio can be improved. Perhaps employing a random masking ratio within a certain range (e.g., $p\\in[0.3, 0.6]$ would be more effective and could result in an optimal point that is applicable to both synthetic and real noise situations. And why does the masking ratio differ for different noise types? What is the main factor in choosing the best masking ratio?\n\nAlso, if this masking ratio is a crucial hyperparameter that changes depending on the image and noise types, it could weaken the paper's key contribution regarding its generalization ability for any noise types.\n\n.\n\nReferences\n\n[1] Lee, Wooseok, Sanghyun Son, and Kyoung Mu Lee. \"Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Zhang, Dan, et al. \"Mm-bsn: Self-supervised image denoising for real-world with multi-mask based on blind-spot network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\nPlease check weaknesses.\n\n1. What is the main reason that higher masking rate $p$ is required for real-world noise?\n\nMinor points and typos\n   - I cannot see any difference in the images in Figure 2, even for the noisy image. Same to Figure 6, it would be better if there are enlarged views.\n   - Missing space in line 89 i.e.Masked -> i.e. Masked\n   - typo in line 248? forward pass 2.3 ?\n   - Maybe a mistake? In the caption Table 4, what does the `defaults in gray` mean?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method that could handle image denoising regardless of the noise types and intensity. To achieve this goal, the proposed method includes two crucial steps: first, the model will be pretrained on a large amount of images (with masking); second, the pretrained model will be fine tuned on the given noisy image so that denoising. The authors validate the efficiency of their method on images corrupted by different noises.\n\nSTRENGTHS:\n1) The paper is well written and it is easy to follow.\n2) The authors have conducted experiments on different types of images including natural images and medical images.\n3) In addition to synthetic noises, the authors also conducted experiments on real-world noise.\n\nWEAKNESSES:\n1) The idea sounds very trial and the novelty is limited. It is similar to MetaDIP[ref1] and DGP[ref2].\n\n2) The idea of \"ensemble for total T steps\" is also proposed by DIP. DIP has shown that by averaging the outputs, the performance could be improved. So, the question here is: when the authors compare the performance with DIP, have you applied this similar \"average smoothing\" to DIP's outputs? If not, the comparisons here may not be fair.\n\n3) For comparisons, the authors may consider other DIP models such as Ref3.\n\n4) How this proposed method compare with other SOTA models such as Diffusion Models Ref4.\n\n\nRef1: Zhang, Kevin, et al. \"MetaDIP: Accelerating deep image prior with meta learning.\" arXiv preprint arXiv:2209.08452 (2022).\n\nRef2: Pan, Xingang, et al. \"Exploiting deep generative prior for versatile image restoration and manipulation.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.11 (2021): 7474-7489.\n\nRef3: Jo, Yeonsik, Se Young Chun, and Jonghyun Choi. \"Rethinking deep image prior for denoising.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nRef4: Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).\n\nQUESTIONS:\nI have listed my questions in the section of [Weaknesses].\n\nLIMITATIONS:\nI have listed the limitations in the section of [Weaknesses].", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a zero-shot image denoising method called Masked Pre-train then Iterative fill (MPI). The key idea is to pre-train a model on natural images using masked image modeling, then apply this pre-trained model to denoise new images in a zero-shot manner through an iterative optimization process. The authors demonstrate that their approach outperforms existing zero-shot denoising methods across a variety of noise types and datasets, while also being more computationally efficient.\n\nSTRENGTHS:\n1. Masked modelling in image denoising and Iterative processing is interesting.\n2. And the above idea works, it is more interesting.\n3. Somewhat good results.\n\nWEAKNESSES:\n1. I have to say that the writing of this paper is problematic. The method of this paper is not complicated, and it is even a little simple. However, the presentation of this paper does not show its core method. Some of the descriptions are irrelevant, but important analysis such as the core of its method is missing. \n 2. This paper uses the idea of ​​pre-training, but follows zero-shot. A natural question is, what does the model learn from pre-training, and how is it applied to zero-shot denoising? I understand that it is difficult to give a theoretical answer, but I think there is at least an abstract explanation to verify and demonstrate the essence of its method.\n3. Assume that the model learns the distribution of images from pre-training. But according to its experiments, the trained model can be used for unnatural images. If the model learns denoising from pre-training, the model can generalize to other noises to a certain extent. Unless it learns a common property of some noise. The key is, what is this property?\n4. The iterative approach is interesting, and I think the author may have been inspired by Diffusion to a large extent. But it is not enough to simply assume without a clearer theoretical motivation to support this approach.\n5. The choice of denoising is also questionable. In fact, I think the method in this article may be helpful for any image restoration task that meets the conditions. Why do we only discuss denoising?\n\nQUESTIONS:\nIs it possible to combine mask prediction and iterative filling, and actually use the statistical properties of noise to denoise? Make predictions under different masks each time, and integrate multiple predictions through iterative filling, and use methods similar to finding the mean between multiple predictions to achieve some denoising purpose? This is about the working nature of this method. I think it is very necessary to use experiments to prove what the model has learned and why it denoises.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel zero-shot image denoising paradigm called Masked Pre-train then Iterative fill (MPI). The key contributions are:\n\nMPI first pre-trains a model on a large dataset of natural images using a pixel-wise masking strategy. This allows the model to learn the underlying distribution and representations of natural images.\nDuring the zero-shot inference stage on a single noisy image, MPI optimizes the pre-trained model to predict the masked regions, and only uses the predictions of the masked regions to assemble the final denoised output.\nThis approach leverages the generic knowledge from pre-training, preventing overfitting during the inference stage and enabling high-quality denoising with a marked reduction in inference time compared to prior zero-shot methods.\nThe paper demonstrates MPI's superior performance across various noise types and its ability to generalize to medical images, which are distinct from the natural images used in pre-training.\n\nSTRENGTHS:\nOriginality:\nThe paper presents a novel zero-shot denoising paradigm that is significantly different from prior approaches. The key innovation is the use of masked pre-training on natural images to build a generic and robust model for zero-shot denoising. This is an original idea that departs from previous zero-shot methods that relied on carefully designed priors or network architectures. Leveraging masked pre-training for this task is a novel and creative approach.\n\nQuality:\nThe technical details and experimental evaluations in the paper are of high quality. The authors provide a clear and thorough explanation of the MPI framework, including the masked pre-training and iterative optimization steps. The experimental setup is comprehensive, analyzing performance on various noise types, real-world datasets, and even medical images. The results demonstrate significant improvements over prior zero-shot methods, validating the effectiveness of the proposed approach.\n\nClarity:\nThe paper is well-written and easy to follow. The introduction provides a clear motivation and background for the problem. The method section explains the MPI framework in a structured and logical manner. The experimental section is organized in a way that allows the reader to easily understand the different evaluations and findings. Overall, the clarity of exposition is a strength of this paper.\n\nSignificance:\nThe problem of zero-shot image denoising is an important and challenging task in computer vision. Prior methods have limitations in terms of generalization and computational efficiency. The MPI approach presented in this paper addresses these limitations in a novel way. If successful, this could lead to significant practical impact by enabling high-quality denoising with minimal user intervention or computational overhead. The ability to generalize to diverse noise types and even medical images further enhances the significance and potential impact of this work.\n\nWEAKNESSES:\nComparison to other zero-shot methods:\nThe paper primarily compares MPI to a few selected zero-shot denoising methods. However, it would strengthen the work to include a more comprehensive comparison to a wider range of zero-shot techniques, including recent advances. This would help contextualize the performance gains of MPI and highlight its specific advantages over the state-of-the-art.\n\n[1] Xie Y, Yuan M, Dong B, et al. Unsupervised image denoising with score function[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[2] Jang H, Park J, Jung D, et al. PUCA: patch-unshuffle and channel attention for enhanced self-supervised image denoising[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[3] Garber T, Tirer T. Image restoration by denoising diffusion models with iteratively preconditioned guidance[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 25245-25254.\n\nQUESTIONS:\nno\n\nLIMITATIONS:\nsee weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}