Masked Pre-training Enables Universal Zero-shot Denoiser
Authors: Xiaoxiao Ma
Zhixiang Wei
Yi Jin
Pengyang Ling
Tianle Liu
Ben Wang
Junkang Dai
Huaian Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: image restoration
image denoising，self-supervised learning
Primary area: other
TL;DR: An efficient yet novel approach for zero-shot denoising

Abstract: In this work, we observe that model trained on vast general images via masking strategy, has been naturally embedded with their distribution knowledge, thus spontaneously attains the underlying potential for strong image denoising.
Based on this observation, we propose a novel zero-shot denoising paradigm, i.e., $\textbf{M}$asked $\textbf{P}$re-train then $\textbf{I}$terative fill ($\textbf{MPI}$).
MPI first trains model via masking and then employs pre-trained weight for high-quality zero-shot image denoising on a single noisy image.
Concretely, MPI comprises two key procedures:
$\textbf{1) Masked Pre-training}$ involves training model to reconstruct massive natural images with random masking for generalizable representations, gathering the potential for valid zero-shot denoising on images with varying noise degradation and even in distinct image types.
$\textbf{2) Iterative filling}$ exploits pre-trained knowledge for effective zero-shot denoising. It iteratively optimizes the image by leveraging pre-trained weights, focusing on alternate reconstruction of different image parts, and gradually assembles fully denoised image within limited number of iterations.
Comprehensive experiments across various noisy scenarios underscore the notable advances of MPI over previous approaches with a marked reduction in inference time.

Masked Pre-training Enables Universal Zero-shot Denoiser
Xiaoxiao Ma Zhixiang Wei Jin Pengyang Ling": 1.2 Tianle Liu'
Ben Wang' Junkang Dai' Huaian Chen
1 University of Science and Technology of China Shanghai AI Laboratory (xiao_xiag chixianguei,, 1pyang277 tleliu, wblzgrsn, wblagren..unkaagddaaa anchen) Smail ust..edu.... (jinyi08) @ustc. edw..cduccc. cn
Aug. Inflrr Time 60
Swdellt
Gazzaza
101
10
10
B2U
Noer
DIP
Dus
75-N0N
LETT-ETT
Oum
Priicct
S&P
SCOR SCOP
PIGHIPP
ICVPRIRS
Noisy
DIP
Ous
JOVPA231
(CVP9120)
XLF
Speclle
(a) Computational costs Parformances
(b) Generalization
(c) Roal-wordd & Medical noise removal
Figure 1: (a) Ours surpasses current zero-shot methods with reduced inference time (on CSet with Gaussian a=25, see Sec. 3.2p. (b) shows better generalization across different noise types than current zero-shot & supervised/unsupeviis correlated real-world noise, results are from SIDD benchmark and FMD (Sec. 3.4 Sec. 3.5
methods (Sec. (.3). (c) And can remove spatial
Abstract
In this work, we observe that model trained on vast general images via masking strategy, has been naturally embedded with their distribution knowledge, thus spontaneously attains the underlying potential for strong image denoising. Based on this observation, we propose novel zero-shot denoising paradigm, i.e., Masked Pre-train then Iterative fill (MPI). MPI first trains model via masking and then employs pre-trained weight for high-quality zero-shot image denoising on a sin- gle noisy image. Concretely, MPI comprises two key procedures: 1) Masked Pre-training involves training model to reconstruct massive natural images with random masking for generalizable representations, gathering the potential for valid zero-shot denoising on images with varying noise degradation and even in distinct image types. 2) Iterative filling exploits pre-traie pre-trained knowledge for effective zero- shot denoising. It iteratively optimizes the image by leveraging pre-trained weights, focusing alternate reconstruction of different image parts, and gradually assem- bles fully denoised image within limited number of iterations. Comprehensive experiments across various noisy scenarios underscore the notable advances of MPI over previous approaches with a marked reduction in inference time. Code available Inttt:tt////// //t:::///titi giggt...b... com/ kr aerennicgg9//10
"qqual contribution. 'Correspondng author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Image denoising 3 as a branch of image restoration, has been the subject of extensive exploration.
The prevalent approach to restore degradd images is learning from multiple noisy instances. Nonetheless, both supervised learning from noisy-clean pairs [5 and unsupervised training [8- [0] necessitate the collection of additional noisy datasets. Moreover, such methods may foster dependencies on specific patterns or intensities of training noise, hindering their performance in unfamiliar noise situations
11.12]
12
As an alternative, zero-shot approaches 13 15 attemnpt to train network on a single noisy image for denoised output, negating the need for additional noisy data collection. Dedicated to obviating concerns about generalization issues, these techniques include blind-spot networks that reconstruct
from corrupted inputs [16 17]. DIPs [33 18 2T which exploit the characteristics of deep networks to learn the mapping from random noise to noisy images, as well as sub-sample based strategies [22 23 utilize spatial correlations to generate training pairs from sub-sumpee samp-ed instances.
However, current zero-shot methods train new networks from scratch for each noisy image, which presents two major issues: 1) Despite success in current zero-shot approaches rely on regularization
or designed priors such as noise perturbations [33. under-paraaetterized networks [8] [1]22]. dropout-
ensemble (14) and blind-spot networks [16). the limited information from single image to train
network often lead to overly blurred content, noise artifacts or sub-optimal quality, Several methods
tend to rely on known noise distribution [3] 20 24] for more information, but their applicability is
limited. 2) Training a new network from scratch for each noisy image is time-consuming. Existing zero-shot methods typically require several minutes [3] or more [14] And attempts at faster
zero-shot denoising [2 23 often compromise on performance.
Compared previous zero-shot approaches, learning the feature distribution from vast natural images offers a more intuitive approach. This is grounded in two considerations: Real natural images are both abundant and readily available, and despite variations in noise patterns, many natural images share common characteristics (25). We seek to enhance zero-shot denoising with minimal reliance on pre-defined priors or regularization, aiming for a better startpoint for various noise patterns instead of from scratch. To this end, we delve into the potential of masked image modelling [26 27] on natural
images with no assumptions about noisy patterns and intensities (28). Specifically, we make the
following observation: combined with simple ensemble operation, masked pre-trained model
can naturally denoise images with unseen noise degradation.
Building upon above observation, we introduce a zero-shot denoising paradigm, i.e., Masked Pre-train then Iterative fill (MPI). MPI first pre-trains model on ImageNet with pixel-wise masking strategy, then the pre-trained model is optimized on a single image with unseen noise for denoised prediction in zero-shot inference stage. The optimization goal in inference is designed to predict masked regions, and only predictions of masked areas are preserved for denoised prediction, thereby minimizing the gap between pre-training and inference. The pre-trained weights provide more generic knowledge, preventing premature over-fitting during inference and reducing the need for strong regularization. We are able handle wider range of noise scenarios with less information about noise patterns or intensities. Remarkably, we find that extracted representation can even generalize to medical images that distinctly different from natural ones [2]. also offers better startpoint than scratch training, enabling high-qualit denoising around 10 seconds, underscoring the potential of our method in practical application. The main contributions of this paper are as follows:
We introduce a novel zero-shot denoising paradigm, i.e., Masked Pre-train then Iterative fill (MPI, which introduces masked pre- trainini in this context for the first time, simultaneously improving both image quality and inference speed on unseen noisy images.
We develop a pre-training scheme with pixel-wise wise random masks to capture distribution knowl- edge of natural images. Based on pre-trained knowledge, we propose iterative filling for zero-shot inference on specific noisy image. This process is optimized using pre-trained weights, and focuses on alternatively reconstruct different parts of noisy image, predictions in iterations are sequentially assembled for high-quality denoised output with efficiency.
Extensive experiments demonstrate MPI's superiority, efficiency and robustness in diverse noisy scenarios. In nutshell, MPI achieves significant performance gains across various noise types with reduced inference time, highlighting its potential for practical applications.

---

2 Methods
In Sec. 2.1 we first investigate the properties of models trained with masking, proving that models trained with masking strategy learn representations beneficial for denoising. Our observation lead us to propose a zero-shot denoising paradigm that includes pre-training (Sec. 2.2) and iterative optimizing (Sec. (.33). We further illustrate how remove spatially correlated real noise in Sec. 2.4
28
26
24
0.5
20.4600.55
29.01007233
3144 in0860
0.4
Noisy| Reference
Directly ensemble
Zero-shot Optim.
20
Noisy image
Directly ensemble
+Zero-shot Optim.
Figure Example of model trained ImageNet with 70% pixel-wise masking, denoised image is obtained by directly ensemble of predictions from fixed pre-trained weights ("Directly ensemble"), its performance can be further im- proved with iterative filling ("+Zero-shh Optim.").
Figure 3: Evaluation on an ImageNet subset shows pre-trained model's in- herent denoising ability, but perfor- mance limited without optimization.
2.1
Motivation
has significantly advanced computer vision by training vast natural image sets to grasp their knowledge distributions. It shows great potential applicability under diverse scenarios and have been proven beneficial for high-level downstream tasks (1)
Masked Image Modeling 266 27 29
To further explore its capability in denoising, we train a model on natural images with pixel-wise random masks (for details, Sec. 2.2 and assess its performance against target image with unseen noise distribution. Surprisingly, we observe that a simple average of predictions from a fixed-state trained model can denoise unseen noise, shown in Fig. 3] sometimes achieve remarkably good performance, as an example is presented in Fig. 2 This observation suggests that a masked pre-trained model can serve which can be attributed to lack of knowledge about specific degradation patterns in the target image.
natural image denoiser. However, artifacts exist in the results,
Drawing prior insight, we develop efficient zero-shot denoising pipeline, leveraging pre-trained knowledge by incorporating noise characteristics from a single noisy image (Fig. 4), i.e., Masked Pre-train then Iterative fill. The model is firstly pre-trained with random masks M and corresponding element-wing wise negation M to acquire natural image distributions, formulated as:
arg max p(1 @ MI @ M;0),
for I indicates natural image without any degradation priors, typically sourced from extensive datasets (e.g. ImageNet (28)). use element-wise multiplication (0). For denoising on specific noisy image pre-trained parameter 0 is loaded and further optimized with known from 1=1 to I=T for T iterations, and predictions are aggregated for final prediction y:
FnssemleeDD, (a))
where Da, (.) is network parameterized by 0,, optimized from pre-trained 0. Masked Pre-training process is detailed in Sec. 2.2 and Ensemble Sec.
2.2 Masked Pre-training
Masking strategy. Given the distinct requirements between low-level and high-level tasks in "seman- tics" [32], we implement specialized masking strategy to achieve finer -reiieg image representations, ie., a pixel-wise masking strategy. Specifically, given an input image E Hx xc" divided into
random patches of size 1, subset of them are randomly replaced by mask token with probability (for further discussion of p. see Sec. 4. When the mask token set to 0, the masked image M O
with random mask M E F xxxxxx CC corresponds bernoulli sampling of the input image I. For
3

---

1. Pre-training (Krowldgee extracioo)
Ensemble
D
Forwart process
Loss calculation
Natural Images
Masked
Reconstructed
Flement wise add
Iterative filling Caaossh
Timestap I
Masked
Iritiialine when t=]
Reconstructed
M, x
M, O
Prediction
Do
Encenbble
diggs
Figure 4: An overview of the proposed MPI paradigm consisting Masked Pre-traing Pre-training and Iterative filling. During pre-training Do(-) learns to reconstruct masked natural images, And the pre-trained weights 8 are saved for zero-shot denoise, i.e., Iterative filling, to denoise specific noisy image During zero-shot inference, network is initialized with pre-trained weights A, then the weights are further optimized on for T steps, results from f-th (t=1,2,... TT-). optimizing steps are gathered to obtain final denoised prediction y. Compared to current zero-shot methods, just adding one more step to load pre-trained model enables faster and high-quality zero-shot denoising.
each element MR² in M, we have:
-
0, with prob. p:
Ma
1,
with prob. 1 p.
Pre-training scheme. During pre-training, the network Dg(-) trained learn recovering natural image / itself with random mask M:
= Da(M @ 1).
(4)
We set the same optimization strategy outlined in (27). focusing loss computation masked prediction areas f. This directs network efforts towards reconstructing these specific regions, with the reconstruction loss denoted Trec:
Lree (I. = |
(5)
The Mean Squared Error (MSE) loss is adopted to learn relatively smoother representation. For the architecture of network D--). we employ the same U-shaped hourglass architecture as in DIP (I3), which has been proven powerful zero-shot denoising architecture [19]. Furthermore, its rela- tively small parameter configuration enables accelerated training, alleviating potential inference computational costs and rendering more appropriate for zero-shot denoising tasks.
Algorithm 1: Iterative filling. Pipeline designed to leverage pre-trained representation 0 for zero-shot denoising.
Input:
Noisy image æ, pre-trained parameter 0, network D(-), exponential weight 3, masking ratio p.
Output: denoised ensemble from predictions of iteration {y:).
load pre-trained parameter 9 for D(.) as 01 initialize
for / from to T do
generate random mask M, with mask ratio p
V1 Da, (M, @ x)
M₁ = M
0++ 0: Vo
Mr à y/
Mr 2
7 + M, 3 (B 7 + +(1 B) y/ + M, T
return y

---

Ganesiant of 25
Referme PSNRVSSIM
DIP
NZV
N2S
ZS NZN 29.1077772
DIP 28.0000 765
Ouas (Caster) 2806300
Ouss 30,2000.00
2012300304
29.7520.776
2200
20/31/07311
Poisson 2-25 192200.044
Refernee
DIP
NZV
N2S
ZS- NZN 79.34/0.775
FIP
Ous (Caster)
Ous
PSNR/SSIM
30,47/0,228
2945
27/65/1722
29,300.005
30.341/945
35/559
Figure 5: Qualitative denoising results on Gaussian and Poisson noise. The quantitative PSNR/SSIM results are provided underneath. Noisy patches are from CBSD-44 and McMaster-14, respectively. Best viewed in color (zoom-in for better comparison).
Table 1:
Quantitative comparison on CSet, McMaster & CBSD for Gaussian noise removal. Best results highlighted and second underlined. See Supp. for poisson noise removal.
DIP [3
N2V 16
N2S*
[17
ZS-N2N 22
FisterDIP 19
Ours (faster)
Ours
10
32.05/0.822
829
31.55/0.885
28.04/0.819
33.8770.883
31,59/0.815
33 8890.889
34,91/0.909
CSet
30.42/0.795
29.39/0.814
28.19/0.777
29.55/0.765
30.19/0.766
30.83/0.824
31.61/0.841
50
24.73/0.533
27.35/0.694
26.62/0.699
16.10/0.624
26.09/0.669
14/0...5
28.26/0.710
10
32.48/0.878
30,98/0.877
28.61/0.839
34.19/0.908
31.48/0.84 842
34.35/0,921
35.46/0.937
McMaster 33
25
31.07/0 856
29.11/0.835
27 17.9990777 776
29.37/0.78 786
29 29.4770.799
30.99/0.862
31.90/0.879
25.72/0.639
24.65/0.676
24.89/0.673
25. 25.82/0.634
24.75/0.663
28. 15/0.797
28.37/0.770
31. 18/0 865
11.1890.918
28.17/0 853
33.73/0.922 923
30,89/0 857
34.20/0990 933
35. 15,14/0,947
CBSD 34
25
29. 199.29/0.80 828
27.51/0.812
26.93/0.796
29.01/0.815
28.57/0. 806
30.00/08854
30.58/0.865
23.06/0.540
25.74/0.700
24.78/0.695
25.37/0.657
24.75/0.669
27.05/0.712
26.85/0.703
Avg. Infer. time (s)
451.9
153.9
147.9
16.8
149.2
10.1
$1.6
2.3
Iterative Filling
Overall design. As observed in Sec. 2.1 an iterative optimization process is designed to leverage pre-trained knowledge for zero-shot denorsing. Unlike other MIM approaches [26] 27 that fine-tune with entire images as input, since only one noisy image is accessible, we employ a self-supervised manner to learn the mapping from noisy image itself. However, this direct self-mapping approach introduces significant gap between the zero-shot inference stage and pre-training stage and lacks constraints for learning a noise identity mapping.
Considering above challenges, we retain the same masking strategy in Sec. 2.2 for both input and loss computation, i.e., network still learns to reconstruct masked regions, but from single noisy image rather than natural images. This leads to a pixel-based iterative refinement process, which resembles mechanism of blind-spot networks [66.. Specifically, for input noisy image æ, random mask M, and its element-wise negation M, in t-th iteration, prediction ye and aggregated result ÿ can be derived:
1/2 Do,
(P,,Mr (Mr x);;
(6a)
Σ ④ M.
(6b)
where 9r denotes network parameter at iteration a, is corresponding coefficient where Σ, a, 1. The optimization objective at each iteration is as follows:
M. on - Me o l
arg min #:
The optimization task, represented by Cyee learns to reconstruct noisy image cropped by random masks, aligns with trainign The alignment minimizes the gap between mre-training and zero-shot inference to avoid wver-fitting, and reduces the inference steps required, thus accelerating the denoising process. Thanks to the well-crafted mechanism, we can accomplish high-quality results with preserved details in reduced time without any other regularization.
Pixel-based iterative refinement. For lower mask ratio and reconstruction of more detailed images, abandon constraints on unmasked regions in previous optimization goals (Eq, 5 and Eq. 7 thus
5

---

Speckle p-11
Restormer
DIP 31.7700SS
ZS-N2N 37,55/0.964
Ours 37.78/0.968
DIP 30.92,0,992
ZS-NZN 29,57007732
Oun 32.61/0.85
Figure 6: Qualitative results patches are from kodim07 and kodim12 12.
noise types. Restormer is trained with Gaussian 0=25. Noisy
Table 2:
Quantitative generalization evaluation results on Kodak. All supervised/unsppeviieed methods trained d=25 Gaussian, tested on 5 unseen noise types. (Average from all 6 settings.)
Supervised
Unsupervised
Zero-sbot
Test Noese
Restormet 36
10
39
ZS-N2N 22
(faster)
Gaussian cw25
12.89/1.895
33,040.8977
32.06/ 880
32.2/0.880
30.05/0.800 806
29 29.46/0.775
30,94/0.848
7857888866
Gaussian ec[10,50]
27.29/0.628
30.00/0.729
28.68/0.713
29.24/0.726
39.5640.783
29.36/0.753
20.89/1.837
31.66/0.446
Poisson 1c
25 06.066622
26.52/0.683
27.31/0.703 703
28.22/0.118
28.63/0.758
28. 17/0.7273
29.94/0.226
30.57/08832
NLF from 40
32.52/0.862
31.71/0.857
$1,88/088
31,98/0.839
29.71/0.821
31,02/0.834
32.26408866
33.15/0.901
Speckle ocT10.500
31.9708841
33.52/0.884
31,31/0.837
31.6500.44
30.73M0.818
33.78/0.891
34.79/0.92
35.79/09933
S&P de(00.22.0.05)
23.96/0.614
23.63/0.613
27.040.688
29.44/0.796
29.54/0.800
35.25/0.952
35,05/0.953
36.87/0.964
Average
28.9400.444
29.7300.777
29.71/0.800
30.47/0.804
29.71/0.998
31.17/0.223
32.31/0.879
33.30/0.890
making information under these areas unreliable, we preserve only the results corresponding to for final denoised outcome y. However, one forward pass provide partial denoising results, an ensemble process is crucial. Specially, we employ an Exponential Moving Average (EMA) strategy to optimize the use predictions during iterations with little increase in inference time (Sec.
y
M, O (B 7 + (1 3) we) + M, +
8
For an in-depth look at proposed ensemble algorithm, see Alg. 1 During inference, the pre-trained weights not only provide better startup but also act regularization for the network, preventing from over-fitting too early and leading to better performance with less inference time (Sec.
2.4 Adaptation to Real-world Noise Removal
Real-world noise exhibit strong spatial correlations, i.e. the noise is correlated across adjacent pixels. In such scenarios, employing a straightforward masking mechanism still allows the model to learn information related noise patterns. To address this problem, we apply larger masking ratios than that used for synthetic noise. Additionally, integrate simple Pixel-shuffle Down-sampling (PD) mechanism during zero-shot inference to reduce spatial correlation noise.
Specifically, instead of directly processing noisy image E R xIxCxxXX in Eq. 6a we handle its down-sampled versions Down(x:) E o& xCx f x W using simple Pixel-shuffle with factor d, and d? sub-samples are concatenated along batch dimension for joint denoising. Following the same iterative filling mechanism described above, apply pixel unshuffle to the denoised result ÿ obtain final denoised outcome Up(T). We add minimal PD operations to address spatial correlated noise, illustrating the effect of pre-trained weights, performance real-world noisy dataset can be improved by applying better sub-sampling approaches (35-37] (Sec. 4.2) as they have been intensively studied.
3 Experiments
Noise2 (N2V) 16 Noise2Self (N2S) [17]. Zero-Shot Noise2Noise (ZS-N2N) (22), and FasterDIP [19]. We modify N2V and N2S to single-image version (N2V* and N2S*). EMA ensemble result of DIP and FasterDIP are reported with their official code. Refer to supplementary material (Supp.) for EMA results N2V* and N and comparison with more blsed [20). diffusion-based [4 zero-shot modifications from unsupervised methods [36 43 44, Only non-ensemble 7.S-N2N is presented due negligible performance differences with EMA version. We compare Peak Signal-to-Nofe Noise Ratio (PSNR) and
We assess our method against typical methods including DIP 13

---

R
2
2
2
R
Natag
Refeemma PSNRYSSIM
DIP 37.3940.48
NZV 24N88
N2S 205E2
ZS NZN 2821/0553
Fid 37,28003300
Ouas (larter) 368600997
Outs 37,5900950
255561
7
Nasy 3161//568
DIP
NAV*
N
N2N
Fodees11P 3189M19500
Ouis (Fzzdus)
Oues 38 12099
PSNRVSSI
23122
36630990
30,28009066
36660994
37.290090
Figure 7: Qualitative results on real noise removal from SIDD and PolyU. Noisy patches are from SIDDval31 and Canon80D 8 8 3200 ball_Ia
Table 3: Quantitative comparison on SIDD, PolyU and FMD for real noise removal.
SIDD
Avg. Infer. time (s)
Methods
PolyU 45
FMD
validation
benchmark
DIP
13
33.68/0.802
33.67/0.863
37.91/0.952
32.85/0.844
333.2
N2V
16
26.74/0.627
25.34/0.595
35.04/0.921
29.79/0.817
98.1
N25
17
26.78/0.573
26.93/0.658
32.82/0.930
31.61/0.759
114.4
7S-N2N
22
25.59/0.422
25.61/0.559
36.04/0.915
31.65/0.768
15.1
FasterDII
199
33.55/0.795
33.55/0.859
37.99/0.957
32.07/0.821
138.2
Ours (faster)
33.68/0.828
33.60/0.896
37.62/0.957
32.68/0 846
7.9
Ours
34.43/0.844
34.32/0.903
38.11/0.962
32.97/0.847
37.2
and real noise (Sec. 3.4). Additional tests on medical images (Sec. 3.5 show our method's adaptability beyond natural images.
Structure Similarity Index Measure (SSIM) on synthetic (Sec. 3.2 Sec. 3.3
3.1
Experimental Setup
Pre-training is performed two Nvidia RTX 3090 GPUs using Adam optimizer with 31=0.9 and 32=0.9. Initial learning rate is 2e and decays 1e-5 with cosine annealing strategy randomly cropped 256x 256
Pre-training.
over 80K iterations with batch size of 64. We initiate pre-training patches from subset of ImageNet (28) with around 48,000 images. Two sets of pre-trained weights with masking probability p (p=0.3 for synthetic noise and a higher ratio of 0.8~0.95 for spatially correlated noise) are trained. Further discussion of p is in Sec.
Zero-shot inference. We set learning rate during inference to 2e - and same masking ratio p as pre- pre-training (0.3 for synthetic, 0.8~0.95 for real noise) is set. EMA weight 3=0.99 for 1000 iterations (specially, 800 iterations for SIDD). Additionally, with 3.0.9, we achieve performance surpassing most zero-shot methods within 200 iterations, denoted as "faster". See Supp. for detailed setting.
3.2 Gaussian Poisson Noise
We investigate Gaussian Noise with 10,,255,500 and Poisson noise with Xe[10, 25, 50) separately on three datasets: CSet (3), McMaster (33) and CBSD (34), with 18 and 68 high-quality images, respectively. Results are shown Table The model is tested across various noise types with same experimental setups, without prior knowledge of noise distribution or intensity.
Analysis. DIP tends to produce over-blurry results and struggles especially with intense noise. While ZS-N2N manages to remove weak noise, its simple down-sample approach falters with stronger noise and cause artifacts. As Fig. Fjillustrates, under Gaussian noise o=25 and Poisson noise X=25, our method excels in both noise reduction and detail preservation. In some cases, see improvement of over 1dB, highlighting the effectiveness our zero-shot paradigm.
Our "faster" version achieve the fastest inference speed while surpassing comparing methods most cases. Even with 0.99, our method exhibits competitive inference time and significantly better performance. Params and FLOPs are in Supp.
Average inference time is listed in Table I

---

PSNN (CTB)
10
Beadine
0.355
PSNRVSIM
33,26/0.01
Pre trannd
Noisy
Refanne
DIP
230
1200
Iteraionn
32.5798879
32.97/0.111
33,9
FedaDDP
Ours (Fastar
Figure 8: Validation of pre-trained represen- tations on imge content differs from natural images. Comparing between "Baseline" (w/o pre-train) and "Ours" (w pre-train). Noisy patch is from TwoPhoton 2aoon_3MM_33.._ See quan- titative comparison at T
1=10
1=00
5000
-=00
500
Figure 9; Effect of pre-trained model. Examples using Gauss o=25 removal on F_ 16 with $=0.99. Pre-traines results are labeled in orange, while de- fault initialized results are labeled blue.
3.3 Generalization on Unseen Noise
We believe zero-shot denoising with natural image knowledge offers new perspectives on im- proving generalizability of denoising methods. We select several recent supervised (SwinIR [7]. Restormer (38) and unsupervised (Neighbor2Neighbor [T0], Blind2Unblind (39]) methods trained on Gaussian with o=25 for demostration. Testing them on 3 unknown noise types on Kodak [46].
Analysis. As illustrated in Table and Fig. 6 although methods trained multiple noisy images achieve better results on noisy cases with the same distribution, they exhibit poor generalization per- formance. In contrast, zero-shot methods often perform better generalization capabilities, especially our method which achieves the best performance across all types of generalization noise.
3.4 Real Noisy Datasets
We assess denoising capability of MPI synthetic noise in previous experiments. However, real- world noise is more complicated and challenging. We test on SIDD (1) and PolyU (45) datasets, including 1280 patches from the SIDD validation and 1280 from SIDD benchmark, and all 100 official patches from PolyU to show our paradigm on real images. Due to the differences between synthetic and real noise, we report results from comparison methods from their optimal iteration.
Analysis. As shown in Table 3, our method excels over other zero-shot approaches on both datasets. This underlines method's effectiveness real-world noise removal. Fig. show method's capability to balance noise removal and detail retention. In essence, our method is adept at real-world denoising, offering robust solution for image quality enhancement in challenging situations.
3.5 Generalization to Medical Images
The pre-trained model, which has learned the feature distributions of natural images, raises a question: Can this knowledge be applied to other image types? To answer this question, we select fluorescence microscopy dataset (FMD) 2 characterized by colors and textures distinctly different from natural images, using all released 48 images in testset for evaluation. See Supp. for more image types.
Analysis. Our method still excels in denoising performance, as seen in Table Despite such monochromatic microscopic images are not included in pre-training dataset and exhibits large differ- ences between natural images, pre-trained knowledge still enhances zero-shot denoising performance, evidenced in Fig. and Table demonstrating the generalizability of pre- pre-trained weights.

---

11,11
11.4
21.34
31,9
Bode
Massdng Ratiop
Figure 11: Effect of masking ratios. 30%
balances noise removal and prevents over- smoothing for synthetic noise.
Table Ablation of pre-training, with de- fault settings noted in gray "SIDD" denotes
Figure 10: CKA (above) [47 and PCA (below) visualization of features extracted from the final timestep of model. Distribution of pre-trained model ("Pretrained")) and from scratch ("Base- line") during inference significantly different in last layers. Pre-trained model tends to restore the complete image, while the baseline model pri- marily focusing on restoring masked regions only.
SIDD validation.
3
Pretrain
CSet [3
SIDD [1
FMD
31.61/0.841
34.43/0.841
32.97/0.447
0.99
x
30.90/0.811
72.41/
31.44/0.786
< 30.83/0.824
33.68/0 828
32. 02.68/0.448
0.90
30.10/0.806
33.42/0.224
32.31/0.833
4 Ablation Study & Discussion
4.1
Ablation
Pre-trained weights. Building on Sec. 2.1 we question the role of pre-trained weights in zero-shot inference by comparing inference with pre-trained weights and optimizing from scratch, resembling standard blind-spot network [16]. As depicted in Fig. annd Table the latter quickly peaks and risks over-fitting due to the simple task of content recovery from masked images, making it challenging to specify an optimal iteration for all images. Conversely, the pre-trained model achieves better initial performance and maintains close-to to-optimal performance for more extended period of time.
Unlike other zero-shot techniques train models from scratch for single image to learn noise-resistant image content, we offer a new perspective by showing that a pre-trained model can aid in zero-shot tasks. The pre-trained weights, encapsulating views from multiple natural images, making it more robust to iteration count and provides better options for faster zero-shot denoising.
Moreover, We investigated the impact of pre-training on inference at the hidden layer level (see Fig. 10P. Features extracted with pre-trained weights exhibit significant divergence from those produced by the baseline, i.e., the usual zero-shot denoising approach. Specifically, pre-trained model restores the complete image, with more distinct features between layers, whereas the baseline model's features are less differentiated between layers, tending to only restore the masked parts, which may result in sub-optimal convergence towards local minima.
Masking ratios. Fig. 11 shows impact of different masking ratios on denoising with Gaussian a=25. Lower masking ratios fails to completely remove noise, while higher masking ratios can cause overly smoothed results. A 30% masking ratio balances detail preservation and noise reduction for synthetic noise. However, a higher p of 0.8~0.955 is required for worldworld noise. See Supp. for more details.
Ensemble strategy. We explore ensemble strategies, including EMA-based ("EMA"), straightforward averaging during iterations (" "'Average") and average after specific optimization step ("Avg after 500e". where 500 is optimal). Due to the inability to obtain predictions for all pixels in single forward pass (Sec. 2.3). the "w/o Ensemble" result comes from the final prediction for each pixel. And "Last" provided for results of final forward prediction, a significant performance drop is caused by unreliable pixels (For details, see Supp.). Additionally, to validate our mask-based ensemble strategy, we remove masks from 7 and Eq. [ with full-pixel loss and ensemble ("EMA w/o Mask"). See results in Table Proposed "EMA" achieve significant better performance, aiding denoising with efficiency.
4.2 Discussion
Our method uses masking and minimal PD during inference to highlight pre-training's 's role without explicit regularization. We now explore further enhancements during inference with more strategics.

---

Table 6: Discussion on over-fitting.
Table 5: Ablation of ensemble strategy. "Time (s)"
Methods/ PSNR
Iteration
Avg. Infer.
denotes Infer. time (s)
1,000 1,100
1,500
time (8)
Ensemble strategy
PSNR/SSIM Time (s)
Ours
31.61
31.58
31.35
62.6
Ours+ES
31.66
31.65 31.66
51.7
Avg after 500e
30.88/0.793
48.3
Average
31.28/0.835
49.7
Table 7: Discussion of downsampling on
EMA w/o mask
23.48/0.441
52.1
SIDD validation with 256x256.
w/o Ensemble
30.23/0.797
51.7
PSNR/SSIM Time (s)
Last
13.73/0.154
49.0
EMA
31.61/0.841
53.5
34.42/0.843 843
29.6
+SSG 37
34.75/0.852
38.3
Over-fitting, Although pre-training mitigates over-fitting for synthetic and real noise, the over- parameterized network may still learn noise patterns over time due lack of explicit mechanisms to avoid identity mapping. This is common challenge in many zero-shot models, we suggest early- stopping (2T) (Table "ES" for early-stopping) avoid over-fitting and reduce inference time.
Additionally, we compare with other over-fitting prevention methods, e.g.. TV regularization of output and augmentation input image. These approaches either resulted in suboptimal performance or longer inference times. More details on over-fitting and prevention strategies can be found in Supp.
Sub-sampling. Shown in Sec. 2.4 minimal pixel-shuffle is used to reduce spatial correlation in real-world noise, but may cause chessboard artifact and reduce performance due to its regular down- sampling strategy. Better down-sampling strategies have been widely studied, and here we choose For more comparison and visual comparison, see Supp.
RSG 37 to illustrate, see results at Table 7
Acknowledgements
This work supported in part by the National Natural Science Foundation of China under Grant 62401532, in part by the Anhui Provincial Key Research and Development Plan 202304a05020072. and in part by the Anhui Provincial Natural Science Foundation 2308085QF226, in part by the Fundamental Research Funds for the Central Universities WK2090000065, and in part by the China Postdoctoral Science Foundation under Grant 2022M720137.
10

---

References
[1] Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A qughiquality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 1692- 1700, 2018
[2] Yide Zhang. Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A maaissongauusian denoising dataset with real fluorescence microscopy images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11710-11718, 2019.
[3] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080-2095,
2007.
[4] Matteo Maggioni, Vladimir Katkovnik, Karen Egiazarian, and Alessandro Foi. Nonlocal transform- domain filter for volumetric data denoising and reconstruction. IEEE transactions on image processing, 22(1):119-13) 2012.
[5] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):1142- 3155, jul 2017.
[6] Kai Zhang, Wangmeng Zuo, and Lei Zhang. Fidnet: Toward fast and flexible solution for CNN based image denoising. IEEE Transactions on Image Processing, 2018.
[7] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference computer vision. pages 1834-1844, 2021.
[8] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In International Conference Machine Learning, pages 2965-2974. PMLR, 2018.
[9] Wenchao Du Hu Chen, and Hongyu Yang. Learning invariant representation for unsupervised image restoration. In Proceedings of the ieeelcuf conference on computer vision and pattern recognition, pages 14483-14492, 2020.
[10] Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2neiighhr: Self-supervised denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14781-14790, 2021.
[11] Hao Chen, Chenyuan Qu, Yu Zhang, Chen Chen, and Jianbo Jiso. Multi-view self-super disentangle- ment for general image denoising. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12281 12281-12291, 12291, 2023.
[12] Haoyu Chen, Jinjin Gu, Yihao Liu, Salma Abdel Magid, Chao Dong, Qiong Wang, Hanspeter Plister, and Lei Zhu, Masked image training for generalizable deep image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1692-1703, 2023.
[13] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedingy of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[14] Yuhui Quan, Minggin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self-supervised denoising from single image. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1890-1898, 2020.
[15] Jun Cheng, Tao Liu, and Shan Tan. Score priors guided deep variational inference for unsupervised real- world single image denoising. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12937- 2023.
[16] Alexander Krull, -liver Buchholz, and Florian Jug. Miise2void-lcarming denoising from single noisy images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2129-2137, 2019.
[17] Joshua Batson and Loic Royer. Noise2selff: Blind denoising by self-supervision. In International Conference on Machine Learning, pages 524-533, PMLR. 2019.
[18] Reinhard Heckel and Paul Hand. Deep decoder: Concise image representations from untrained non- convolutional networks. In International Conference on Learning Representations, 2018
[19] Yilin Liu, Jiang Li, Yunkui Pang, Dong Nic, and Pew-Thee Thian Yap. The devil is in the upsampling: Architectural decisions made simpler for denoising with deep image prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12408-12417, 2023.
[20] Yeonsik Jo, Se Young Chun, and Jonghyun Choi. Rethinking deep image prior for denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5087-5099 2021.
11

---

[21] Zenglin Shi, Pascal Mettes, Subhransu Maji. and Cees GM Snoek. On measuring and controlling the spectral bias of the deep image prior. International Journal Computer Vision, 138(4)8885-908, 2022.
[22] Youssef Mansour and Reinhard Heckel. Zero-shot noise2noise: Efficient image denoising without any data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 14018-14027, June 2023.
[23] Jason Lequyer, Reuben Philip, Amit Sharma, Wen-Hsin Hsu, and Laurence Pelletier. A fast blind zero-shot denoiser. Nature Machine Intelligence, 4(11):953-963, Nov 2022.
[24] Markku Makitalo and Alessandro Foi. Optimal inversion the anscombe transformation low-count
poisson image denoising. I.F.F Transactions on Image Processing 20(1):99-109) 2011.
[25] Anish Mital, Rajiv Soundararajan, and Alan C Bovik. Making a "completely blind" image quality analyzer. IEEE Signal processing letters, 20(3):209-212. 2012.
[26] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Gitshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[27] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEL/CVF Conference Computer Vision and Pattern Recognition, pages 9653 9663, 2022.
[28] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[29] Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2021.
[30] Yunyao Mao, Jiajun Deng, Wengang Zhou, Yao Fang, Wanli Ouyang, and Houqiang Li. Masked motion predictors are strong 3d action representation learners. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10181-10191, 2023.
[31] Jiang-Tian Zhai, Xialci Liu, Andrew D Bagdanov, Ke Li, and Ming-Ming Cheng. Masked autoencoders are efficient class incremental learners. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19104-19113, 2023.
[32] Yihao Liu, Anran Liu, Jinjin Gu, Zhipeng Zhang, Wenhao Wu, Yu Qiao, and Chao Dong. Discovering distinctive" semanties" in super-resolutee networks. arXiv preprint arXiv:2108 10Xi::2008.00006 2021.
[33] Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic imaging, 20(2):023016 -023006, 2011.
[34] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithus and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416-423 IEEE, 2001.
[35] Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang. Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 13074-13000 2020.
[36] Wooseok Lee, Sanghyun Son, and Kyoung Mu Lee. Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17725-17733 2022.
[37] Yizhong Pan, Xiao Liu, Xiangyu Liao, Yuanzhouhan Cao, and Chao Ren. Random sub-samples generation for self-supervised real image denoisinn. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12150-12159, 2023.
[38] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 5728-5739. 2022.
[39] Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. BlindZunblind: Self-supervised image denoising with visible blind spots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2027-2036, 2022.
[40] Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1586-1595, 2017.
[41] Yinhuai Wang. Jiwen Yu, and Jian Zhang, Zero-shot image restoration using denoising diffusion pull-space model. arXiv preprint arXiv:2212.004900 2022.
12

---

[42] Tomer Garber and Tom Tirer. Image restoration by denoising diffusion models with iteratively precondi- tioned guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 25245-25254, 2024.
[43] Dan Zhang, Fangtang Zhou, Yuwen Jiang, and Zhengming Fu. Mm-bsn: Self- supervised image denoising for real-world with multi-mask based on blind-spot network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 4189-4198, 2023.
[44] Hyemi Jang, Junsung Park. Dahuin Jung. Jaihyun Lew, Ho Bae, and Sungroh Yoon. Puca: patch-unshuflle and channel attention for enhanced self-supervised image denoising, Advances in Neural Information Processing Systems, 36, 2024.
[45] Jun Xu, Hui Li. Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: A new benchmark. arXiv preprint arXiv: 100.00003,,0,0 2018
[46] Rich Franzen. Kodak lossless true color image suite, source: http://vkk. ws/graphics/kodck 4(2):9, 1999.
[47] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pages 35 151993529. PMLR, 2019.
[48] Jun Xu, Yuan Huang. Ming-Ming Cheng, Li Liu, Fan Zhu, Zhou Xu, and Ling Shao. Noisy-as-clean- Learning self- supervised denoising from corrupted image. IEEE Transactions on Image Processing, 29:9316-9329, 2020.
[49] Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12064-12072, 2020.
[50] Tongyao Pang. Huan Zheng. Yuhui Quan, and Hui Ji. Recorupted-to-rccomuup Unsupervised deep learning for image denoising. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2043-2052, 2021.
[51] Yi Zhang, Dasong Li, Ka Lung Law, Xiaogang Wang. Hongwei Qin, and Hongsheng Li. Idr: Self- supervised image denoising via iterative data refinement. Proceedings of the IELE/CVF Conference on Computer Vision and Puttern Recognition, pages 2098-2107, 2022.
[52] Reyhaneh Neshatavar, Mohsen Yavartanoo, Sanghyun Son, and Kyoung Mu Lee. Cvf-sid: Cyclic multi- variate function for supervied image denoising by disentangling noise from image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17583-17591, 2022.
[53] Xin Lin, Chao Ren, Xiao Liu, Jie Huang, and Yinjic Lei. Unsupervised image denoising in real-world scenarios via self-collaboration parallel generative adversarial branches. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12642-12652, 2023.
[54] Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European conference on computer vision, pages 32-368.. Springer. 2020.
[55] Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. Advances in Neural Information Processing Systems, 32, 2019.
[56) Zichun Wang, Ying Fu, Ji Liu, and Yulun Zhang. Lg-bpn: Local and global blind-patch network for self-supervised real-world denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18156-18165 2023.
[57] Yaochen Xie, Zhengyang Wang, and Shuiwang Ji. Noise2same: Optimizing self-supervised bound for image denoising. Advances in neural information procexsing systems, 33:20320- 2020.
[58] Yeong Il Jang, Keuntek Lec, Gu Yong Park, Seyun Kim, and Nam Ik Cho. Self-supervised image denoising with downsampled invariance loss and conditional blind- spot network. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12196-12205, October 2023.
[59] Junyi Li, 7hilu Zhang, Xiaoyu Liu, Chanyu Feng, Xiaotao Wang, Lei Lei, and Wangmeng Zun. Spatially adaptive self-supervised learning for real-world image denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 9914-9924, 2023.
[60] Kwanyoung Kim and Jong Chul Ye. Noise2score: tweedic's approach to self supervsed image denoising without clean images. Advances in Neural Information Processing Systems, 34:864- 048864-874, 2021.
[61] K Kwanyoung Kim, Tacsung Kwon, and Jong Chul Ye. Noise distribution adaptive self-supervised image denoising using tweedie distribution and score matching. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2008-2016, 2022.
[62] Zongsheng Yue, Hongwei Yong. Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. Advances in neural information processing syxtems, 32, 2019.
[63] Mohammad Zalbagi Darestani and Reinhard Heckel, Accelerated mri with un nn-trained neural networks. IEEE Transactions on Computational Imaging, 7:724-733, 2021.
13

---

[64] Metin Eisin Arican, Ozgur Kara, Gustav Bredell, and Ender Konukoglu. Isnas-dip: Image-specific neural architecture search for deep image prior. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1960-1968, 2022.
[65] Chao Wang. Zhedong Zheng. Ruijie Quan, Yifan Sun, and Yi Yang. Context- pretraining for efficient blind image decomposition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18186-18199 18195, 2023.
[66] Naishan Zheng, Man Zhou, Yanmeng Dong, Xiangyu Rui, Jie Huang, Chongyi Li, and Feng Zhao. Empowering low-light image enhancer through customized learnablc priors. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12559-12569, 2023.
[67] Guilin Liu, Fitsum A Reda, Kevin Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro, Image inpainting for irregular holes using partial convolutions. In Proceedings of the European conference on computer vision (ECCV), pages 85-100, 2018.
[68]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[69] Abdelrahman Abdelhamed, Marcus Brubaker, and Michael Brown. Noise flow: Noise modeling with conditional normalizing flows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3165 3173,, 2019.
[70] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unpro- cessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1103--10455 1104,, 2019.
[71] Chang Qiao,
Yuting Guo, Chong Liu, Tao Jiang, Qionghai Dai, and Dong Li. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature Methods, 18(2):194-20,, 2021.
[72] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. Learning in the dark. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3291-3300, 2018.
14

---

A Introduction
This document provides supplementary materials for the main paper. Specifically, a brief conclusion of works related to ours is in Sec. B Sec. presents more details and demonstrations about proposed
iterative filling (Sec. C.I. Sec.
C and different strategies to adapt to real-world spatial-correlated
noise (Sec. C.3 Sec. D presents more discussion about masking ratio (Sec. DD.p. over-fitting
(Sec. D22p and different downsampling strategies (Sec. D.3p. Additionally, we conduct further
analysis of pre-training (Sec. [E] and expanded our framework to other network structures (Sec. F).
More experimental details and qualitative comparison results can be found in Sec. G and Sec.
B Related Works
B.1
Unsupervised Image Denoising
Unlike supervised approaches (5 7,388 unsupervised denoising focuses on situations when paired data is unavailable. Methods in this category include:
Paired noisy-noisy images. To learn consistent representations from varied noise observations of the same scene, Lehtinen et al. [8] train mapping from two noisy observations of the same scene.
Additional approaches utilize synthetic noise to generate noisy pairs, as seen in 48 52 as well
as I1 learns shared latent from multiple noise observations.
Unpaired noisy-clean images. Du et al. (9] propose learn decoupled representations of contents and noise from images. Lin et al. (53) extend this by using separated noise representations to guide noise synthesis, thereby enhancing the denoising process. Additionally, Wu et al. (54) employ a
distillation loss from both real and synthetic noisy images.
Noisy images only. Techniques like blind-spot [7 39 55 55-57, 57). substitution followed by image reconstruction (16 17). multiple sub-sampled images from a single noisy scene (TO) or above approaches combined (6,37||5 558) are developed when only one observation available from a noise scene. Li et al. [[9]] integrate blind spot strategies and structural insights for adaptive denoising. Score matching and posterior inference are also utilized in 66 61].
In the context of zero-shot denoising tasks, only a single noisy image is visible during training, presenting greater challenges than methods described above.
B.2 Zero-shot Image Denoising
Compared unsupervised methods, zero-shot denoising is more challenging it aims to train a network for denoising when a single noisy image available. Typical strategies involve utilizing spatial correlations 3 variation based priors [15 62 low-frequency characteristics of images, corrupting and reconstructing part of images [14 16 17], or constructing paired training sets from
sub-sampled noisy images [22 Among which Noise2 Void (16) is initially designed for learning from multiple noisy images, shows promise in its zero-shot version. Noise2Self (17 reconsttruct cyclically masked regions of input noisy image. Although there is gap compared to supervised or unsupervised methods. While dropout-ensembee performance, leads to over-smoothing and incurs large computational costs. Noise2Fast (23] and Zero-Shot Noise2Noise [22] are fast but struggle to completely remove noise from images, especially spatially-correlatedd real noise, resulting in suboptimal visual results. DIP (13) and its variants [18
(14) is adapted on a noisy image for better
exploits the features of deep networks to learn mappings from random noise to images. Early
stopping (2T) or other approaches [20 are used to prevent over-fitting, FasterDIP (19] further discusses the influence of network structure on its performance. However, current zero-shot methods often takes a long time, and parameter settings are carefully selected for various image contents and noise degradation.
B.3 Masked Image Modeling
Masked Image Modeling (MIM) helps in learning pre-trained representations for downstream tasks by masking a portion of input images 2 and training models to predict the masked contents. Due to its impressive effects in high-level tasks [30 31,, Masked Image Modeling has also found

---

applications in low-level visual tasks. For instance, Wang et al. [65 applies random patch masks during the pre-training image deraining and desnowing models in handling adverse weather conditions. Zheng el al. [66] integrates Masked Autoencoder (MAE) to learn illumination--elated structural information in supervised low-light enhancement framework. Notably, despite the successful applications of Masked Image Modeling in several low-level vision tasks, its application in the pre-training scheme for denoising models has not yet been explored.
C More Details about MPI
C.1 More Details about Iterative Filling
In the main paper, we mention that Iterative Filling is optimization steps based on pre-trained weights for zero-shot denoising. And depicted in Sec. 3.3 in main paper, fully leverage the results of each optimization step and preserve more image detail, only the masked regions M, yt are constrained by the loss at each optimization timestep and considered reliable, others are labeled as "unreliable pixels" and discarded (see Sec. C.2 to know why we need "unreliable pixels"). During the iterative optimization, we maintain an ensemble version y, assembling reliable parts M, @ y of each prediction via EMA while keeping the rest unchanged, shown in Fig. 12
With sufficient iterations. ensemble from each pixel is ensemble from hundreds of predictions. ensuring a high-quality ensemble outcome.
Yt x (11-1
Prediction
Replace in y,
M, y
Ensemble yt in thstep
xB
Ensemble Te+1 (++1) tyttttt
Figure 12: Details of EMA process in Iterative filling. Masked regions of predictions from each optimization step t assembled to a ensemble 7.
C.2 Why Discard "Unreliable Pixels"
During the zero-shot denoising process, unmasked parts of each optimization result M, @ ys needed be discarded, referred to as "unreliable pixels", primarily due to the following reasons:
The pre-training task is to reconstruct masked regions, that only masked areas are constrained for reconstruction, while pixels in unmasked areas significantly differ from the actual pixels in the image. This discrepancy might result from the skip connections in the network architecture of DIP [13]. To maximize the utilize of pre-trained weight and avoid conflicting, pixels corresponding these unmasked regions should not be considered during optimizing.
2) For spatially uncorrelated noise, we employ an extremely low mask ratio and distinct mask settings for each color channel preserve as much image information possible. In this scenario, it is impractical to expect the network to retain all pixel values in its output, as this can easily lead to identity mapping.
3) While partial convolution (67] or others designed for image inpainting can mitigate this problem, they often lead to sub-optimal performance with risks of over-fitting to noise, and require specialized network architectures, limiting the adaptability of proposed framework to other network structures.
For our zero-shot denoising framework, which obtains denoised images via iterative optimization, using a portion of noisy image as cues and training network to complete this "fil-in-the-bllnk task proves most effective. Moreover, the optimization generates reconstructions with various masks in iterations, assembling these results to achieve final denoised prediction requires only maintaining an ensemble result additionally, incurring negligible time and space resources.

---

inv mask M,
M
D0
Y
M
ZIGA
Down())
Inputxx
Up())
Yt
Figure 13: An overview of the zero-shot denoising stage with adaptation to real-world noise. We adapt downsample " "Down(())" and upsample "Up(-)" to achieve noisy subsamples with less spatial correlation in noise, labeled in green arrows, and larger masking ratio is used to further deal with remaining spatial correlations. Actually, not real noisy images needed to sampled, we only adapt "Dwn(-)" and "Up(-)" to SIDD dataset.
C.3 Different Strategies in Dealing with Synthetic & Real Noise
We adapt downsample " "Down(-)" and upsample "Up(-)" to achieve noisy subsamples with less spatial correlation in noise, and larger masking ratio 80%~995% (90% for SIDD and 85% for others) with a unified mask for all channels used to further deal with remaining spatial correlations. Actually, not all real noisy images needed be sub-sampled, only adapt "Down(-)" and "Up(.)" to SIDD dataset. See Fig. 13 for a simple framework, which specifically designed for real-world noise is labeled in green arrows.
D Additional Discussion
D.1 Masking Ratio
In the main paper, discussed the optimal masking ratio for removing spatially uncorrelated synthetic noise and notes that significantly larger mask ratios are used for real noise. For real spatially correlated noise, the situation becomes more complex, with single optimal mask ratio. Based on experience, the most effective mask range is between 80%~~95% which is influenced by the noise's spatial correlation, noise intensity, and the information in image.
For the SIDD dataset, we investigated the impact of the masking ratio on SIDD validation, as shown in Fig, 14 finding that 90% masking ratio is optimal. This is attributed to SIDD images containing limited content information and the noise exhibiting strong spatial correlation.
However, the optimal masking ratio for SIDD differs from that for synthetic noise, primarily due to the spatial correlation of the noise within the image. Synthetic noise spatially uncorrelated, meaning noise signals at neighboring positions do not influence each other. In contrast, real noise, after undergoing a series of ISP processes, exhibits a more complex distribution, resembling blurred spots rather than independent points (see Fig. [5]. For synthetic noise, selecting small masking ratio allows for quicker recovery of image details. Conversely, for real noise, a small masking ratio
34.5
34.28
34.40
34.21
PSNR (dB)
34.07
34.0
33.25
33.0
Masking Ratio p (%)
Figure 14: Effect masking ratios real-world noise (SIDD validation). 90% for removing strong spatial correlated noise.

---

Figure 15: Illustration of
real- real-world noise (right) and synthetic noise (left).
Table 8: Extension of proposed pre-training strategy into other network architectures. A performance improvement can be observed for both settings of beta=0.9 and 0.99 in the experiment across various network architectures.
Methods/ PSNR
Iteration
Avg. Infer. time (s)
800
900
1,200)
Ours
34.43
34.34
33.48
39.9
Ours+ES
21
34.64
34.63
34.60
34.5
may lead the model fitting the noise distribution by relying neighboring pixel values. In such cases, larger masking ratio helps mitigate the influence of noise.
D.2 Over-fitting Regularization
In the main text, highlighted that our proposed zero-shot denoising framework still faces over- fitting issues with increased iteration counts, with Table and Fig. showing this problem's 's impact across more datasets and more regularization. To address this issue, after validation and comparison, we recommend adopting simple stoypig strategy to prevent over-fitting a straightforward, effective approach without additional computational costs. We also compared other strategies:
Employing TV regularization helps against overfitting but still leads to performance drop and lower peak PSNR as iterations increase.
Adding random transformations to the input image, including flips and random translations, lead to steady performance improvement and higher peak PSNR over more iterations, but increase inference time.
Early stopping stops close peak performance with minimal calculation, providing stable, high-quality results with no added time.
PSNN (dB)
Ours
+TV
24
Fastor
+Aug
+ES
1500
3000
Iterationt
Figure 16: Influence of different regularization strategies during iterations, including Total Variation ("+TV"), random image augmentation ("+Aug"), and early stopping ("+ES"). "Ours" and "Faster" are the methods evaluated in mainpaper. Example is tested on F16 1_6r_ with Gaussian o=25.
D.3 Down-sampling in Real-world Denoising
As illustrated in Fig. 13 specialized downsampling is employed to reduce the spatial correlation of real noise, with different downsampling strategies yielding varying outcomes. Simple pixelshuffle ("+PD") can easily lead to checkerboard artifacts, whereas more randomized sub-sampling strategies [37

---

Table Discussion of noise in pre-trained dataset. Additional assumptions of noise in pre-trained dataset result in lower performance ("+Gauss (0=25)(N2N))).
Pre-train Mode
PSNR
SSIM
Ours
31.61
0.841
+Gauss (0=25)NNN)
31.24
0.827
("+RSG") can more effectively disrupt noise spatial correlations and, due their randomness, avoid checkerboard artifacts, as depicted in Fig. 17 A potential issue with this approach is the over-smoothed denoising predictions. Therefore, downsampling only applied in strong spatially- correlated real noise.
Noua
Referacc
Nois
Refencee
27,6994.100
PSNRUSSIM
27.4904.17
PSNRSSIM
+P13
+HSC
+P1 312909433
HSSC 4000009
40.6209344
41,25/0:58
Figure 17: Validation of different down-sampling strategies real-world denoising. Better down- sampling strategies can further enhance denoising performance of our pipeline. Noisy patches are from SIDDval 1aoval_1_222__ and SIDDval 2.
E
Additional Analysis on Pre-training
E.1
Noise Intensity
In the main text, we have already mentioned that pre-training aids in the removal of various types of noise. We validated the relationship between pre-trained weights and different input noise intensities on the CSet dataset, as shown in Fig. [8] Pre-training enhances denoising performance across different noise levels, particularly the case of strong noise, where the knowledge provided by pre-training effectively avoids over-fitting to the noise.
E.2 Masking Ratio
We analyze the impact of pre-training on different paradigms under various masking ratios, as shown in Fig. 19 Our study reveals that pre-training plays significant role in enhancing denoising performance across various masking ratios (especially in cases of 20 S S 80).
E.3 Discuss of Noise in Pre-training
In the main text, use the well-known natural image dataset ImageNet without making any assumptions about the presence or type of noise in each image, hoping to learn the statistical distribution rules from a large number of natural images. Here, we add synthetic noise of specified distribution and intensity during pre- pre-training, and perform pre-training from noise to itself (denoted as "+Gauss (o=25) (N2N)"), and adopt the same iterative denoising strategy, proving that additional assumptions about specific noise type noise level in pre-training leads to a decline in effectiveness, as shown in Table. 9 Networks that are too small fail learn sufficient denoising information, falling short of the corresponding zero-shot approaches.
5

---

32
30
PSNR (dB)
32
Basdine
Baddine
26
Pre-trained
24
Pre trannzinned
10
10
20
30
40
50
Noise Ø
Noiselevel 2
Figure 18: Effect pre-training on different noise levels from Gaussian (left) and Poisson (right) on CSet. Pre-training is beneficial for all 6 noise levels, especially in cases of intense noise.
31
PSNR (dB)
30
Baseline
Pre traindd
10 20
40
60
70 80
90
Masking Ratio p (%)
Figure 19: Effect of pre-training on different masking ratios p from Gaussian noise on CSet. Pre-
training is beneficial for all masking ratios, especially in cases of 20 I p < 80. Table 10: Extension of proposed pre-training strategy into other network architectures. A performance improvement can be observed for both settings of beta=0.9 and 0.99 in the experiment across various network architectures.
Params (M)
8
Pre-trained
Baseline
Infer.time (s)
0.9
30.49/0.812
26.76/0.720
15.1
DnCNN 5
0.56
0.99
31.69/0.845
30.68/0.825
75.0
0.9
30.46/0.812
29.20/0. 778
8.0
ResNet
0.99
31.43/0.838
31.16/0.836
39.4
Extension other network structures
In the main text, we discuss the effect of pre-training on the proposed model using the same network architecture as DIP [33. We are curious whether this prior knowledge could be applied to other model architectures. Here, compare the impact of pre-training under differrnt model settings. Specifically, we evaluate the removal of Gaussian noise with sigma=25 on CSet using three additional network architectures (DnCNN (5), ResNet (68)), as shown Table TO The pre-training approach consistently brings performance gains across various network architectures.
G
More Experimental Settings & Results
G.1
Quantitative analysis of Poisson noise removal
Due to space limitations in the text, there is no quantitative comparison of Poisson noise, which is listed in this section, see Table 11
G.2 Quantitative comparison with more methods
Here we compare more recent methods including additional DIP-based zero-shot method (DIP-
SURE 20)., diffusion-based methods (DDNM [4T], DDPG [42 ), and zero-shot modifications from
unsupervised methods (AP-BSN (36], MM-BSN 433, PUCA 44 refer to Table 12 for results.

---

Table 11: Quantitative comparison CSet, McMaster and CBSD dataset for Poisson noise removal (Xe[10,5,550). For best results highlighted and second underlined.
DIP [13]
N2v [16]
N2S* 17
7S-N2N [2]
FasterDIP [9
Ours (faster)
Ours
10 22. 01.88/00449 495
26.50/0.650
25.34/0.661
25. 70/06688
25.02/0.633
27.79/0.714
27. 27.55/0.c0 696
CSet 3
27.57/0.681
27.16/0.755
28.06/0.711 27.16/0.750
27.73/0.710
7884
30.02/0.785
50
30.03/0.775
29.88/0.818
27.68/0.780
29.79/0.780
28.86/0.749
31.00/0.830
31.68/0.841
24.45/0.66m 644
25.97/0.696
25. 25.68/0.735
26. 26.14/0.730
28.26/0.793
28.15/0.770 770
McMaster 33
25
29..23008001
28.84/0.807
27.28/0.782
28.49/0.775
28.03/0.783
30.34/0.856
30.92/0.86
50 31.13/0.856
30.60/0.871
27.78/0.803
30.34/0.834
28,70/0.792
816
32.64/0.9MM
10
21.81/0.554
77.16606686
24.49/0.681 681
23.53/0.643
26.27/0.716
26.05/0. 709
CBSD
34
25
26.83/0.741
26.96/0.798 798
26.20/0.775 775
27.66/0,776
26.64/0.757 757
28 28.91/0 11.08.23
29.00/0.824
50 29.35/0.828
28.25/0.835
26.95/0.808
29.55/0,833
27.91/0.781
30.45/0.871
31.00/0.881
Table 12: Quantitative comparison with other DIP-based method (DIP-SURE (20)) ) Diffusion
methods (DDNM. DDPG), zero-shot methods modified from other Self-supervised methods (AP-
BSN, MM-BSN, PUCA). DIRE DDNM and DDPG requires additional noise variance as input, and DIP- SURE applies different iterations for each image.
CSct+Gaussian
SIDD validation
Avg. Infer. time (s)
Method
PolyU
FMD
a=10
0=25
0=
DIP SURE(peak) 35.37/0.916
31.88/0.855
28.81/0.775
30.45/00727
35.87/0.944
32.04/0.798
DIP
DIP (las(
34.98/0.908
31.50/0.840
28.76/0.762
26.63/0.649
35.78/0.942
32.07/0.793
367,3
DDNM
96222/0.927
32.4/0.859
29. 29.99/0.793
28.1 1/0.597
37.15/0.935
28. 99/9.96588
26.7
Diff.
DDPG
32.43/0.826
27.07/0.606
15.95/0.183
29.8 29.84/0.612
35.79/0.887
30.41/0.735
24.3
AP-BSN*
25.04/0.671
33. 33.34/0.844 847
22.64/0.928
19,277/0.79
351.4
Self-
supervised
MM-BSN*
25.27/0.676
33.36/0 843
33.07/0.930
29.73/0.810
505.3
PUCA
24.74/0.640
33.52/0.816
33.31/0.927
30.22/0.808
450.0
Ours
34.91/0.909
31.61/0.811
28.26/0.710
34.43/0.444
38.11/0.962
32.97/0.847
45.8
Specifically, for DIP-SURE, we report both peak performance and the performance at the final iteration. Since DIP- is specifically designed for Gaussian and Poisson noise, and requires the input of Gaussian noise variance, for real-world denoising tasks we provide estimated variance from paired data, to report the best performance. For diffusion- exclusively on Gaussian noise and also require variance as prior, we use the same variance estimation approach to report their best results. For self-supervised methods, which can be easily adapted to single image with minimal changes, we follow their original settings. In each iteration, we crop eight same-size patches from the noisy image to form batch, and perform inference the full image every 10 iterations, and combine denoised images using the same ensemble strategy as our method for fairness.
methods, which are trained
We observe that SURE due to its priors on noise type and variance, performs slightly better than our method under Gaussian noise settings, However, its performance significantly drops when dealing with real noise, especially when reporting the last performance. Since diffusion models are inherently Gaussian denoisers, they perform well on Gaussian noise when the variance is known, but also face challenges with real noise. The modified blind-spot network-based methods can handle severe real noise relatively well, but they may suffer from potential image detail loss and require long inference times.
G.3 Ensemble results of N2V and N2S
In the main text, we present versions of DIP 113 and FasterDIP 19 with EMA (Exponential Moving Average) ensembles, as these processes are included in their source codes. To provide additional information for comparison, we also adapted N2V* [16] and N2S* [17] to their corresponding EMA ensemble versions, as shown in Table 13
Generally, the ensemble versions of these methods can improve the PSNR by 1~2 dB. However, even though the enhanced N2V may outperform our Faster version in some cases, it does not affect the performance comparison with our 3=0.99 version, which remains the best. Moreover, our 3=0.99 version achieves this with less than half the inference time required by these methods.
14 15
7

---

Noisy
GT
DIP-SURE
DDNM
DDPG
MM-BSN
PUCA
Ours
Figure 20:
Comparison of different methods under SIDD validation
(SIDDval334_2)
Table 13:
Quantitative comparison of ensemble version of N2V* and N2S* CSet, McMaster and
CBSD dataset for Gaussian and Poisson noise removal.
Gaussian
Poisson
B=10
6=25
8=50
A=10
A=25
A=50
CSet 3
34.05/0.89
30.99/0.825
77.70/0.703
27.02/0.675
39.55/0.790
30,95/0.830
N2V 16
McMaster 33
34.31/0.920
30.97/0.862
27.94/0. 766
26, 26.93/0.751
30.06/0.812
31.78/0.884
CBSD
33.15/0.918
30.05/0.850
26.27/0.702
25 25.42/0.714
28.55/0.811
30.40/0.869
CSet 3
29.92/0.843
28.76/0.787
26.67/0.704
26.83/0.670
27.94/0.765
28.91/0.797
N2S*
17
McMaster 33
29.85/0.867
28 78.5.20.785
25.07/0.678
76.
28.24/0.814
28.99/0.836
CBSD 34
28.50/0.854
27.51/0.803
25.07/0.705
24.83/0.714
26.59/0.785
27.53/0.817
G.4 Details of Unseen Noise
Gaussian Noise. Gaussian noise follows a normal distribution and is commonly encountered in digital imaging, especially during sensor data acquisition and transmission. represents random variations in intensity and color information in images, making it fundamental noise model in image processing. For each element in clean image /ja]- represented by:
[n] [M]] + a. Nis.
where Nuy represents random variable sampled from standard normal distribution, which is characterized by its standard deviation (o).
Poisson Noise. Poisson noise is prevalent in scenarios with low-light conditions, such as astronomical imaging or medical imaging, where the photon count inherently random and follows a Poisson distribution. Poisson noise models the variation of intensity based Poisson distribution, is generally expressed as:
P(Tp A)/X,
(10)
where À indicates the event rate, and P(-) denotes random variable generated from Poisson distribu- tion.
Noise Level Function (NLF). Noise level function, also referred to heteroscedastic gaussian model [69]. is commonly described by a varying standard deviation across the image. This type of noise is widely used express the read-shot noise in camera imaging pipeline, where different parts of the image exhibit different noise levels. It is typically modeled as:
iM Ni = -py, o² o, + 0 41a)
(11)
where a, and or represent different standard deviations in distinct regions of the image. Noise parameter calibrated for (40] work (70] obeys a log-linear rule:
log(o,) 2.18 log(a,) 1.2
(12)
We choose a [0.01, 0.012) better illustrate the generality.
Speckle Noise. Speckle noise is an interference pattern produced by the coherent processing of a signal, especially in active radar and ultrasound imaging. This noise is particularly common in radar and ultrasound images, where can significantly degrade the quality of the image. Its mathematical representation is:
i = [R]] Ts Up},
(13)
where Uny sampled from uniform distribution with mean and v representing the standard deviation of the noise.

---

Table 14: Quantitative comparison of ensemble version of N2V* and N2S*
Kodak with 5 noise
types for generalization evaluation.
Gaussian
Poisson
Speckle nE[10,5]]
S&P
NLF
Average
2=
c[10,50]
A0[10,50]
dc[0.02,0.05]
N2V* 30.95/0.850
30.90/0.836
29.55/0.822
93.82/0.917 28.82/0.840
32.26/0.888
34.26/0.948 28.88/0.828
31.95/0.877 28.33/0.816
N2S* 28.34/0.804
27.91/0.790 27.21/0.779
28. 84/0.46886
Table 15:
Quantitative comparison of ensemble version of N2V* and N25* on SIDD, PolyU and FMD dataset for real-world noise removal.
SIDD
Methods
PolyU 45
FMD 2
validation
henehmark
22V*
16
28.51/0.670
27.19/0.645 6990
36. 36.11/0.921 35.39/0.939
30.85/0.754 31.72/0.762
N2S
27.41/0.584
27.59/0.684
Peepper Noise (S&P). Salt-and- noise, also known as impulse noise, characterized by sharp and sudden disturbances in image signal. It's typically represented sparse white and black pixels, hence the name. This noise can be caused by sharp and sudden disturbances in the image signal, often due to transmission errors, faulty memory locations, timing errors in digital image sensors. Its mathematical representation is:
[mn]
[[t] + S1A] -jej+
(14)
where S/g and Pi represents salt and pepper noise, respectively. For each them are Bernoulli sample with probability d of Lmaz/ Imaz/Imin Imin and probability d of O, which makes the probability total affected is 2 d.
G.5 Additional Computational Costs
In analyzing the performance of deep learning models, it's crucial to consider both the Floating Point Operations (FLOPs) and the model parameters. FLOPs give an insight into the computational complexity of the model, which affects inference time and resource utilization, while the number of parameters indicates its capacity to learn and adapt complex data patterns. A balance between these two aspects is essential for efficient and effective model performance. Our analysis, reflected the comparison between Table [6] demonstrates that our method successfully achieves this balance. It maintains computational efficiency without compromising the model's ability to accurately process and analyze data, an essential factor for practical application varied computational environments.
G.6 Zero-shot Denoising on More Image Types
In the main text, we demonstrate the ability of proposed MPI generalize other types of images through medical imaging dataset. Further here, we explore new types of images, including microscopy imaging dataset BioSR [71) and extremely low-light dataset SID [72.. See Fig. 21 and Fig. 22 for qualitative examples.
Table 16: Efficiency comparisons of deep learning based methods on Params and FLOPs under input size 256 256 with single forward step. Iterations used for synthetic noise is provided for reference.
Method
Params (M)
FLOPs (G)
Iters
DIP
2.3
19.66
3,000
N2V*
16
1.2
80.50
1,500
N
17
0.07
1.57
1,800
ZS N2-
22
0.02
1.45
2,000
FasterDIP
19
0.05~~0.92
0.5-8.8
3,000
Ours(faster)
0.73
8.11
1,000
Ours
0.73
8.11
200

---

Noisy Image (b) Denoised by Ours Figure 21: On noisy microscopy image (a) using the proposed MPI to denoise retaining the structural information in the image as much as possible (b), see the noise map (c).
(c) Estimated Noise Map
(c) Denoised by Ours Figure 22: For extremely low-light images, there is serious color bias in the expected denoising result (a) and captured noisy image (b). This color bias is still retained in the denoising result, but the noise basically removed. This is discussed Sec. H.2
(a) Reference Image
(b) Noisy Image
H Concluding Remarks
In this study, we introduce Masked Pre-train then Iterative fill (MPI), a zero-shot denoising paradigm utilizing pre-trained model with random masks natural images. The pre-trained weights is opti- mized on a specific noisy image through Iterative filling process, and predictions with corresponding masks during inference combined for enhanced quality and faster inference.
H.1 Broader impacts
From the perspective of our work, we have pioneered the use of generalizable knowledge from natural images without any assumptions about noise degradation, offering efficient framework for handling diverse synthetic and real noises with significantly reduced inference time, which is critical issue in zero-shot denoising and makes their practical applications feasible. Notably, our zero-shot method excels in generalization compared to current supervised and unsupervised methods, offering new insights into denoising.
H.2 Limitations
Although proposed MPI has shown effects in removal of various types of noises, the mask-based noise-supervised denoising setting does not seem allow the removal of non-zero mean noise. So when dealing with extremely low-light images with severe color bias, the color bias still remains after denoising; this common problem in zero-shot denoising, because there no prior regarding noise-clean image pair in specific domain, but it may limit several practical applications, and we are currently trying solve this problem other ways.
10

---

I
Additional Qualitative Results
The following figures show the denoising comparison on both synthetic noise removal (see Fig.
Fig. 31 and denoising real noise data (Fig. 32 Fig. 35
Gaussian a=10 28.13/0.600
Reference PSNR/SSIM
DIP 34.03/0.223
N2S 30.57/0.893
2S-N2N 34.33/0.90
34.57/0.940
FPsterD1P
Ours (Faster)
Ours 36.84/09557
34.17/0.920
35.81/0.946
Figure 23: Qualitative comparison of results on CBSD 34 with Gaussian ø=10. Noisy patch is from CBSD-11.

---

Coussian 25
Reference
DIP
20.53/0.352
PSNR/SSIM
20.
N2V 39.19/0.843
N2S 28.2200.777
2S-N2N
29.3280773
PP
Ours (Fnster)
Ours 31.39/0.864
29.62/0.805
30.85/0.851
Figure 24: Qualitative comparison of results CBSD CBSD-31.
34
with Gaussian 0=25. Noisy patch is from
12

---

Goussian 550
Referunce
DIP
15.89/0.133
PSNR/SSIM
29.699L880
N2V
N2S 23.06/0.776
25-N2N 27.15/0666
27.10/0.775
FPs 29.31/0.784
Ours (Caster)
Ours 30.14/0.793
29.56/0.809
Figure 25: Qualitative comparison of results Kodak 46 kodim20.
with Gaussian 0=50. Noisy patch is from
13

---

Poisson A=10
Reference
DIP
14.62/0.462
PSNR/SSIM
19.73/0.671
N2V
N2S
78 N2N
20.28/0.730
20.83/0.693
21.75/0.733
IPsterDIP 20.65/0.637
Ours (foster)
Ours 22.18/0.751
22.15/0.744
Figure 26: Qualitative comparison of results on CBSD 34 with Poisson A=10. Noisy patch from CBSD-33.
14

---

Poisson A=25 19.21/0.465
Reference
DIP
PSNR/SSIM
26.61/0.785
N2V
N2S
75 N2N 26. 26.84/0.796
25.56/0.803
25.39/0.772
PasterDIP 25.70/0.715
Ours (faster)
27.58/0.819
27.80/0.833
Figure 27: Qualitative comparison of results on CBSD 34 with Poisson A=25. Noisy patch from CBSD-56.
15

---

CASTROL
L05170
CASTROL
COSTrO
Poisson 2=50
Reference
DIP 29.68/0.794
20.63/0.419
PSNR/SSIM
N2S 27,07/0.761
2S-N2N 29.00/0.772
28.74/0.808
CASTROL
CASTROL
FasterDIP
Ours (faster)
Ours 30.5000.830
28.43/0.756
30.0300816
Figure 28: Qualitative comparison of results on CBSD 34 with Poisson A=50. Noisy patch is from CBSD-05.
S&P d=1.045
Refermoe PSNR/SSIM
SwinIR 25.94/0.707
Restormer 24.34/0.663
Nbr2Nbr 28.23/0.731
B2U 28.57/0.160
DIP 29.30/0.765
75 NZN
36.40/0.958
Durs (faster)
Our
36.11/0.962
28.82/0.975
Figure 29: Qualitative comparison of generalization on Kodak [46
with S&P d=0.045. Noisy patch
from kodimll.
16

---

Reference
Restomer 29.74/0.756
24.56/0.49
PSNR/SSIM
23..21
Nbv2Nbr
B2U 31 44/0.784
DIP
75- 2SN 32.98/0.855 853
22.26/00803
32.81/0.819
Ours (fester)
Ours 34.94/0.893
33.92/0.870
Figure 30: Qualitative comparison of generalization Kodak from kodim02.
46
with NLF
40
Noisy patch is
17

---

mD Tum
am
Poisson 2 m40
Reference
SwinIR
Restormer 29.31/0.754
20.23/0.304
PSNR/SSIM
30.26/0.765
Nbr2Nbr
B2U
DIP
ZS-N2N 30.22/0.758
31.41/0.810
29.29/0.714
31.55/0.807
Bmi Tims
Ours (fester) 32.09/0.841
Ours 32.89/0.857
Figure 31: Qualitative comparison of generalization on Kodak 466 from kodim04.
with Poisson A=40. Noisy patch
18

---

Noisy
Reference
38 EBL.48/0.932
PSNR/SSIM
N2V
N2S*
Z.S. N2N 38.37/0.931
28.63/0.941
32.96/0.896
1P 38.47/0.930
Ours (Faster) 38.05/0.946
Ours 39.79/0.952
Figure 32: Qualitative comparison of realnoise on PolyU 3,200_ 1600 classroom__k 14.
45
Noisy patch is from Sony_3 3-
19

---

PolyU
Reference PSNR/SSIM
36.68 68.68/09924
24/.0/0888
N2V
N25* 31.96/09199
2S-N2N 36.85/0.927
35.37/0.934
PasterD 34.48/0.879
Ours (faster) 37.13/0.935
Ours 37.56/0.941
Figure 33: Qualitative comparison of realnoise PolyU 45
Noisy patch is from
Canon5D2_5 200 0200 toy__.
SIDD
Reference PSNR/SSIM
27.9100.506
334.340.888
16/0.833
31 31.53/0.776
ZS-N2N 29.300.591
FasterDIP 32.47/0.854
Ours (faster)
32.61/0.867
34.7 78/99499
Figure 34: Qualitative comparison of realnoise SIDD 11] Noisy patch is from SIDDval_20 _0_8.
20

---

ixel
ixel
iiel
iixel
Pixel
ne by Ca
by
ne by C
ne by
ne by G
Reference
N2V
25.73/0.390
PSNR/SSIM
37.65/0.940
30.40/0.850
30.07/0.706
iixel
Pixel
Pixel
Pixel
ne by
ne by
ne by
7S-NZN
FP 37.13/0.939
Ours (fester)
Ours
27.49/0.499
35.21/0.913
37.74/09944
Figure 35: Qualitative comparison of realnoise SIDD [
Noisy patch is from SIDDval 13 19.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set assumptions and a complete (and correct) proof?
Question:
Answer: [NA].
22

---

Justification: NA
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments. the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might may be
suffice, or if the contribution necessary to either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that appropriate the research performed.
specific model and empirical evaluation,
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reprodueibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make it clear how
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or way to reproduce the model (c.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
23

---

Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https
onPoiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparame- ters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: NA
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar the standard deviation or the standard error the mean.

---

It is OK to report 1- 1-sigma error bars, but one should state it. The authors should preferably report 2-sigma error bar than state that they have a 96% CI, if the hypothesis Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer:
[Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hatps:///oorrrsssccooool///hlcld
Question:
Answer:
[Yes]
Justification: NA
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), faimeess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
25

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ hommdatasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: NA
Guidelines:
The answer NA means that the paper does not release assets.
26

---

Researchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "xuHvgGtoKN", "reviewer": "Reviewer_mCpu", "summary": "The paper introduces a novel zero-shot image denoising paradigm called Masked Pre-train then Iterative fill (MPI). The key contributions are:\n\nMPI first pre-trains a model on a large dataset of natural images using a pixel-wise masking strategy. This allows the model to learn the underlying distribution and representations of natural images.\nDuring the zero-shot inference stage on a single noisy image, MPI optimizes the pre-trained model to predict the masked regions, and only uses the predictions of the masked regions to assemble the final denoised output.\nThis approach leverages the generic knowledge from pre-training, preventing overfitting during the inference stage and enabling high-quality denoising with a marked reduction in inference time compared to prior zero-shot methods.\nThe paper demonstrates MPI's superior performance across various noise types and its ability to generalize to medical images, which are distinct from the natural images used in pre-training.", "questions": "no", "limitations": "see weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThe paper presents a novel zero-shot denoising paradigm that is significantly different from prior approaches. The key innovation is the use of masked pre-training on natural images to build a generic and robust model for zero-shot denoising. This is an original idea that departs from previous zero-shot methods that relied on carefully designed priors or network architectures. Leveraging masked pre-training for this task is a novel and creative approach.\n\nQuality:\nThe technical details and experimental evaluations in the paper are of high quality. The authors provide a clear and thorough explanation of the MPI framework, including the masked pre-training and iterative optimization steps. The experimental setup is comprehensive, analyzing performance on various noise types, real-world datasets, and even medical images. The results demonstrate significant improvements over prior zero-shot methods, validating the effectiveness of the proposed approach.\n\nClarity:\nThe paper is well-written and easy to follow. The introduction provides a clear motivation and background for the problem. The method section explains the MPI framework in a structured and logical manner. The experimental section is organized in a way that allows the reader to easily understand the different evaluations and findings. Overall, the clarity of exposition is a strength of this paper.\n\nSignificance:\nThe problem of zero-shot image denoising is an important and challenging task in computer vision. Prior methods have limitations in terms of generalization and computational efficiency. The MPI approach presented in this paper addresses these limitations in a novel way. If successful, this could lead to significant practical impact by enabling high-quality denoising with minimal user intervention or computational overhead. The ability to generalize to diverse noise types and even medical images further enhances the significance and potential impact of this work.", "weaknesses": "Comparison to other zero-shot methods:\nThe paper primarily compares MPI to a few selected zero-shot denoising methods. However, it would strengthen the work to include a more comprehensive comparison to a wider range of zero-shot techniques, including recent advances. This would help contextualize the performance gains of MPI and highlight its specific advantages over the state-of-the-art.\n\n[1] Xie Y, Yuan M, Dong B, et al. Unsupervised image denoising with score function[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[2] Jang H, Park J, Jung D, et al. PUCA: patch-unshuffle and channel attention for enhanced self-supervised image denoising[J]. Advances in Neural Information Processing Systems, 2024, 36.\n[3] Garber T, Tirer T. Image restoration by denoising diffusion models with iteratively preconditioned guidance[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 25245-25254.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your suggestions. We studied the three works you provided carefully. The first is a score-based denoising algorithm, the second is an improvement on blind-spot networks (PUCA), and the third is a diffusion-based image restoration method (DDPG). The first two seem to be unsupervised denoising approaches based on datasets, which learn to recover denoised images unsupervisely from a dataset containing only noisy images.\n\nFor the first paper, we could not find their code on GitHub. Reproducing this paper within a week is challenging, so we did not compare our method with theirs.\n\nFor the second paper, we provide results for its modified zero-shot version PUCA* and the original PUCA in the table below. The zero-shot method uses an ensemble approach to ensure a fair comparison.\n\nFor the third paper, we provide results (DDPG) in the table below. This diffusion method requires the noise variance as known information. We use the variance calculated from the difference between the noisy and clean images to ensure the best performance for this method (though this might risk ground truth leakage).\n\nWe have also compared several other self-supervised and diffusion methods. If interested, please see $\\textit{Table 1}$ in PDF. For the visual results on SIDD, see $\\textit{Fig. 5}$ in PDF.\n\n|                      |      PUCA*      |       PUCA       |      DDPG       |\n|:--------------------:|:---------------:|:----------------:|:---------------:|\n| CSet+Gauss σ=10     |        -        |        -         |    32.43/0.826  |\n| CSet+Gauss σ=25     |   24.74/0.640   |        -         |    27.07/0.606  |\n| CSet+Gauss σ=50     |        -        |        -         |    15.95/0.183  |\n| SIDD validation     |   33.52/0.816   |   37.49/0.880    |    29.84/0.612  |\n| PolyU               |   33.31/0.927   |        -         |    35.79/0.887  |\n| FMD                 |   30.22/0.808   |        -         |    30.41/0.735  |\n| Avg. Infer. time (s) |    450.0        |        -         |     24.3        |\n\n\n$\\textbf{Analysis:}$\nBlind-spot methods like PUCA handle strong spatially correlated noise well in real-world scenarios. However, in its zero-shot version, it may risk over-fitting due to a lack of sufficient training data.\n\nDiffusion methods like DDPG can handle various types of degradation and are inherently robust to noise due to their training on Gaussian noise. However, they struggle to generalize to real-world noise scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dGsAxBeY2J", "reviewer": "Reviewer_iS6g", "summary": "This paper proposes a zero-shot image denoising method called Masked Pre-train then Iterative fill (MPI). The key idea is to pre-train a model on natural images using masked image modeling, then apply this pre-trained model to denoise new images in a zero-shot manner through an iterative optimization process. The authors demonstrate that their approach outperforms existing zero-shot denoising methods across a variety of noise types and datasets, while also being more computationally efficient.", "questions": "Is it possible to combine mask prediction and iterative filling, and actually use the statistical properties of noise to denoise? Make predictions under different masks each time, and integrate multiple predictions through iterative filling, and use methods similar to finding the mean between multiple predictions to achieve some denoising purpose? This is about the working nature of this method. I think it is very necessary to use experiments to prove what the model has learned and why it denoises.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Masked modelling in image denoising and Iterative processing is interesting.\n2. And the above idea works, it is more interesting.\n3. Somewhat good results.", "weaknesses": "1. I have to say that the writing of this paper is problematic. The method of this paper is not complicated, and it is even a little simple. However, the presentation of this paper does not show its core method. Some of the descriptions are irrelevant, but important analysis such as the core of its method is missing. \n 2. This paper uses the idea of ​​pre-training, but follows zero-shot. A natural question is, what does the model learn from pre-training, and how is it applied to zero-shot denoising? I understand that it is difficult to give a theoretical answer, but I think there is at least an abstract explanation to verify and demonstrate the essence of its method.\n3. Assume that the model learns the distribution of images from pre-training. But according to its experiments, the trained model can be used for unnatural images. If the model learns denoising from pre-training, the model can generalize to other noises to a certain extent. Unless it learns a common property of some noise. The key is, what is this property?\n4. The iterative approach is interesting, and I think the author may have been inspired by Diffusion to a large extent. But it is not enough to simply assume without a clearer theoretical motivation to support this approach.\n5. The choice of denoising is also questionable. In fact, I think the method in this article may be helpful for any image restoration task that meets the conditions. Why do we only discuss denoising?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1: Lack of essence of our method}$\n\n$\\textbf{A1:}$ Thank you for your pointing out. The core of our method lies in the inherent denoising capability of a model pre-trained with masked natural images. This motivation is demonstrated in main text. The pixel-level random masking we employ can be viewed as a form of noise, disrupting the image structure, and the model is trained to restore these structures. Due to training on a large set of natural images, the limited model parameters are unable to precisely fit all image distributions and instead tend to prioritize learning the main features of the images while discarding noise that varies across different images, thus acquiring a certain denoising capability. Similar masked training strategies are used in many self-supervised denoising tasks [Ref4, Ref5, Ref6 in author rebuttal], where models are trained on a large set of noisy images to learn the corresponding clean images. We further extend this to a zero-shot version, combining it with a large amount of easily accessible natural images to enhance the performance of zero-shot methods.\n\n$\\textbf{Q2: What model learns from pre-training and how it is applied to zeroshot denoising}$\n\n$\\textbf{A2:}$ We address your concerns from three aspects:\n\n$\\textit{What does the model learn from pre-training?}$\n\nThe model learns to restore randomly masked image content from a large set of natural images. This restoration process is somewhat noise-resistant. In short, a model pre-trained on a large dataset tends to recover denoised image content, functioning as a kind of denoising autoencoder (see $\\textit{Fig.1}$ in the PDF for motivation).\n\n$\\textit{How is it applied to zero-shot denoising?}$\n\nThe pre-trained weights are iteratively optimized on a noisy image through random masking, and predicted pixels are integrated using an exponential moving average, resulting in a denoised image. The learned denoising representation provides a better initial weight and helps avoid over-fitting to the noisy image.\n\n$\\textit{Analysis of why pre-trained features helped zero-shot denoising.}$\n\nWe analyzed the impact of pre-trained features on zero-shot denoising at the hidden layer level (see $\\textit{Fig.3}$ in the PDF). Features extracted with pre-trained weights significantly differ from those extracted from scratch (Baseline, i.e., the usual zero-shot denoising approach). The pre-trained model restores the complete image, with more distinct features between layers, whereas the baseline model's features are less differentiated between layers, tending to only restore the masked parts, leading to local optima.\n\n$\\textbf{Q3: Why a model does not learn denoising can denoise and generalize to unnatural images}$\n\n$\\textbf{A3:}$ As per $\\textbf{A1}$, we believe that the model pre-trained with masked images acquires a certain robustness to noise, enabling it to perform denoising. Regarding its applicability to non-natural images, the use of pixel-level random masking allows the model to restore masked content based on the unmasked areas. Since the unmasked pixels are spatially distributed evenly, the model tends to focus more on local, low-level features such as texture and color rather than high-level semantic features. These low-level features share some commonalities across all types of images, allowing the model to be applicable to significantly different types of images, such as medical images and natural images.\n\n$\\textbf{Q4: Motivation of our approach}$\n\n$\\textbf{A4:}$ Thank you for pointing out, our motivation comes from the blind spot network in self supervised denoising, which learns to reconstruct noisy images cropped by blind-spots (much resemble pixel-wise random masking) to denoise on noisy datasets that do not include ground truth. This is a widely explored field, and we further expand it with using natural images to obtain a better zero shot denoising algorithm.\n\n$\\textbf{Q5: Why only discuss denoising}$\n\n$\\textbf{A5:}$ Thank you for your suggestion. Our motivation stems from findings in the self-supervised denoising field, so this paper focuses solely on denoising. As per $\\textbf{A1}$, the trained model tends to acquire smooth representations beneficial for denoising. Since our current model uses a small number of parameters for efficient denoising, it might struggle to generate sufficient details in other tasks such as deblurring or super-resolution. However, we believe that with further improvements, this method has great potential in other types of degradation tasks.\n\n$\\textbf{Q6: Answer to Questions}$\n\n$\\textbf{A6:}$\n1. During the inference phase, we generate predictions using random masks and iterative filling, with the final denoised image comes from the exponential moving average of multiple inferences for each pixel. The only difference is that we use weights pre-trained on natural images with random masking as the initial weights, which speeds up inference and reduces the risk of over-fitting.\n\n2. Regarding what the model has learned and why it denoises, we believe that masked pre-training on large datasets causes the model to focus on the main features of images (relatively lower-frequency, easier-to-recover features), which gives the model robustness to noise and enables it to perform denoising.\n\n3. we have analyzed the differences in features between pre-trained and from-scratch models during inference to demonstrate the advantages of pre-training over starting from scratch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VulxTbXl4r", "reviewer": "Reviewer_VFTG", "summary": "This paper proposes a method that could handle image denoising regardless of the noise types and intensity. To achieve this goal, the proposed method includes two crucial steps: first, the model will be pretrained on a large amount of images (with masking); second, the pretrained model will be fine tuned on the given noisy image so that denoising. The authors validate the efficiency of their method on images corrupted by different noises.", "questions": "I have listed my questions in the section of [Weaknesses].", "limitations": "I have listed the limitations in the section of [Weaknesses].", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is well written and it is easy to follow.\n2) The authors have conducted experiments on different types of images including natural images and medical images.\n3) In addition to synthetic noises, the authors also conducted experiments on real-world noise.", "weaknesses": "1) The idea sounds very trial and the novelty is limited. It is similar to MetaDIP[ref1] and DGP[ref2].\n\n2) The idea of \"ensemble for total T steps\" is also proposed by DIP. DIP has shown that by averaging the outputs, the performance could be improved. So, the question here is: when the authors compare the performance with DIP, have you applied this similar \"average smoothing\" to DIP's outputs? If not, the comparisons here may not be fair.\n\n3) For comparisons, the authors may consider other DIP models such as Ref3.\n\n4) How this proposed method compare with other SOTA models such as Diffusion Models Ref4.\n\n\nRef1: Zhang, Kevin, et al. \"MetaDIP: Accelerating deep image prior with meta learning.\" arXiv preprint arXiv:2209.08452 (2022).\n\nRef2: Pan, Xingang, et al. \"Exploiting deep generative prior for versatile image restoration and manipulation.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.11 (2021): 7474-7489.\n\nRef3: Jo, Yeonsik, Se Young Chun, and Jonghyun Choi. \"Rethinking deep image prior for denoising.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\nRef4: Wang, Yinhuai, Jiwen Yu, and Jian Zhang. \"Zero-shot image restoration using denoising diffusion null-space model.\" arXiv preprint arXiv:2212.00490 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1: Similarities and differences compared to MetaDIP and DGP}$\n\n$\\textbf{A1:}$ Thank you for your pointing out. Our method shares some similarities with MetaDIP and DGP. MetaDIP learns denoising by obtaining initial weights beneficial for downstream tasks, while our method uses masked training on natural images to enhance downstream denoising. However, MetaDIP and DGP seem to require known degradation models, which may not be applicable for some noise types. In contrast, our method learns to recover denoised images from masked noisy ones, without relying on models tailored to specific degradation types. This eliminates the need for degradation modeling, making it adaptable to unknown noise types, real noise, and various image types.\n\n$\\textbf{Q2: Comparison fairness issues caused by ensemble}$\n\n$\\textbf{A2:}$ Thanks for your attention. We acknowledge that ensemble technique are existing methods. Our focus is on demonstrating the advantages of masked pre-training priors.\n\nTo ensure fair comparison, we explained this in the experimental section of the main text. We used EMA ensemble for DIP and FasterDIP, as detailed in $\\textit{lines 170-173}$ in main text. Results for other comparison methods (N2V and N2S) with EMA ensemble are shown in $\\textit{Supplementary Materials G.2 (lines 600-608, Tables 12 and 13)}$. Our “faster” version performs not the best in some settings, but the 1000-step version consistently leads.\n\nWe chose the EMA version for other methods because our experiments showed that EMA yields the best results compared to other ensemble methods (like averaging), as detailed in $\\textit{Table 5} of the main text.\n\n$\\textbf{Q3: Comparison with other DIP methods}$\n\n$\\textbf{A3:}$ We considered the work you mentioned (DIP-SURE) when selecting comparison methods. They designed denoising solutions for Gaussian and Poisson noise to achieve high-quality denoising. However, their method requires additional noise variance as input, which could lead to unfair comparisons. Additionally, their method seems to be specific to certain types of noise, and their official code only supports Gaussian and Poisson denoising. This is why we did not compare with their method initially.\n\nWe included results for DIP-SURE using EMA ensemble, reporting both peak performance and final performance. For a fair comparison, we should compare our method with the final performance, as peak PSNR is not known without ground truth. Please refer to $\\textit{Table 1}$ in PDF for a comparison between our method and DIP-SURE.\n\nFor real datasets (SIDD, PolyU, and FMD), we computed the variance from the difference between noisy and clean images to obtain denoised results. Their approach cannot remove real-world noise well, and some artifacts exists (refer to $\\textit{Fig.5}$ in provided PDF).\n\n$\\textbf{Q4: Comparison with other Diffusion methods}$\n\n$\\textbf{A4:}$ Existing SOTA diffusion models, like DDNM, recover degraded images by decomposing into the null-space and adjusting the noise scheduler for denoising. Although diffusion models are Gaussian denoisers and can handle Gaussian noise well with proper adjustments (see PDF $\\textit{Table 1}$), they also require known noise variance in advance. Additionally, when set for denoising, the diffusion model relies solely on the noise scheduler, effectively becoming a Gaussian denoiser. This limitation may prevent the model from fully removing more complex real-world noise (see $\\textit{Fig. 5}$ in PDF).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3WpIwL6Qcw", "reviewer": "Reviewer_uJXC", "summary": "The paper proposes a novel zero-shot image denoising method named Masked Pre-train then Iterative fill (MPI). This method leverages a pre-trained model on vast natural images using a masking strategy to learn generalized image distributions, enabling effective denoising without prior knowledge of the specific noise type.", "questions": "Please check weaknesses.\n\n1. What is the main reason that higher masking rate $p$ is required for real-world noise?\n\nMinor points and typos\n   - I cannot see any difference in the images in Figure 2, even for the noisy image. Same to Figure 6, it would be better if there are enlarged views.\n   - Missing space in line 89 i.e.Masked -> i.e. Masked\n   - typo in line 248? forward pass 2.3 ?\n   - Maybe a mistake? In the caption Table 4, what does the `defaults in gray` mean?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Novel and sound idea, with clear benefits.\n- Good generalization performance for unseen noise types.\n- Thorough analysis of their method and results.", "weaknesses": "- Regarding real noise removal, it is well known that spatial correlation of the real noise makes pixelwise masking based methods like N2V or N2S to fail at denoising, making them inappropriate comparatives. Several self-supervised denoising methods have been designed to remove real world noise (e.g., AP-BSN [1], MM-BSN [2], etc.). While these are not zero-shot methods, zero-shot modifications of these blind spot networks would serve as better comparative methods, especially since the authors have already modified N2N and N2S into zero-shot versions.\n\n- Understanding and discussion of masking ratio can be improved. Perhaps employing a random masking ratio within a certain range (e.g., $p\\in[0.3, 0.6]$ would be more effective and could result in an optimal point that is applicable to both synthetic and real noise situations. And why does the masking ratio differ for different noise types? What is the main factor in choosing the best masking ratio?\n\nAlso, if this masking ratio is a crucial hyperparameter that changes depending on the image and noise types, it could weaken the paper's key contribution regarding its generalization ability for any noise types.\n\n.\n\nReferences\n\n[1] Lee, Wooseok, Sanghyun Son, and Kyoung Mu Lee. \"Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Zhang, Dan, et al. \"Mm-bsn: Self-supervised image denoising for real-world with multi-mask based on blind-spot network.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1: More comparison with zero-shot modifications of blind spot methods}$\n\n$\\textbf{A1:}$ Thank you for your thorough consideration. We have revised AP-BSN, MM-BSN, and PUCA (as shown in the overall rebuttal to all reviewers), and the results are listed in $\\textit{Table 1}$ in provided PDF. Here are some Implementation details below:\n\nWe followed the original settings of these methods. In each iteration, we cropped 8 same-size patches from a noisy image to form a batch for training. Every 10 iterations, we performed inference on the full image to obtain denoised images. These denoised images were then combined using the same ensemble strategy as our method to ensure fairness.\n\nThese methods can effectively denoise spatially correlated noisy images. However, using only one noisy image for training can lead to overfitting and produce artifacts, as shown in $\\textit{Fig. 5}$ of the PDF.\n\n$\\textbf{Q2: More discussion about masking ratio}$\n\n$\\textbf{A2:}$ Thank you for your suggestion; it is very insightful. We are still exploring this issue. However, we are concerned that using a large masking ratio during inference might degrade synthetic noise removal performance. Here is a summary of the impact of different masking ratios $p$ on denoising:\n\nSmall $p$: Risk of learning noise patterns from real noisy images.\n\nLarge $p$: Loss of detail in the denoised image.\n\nThe difference in masking ratios for various noise types is mainly because real noise has strong spatial correlation, necessitating a larger masking ratio to avoid learning the noise distribution (see more analysis in $\\textbf{A3}$).\n\nChoosing the masking ratio is crucial. When noise is spatially uncorrelated (e.g., Gaussian, Poisson, S&P noise), a consistent $p$=30 works well across all cases. However, the spatial correlation of real noise complicates this issue. Many blind-spot networks (e.g. AP-BSN) designed for self-supervised denoising also aim to address this problem.\n\n$\\textbf{Q3: Main reason for different $p$}$\n\n$\\textbf{A3:}$ This primarily depends on the spatial correlation of the noise in the image. Synthetic noise is spatially uncorrelated, with noise signals are uncorrelated in neighbor positions. In contrast, real noise, after passing through a series of ISP processes, has a much more complex distribution, resembling blurred spots rather than independent points (refer to $\\textit{Fig. 2}$ in PDF). For synthetic noise, choosing a small masking ratio can help quickly recover more details in the image. However, for real noise, with a small masking ratio, the model can fit the noise distribution based on neighboring pixel values. In this case, a larger masking ratio can mitigate the impact of noise. One paper [Ref1] also discusses using different dropout ratios for different types of images.\n\n$\\textbf{Q4: Minor points and typos}$\n\n$\\textbf{A4:}$ Thanks very much for your corrections. \n\n1. Regarding Fig. 2 and Fig. 6 in main text, we have provided enlarged views, as seen in $\\textit{Fig. 1 and Fig. 4}$ in PDF.\n\n2. Yes, we have addressed the spacing issue on line 89.\n\n3. \"2.3\" on line 248 refers to Section 2.3, which we have clarified in the main text.\n\n4. Due to formatting constraints, we shortened the caption. \"Defaults in gray\" means that the gray background in the table indicates the default settings used in our work for comparison with other methods.\n\n$\\textbf{Ref:}$\n\nRef1: Self2Self+: Single-Image Denoising with Self-Supervised Learning and Image Quality Assessment Loss, Arxiv'23", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the time and efforts of all the reviewers, as well as their valuable suggestions provided during the review process. We are encouraged by the reviewers' recognition of our work and acknowledge that there are still many weaknesses in our current work. We carefully considered and responded to every question from the reviewer to the best of our ability.

Here we list some common weaknesses and corresponding brief responses, more details can be found in responses to each reviewer.

$\textbf{1. More comparison methods}$ including additional zero-shot methods, such as DIP-based (DIP-SURE [Ref1]) and diffusion-based methods (DDNM [Ref2], DDPG [Ref3]), and zero-shot modifications from unsupervised methods (AP-BSN [Ref4], MM-BSN [Ref5], PUCA [Ref6]), refer to $\textit{Table 1}$ in provided PDF. The implementation details of each method can be found in rebuttal to each reviewer.

$\textbf{2. Analysis of the Priors Learned from Pretraining}$ (Refer to $\textit{Fig. 3}$ in the provided PDF). In the main text, we have explained that this masked pre-training not only helps provide a better starting point for the model, but also helps prevent the model from over-fitting to some extent (see $\textit{Section 4.1}$ Pre-trained weights in main text), offering a more stable zero-shot denoising process. Here we analyze the representations learned by the pre-trained model and try to explain why this helps avoid over-fitting. In Fig. 3, we used CKA analysis [Ref7] to show that the image features extracted by the pre-trained model are significantly different from those extracted by the model trained from scratch (“Baseline”). Due to insufficient data, baseline model tends to acquire similar representations across layers and focus more on recovering the masked parts of the image. This can lead to local optima and early over-fitting to noise. In contrast, pre-trained model learns from a larger variety of images and provides a better startpoint. They focus more on recovering the entire image rather than just the masked parts, which is smoother and reduces the risk of over-fitting.

$\textbf{3. Restatement of Motivation and Focus in This Work.}$ This paper focuses on leveraging a large amount of easily accessible natural images to achieve better zero-shot denoising performance. To ensure the method is simple and efficient, and to highlight the importance of pre-training, we do not extensively explore more complex network mechanisms, since it is not the focus of our paper. By relying on minimal dependencies for denoising, our approach can be further improved when combined with existing methods like random sub-samples (see $\textit{Section 4.2}$ in main text for discussion).

$\textbf{Ref:}$

Ref1: Rethinking deep image prior for denoising, ICCV’21

Ref2: Zero-shot image restoration using denoising diffusion null-space model, ICLR’23

Ref3: Image restoration by denoising diffusion models with iteratively preconditioned guidance, CVPR’24

Ref4: Ap-bsn: Self-supervised denoising for real-world images via asymmetric pd and blind-spot network, CVPR’22

Ref5: Mm-bsn: Self-supervised image denoising for real-world with multi-mask based on blind-spot network, CVPR’23

Ref6: PUCA: patch-unshuffle and channel attention for enhanced self-supervised image denoising, NIPS’24

Ref7: Similarity of Neural Network Representations Revisited, Arxiv’19

Meta-review: Scores 6 4 5 6 

Nice paper, however, from the reviews just at the border between reject and accept.

The reviewers raised concerns about the evaluations, in particular to compare against more recent methods.  These concerns were addressed by the rebuttal, where they added more comparisons.  One reviewer found the novelty limited due to similarities to the methods MetaDIP and DGP.  The rebuttal addressed this and expalined the differences.

Overall, I am leaning to accept the paper, since the reviewers are mainly positive about the paper and it presents solid work.  Thus: accept.
Decision: Accept (poster)