Generative Recorrupted-to-Recorrupted: An Unsupervised Image Denoising Network for Arbitrary Noise Distribution
Authors: wan Bowen
Daming Shi
Venue: Submitted to ICLR 2023 (2023)
Keywords: Image denoising
Unsupervised
Deep learning

Abstract: With the great breakthrough of supervised learning in the field of denoising, more and more works focus on end-to-end learning to train denoisers. The premise of this method is effective is that there is certain data support, but in practice, it is particularly difficult to obtain labels in the training data. To this end, some unsupervised denoisers have emerged in recent years, however, the premise of these methods being effective is that the noise model needs to be known in advance, which will limit the practical use of unsupervised denoising. In addition, inaccurate noise prior from noise estimation algorithms causes low denoising accuracy.  Therefore, we design a more practical denoiser that requires neither clean images as training labels nor noise model assumptions. Our method also needs the support of the noise model, the difference is that the model is generated by a residual image and a random mask during the network training process, and then the input and target of the network are generated from a single noisy images and the noise model, at the same time, train an unsupervised module and a pseudo supervised module. Extensive experiments demonstrate the effectiveness of our framework and even surpass the accuracy of supervised denoising.

Under review as a conference paper at ICLR 2023
GENERATIVE RECORRUPPED-DO-REEEOUUUDDD) AN UNSUPERVISED IMAGE DENOISING NETWORK FOR ARBITRARY NOISE DISTRIBUTION
Anonymous authors Paper under double-bling brind review
ABSTRACT
With the great breakthrough of supervised learning in the field of denoising, more and more works focus on end-to-end learning to train denoisers. The premise of this method is effective is that there is certain data support, but in practice, it is particularly difficult to obtain labels the training data. To this end, some unsu- pervised denoisers have emerged in recent years, however, the premise of these methods being effective that the noise model needs to be known in advance, which will limit the practical of unsupervised denoising. In addition, inaccu- rate noise prior from noise estimation algorithms causes low denoising accuracy. Therefore, design more practical denoiser that requires neither clean images as training labels nor noise model assumptions. Our method also needs the sup- port of the noise model, the difference is that the model is generated by residual image and a random mask during the network training process, and then the input and target of the network are generated from single noisy images and the noise model, at the same time, train an unsupervised module and pseudo supervised module. Extensive experiments demonstrate the effectiveness of our framework and even surpass the accuracy of supervised denoising.
INTRODUCTION
Image denoising is traditional topic in the field of image processing, and it is the basis for the success of other vision tasks. A noisy image can be represented by y = x + n, and our task is to design denoiser to remove the noise in the noisy image.
The denoising convolutional neural network DNCNN Zhang et al. 2017 can be considered a bench- mark of the use of deep learning for image denoising, and it introduced residual learning and batch normalization, which speed up the training process as well as boost the denoising performance. The fast and lexible denoising neural network FFDNET Zhang K. & Zhang (2018) treated the noisy model as a prior probability distribution, such that it can effectively handles wide a range of noise levels The convolutional blind denoising CBDNET Guo et al. (0199) went further than the FFDNET Zhang K. & Zhang (2018) and aimed at real photographs though synthesized and real images were both used in training.
A common treatment for the above methods is that it all need to take noisy-clean image pairs in training. However, in some scenarios such as medical and biological imaging, there often lack clean images, leading to an infeasibility of the above methods. To this end, the noise-to-nee (N2)ss method Lehtinen al. (2018) was the first to reveal that deep neural networks (DNNs) can be trained with pairs of noisy- images instead of noisy-clean images, in other words, training can be conducted with only two noisy images that are captured independently in the same scene. The N2N can be used in many taskBuchholz et al. (2019); Ehret et al. (2019); Hariharan et al. (2019); Wu et al. (2019); Zhang et al. (2019), since creatively addressed the dependency on clean images. Unfortunately, pairs of corrupted images are still difficult to be obtained in dynamic scenes with deformations and image quality variations.
To further bring the N2N into practice,
2019;; Batson & Royer (201)); research Krull al.
Krull et al. (2020); |Hunn et al. (202T) concluded that it is still possible to train the network without using clean images if the noise between each region of the image is independent. Among them,

---

Under review as a conference paper at ICLR 2023
Neighbor2Neighbbor (NBR2NBR) method Huang et al. (2021 proposed a new sampling scheme to achieve better denoising effects with a single noisy image. The advantage of this approach is that it does not need prior noise model prior, like the Recorrupted teeeetedddtt--eeeeeeee to-Recorrupted (R2R) Pang et al.
(2021), nor does it lose image information, like the Noise2Void (N2V) self-supervised methods N2V
Krull et al. (2019).
However, Krull et al. (009)) : Batson & Royer (2019) Krull et al. 2020); Huang et al. (2021) Pang al.] (2021) are valid under the assumption that the noise on each pixel independent from each other, which means that they are not as effective in dealing with noise in real scenes as supervised denoising. To deal with more complex noise, some unsupervised methods have been proposed. Noise2Grad (N2G) Lin et al. (2021) extracted noise by exploiting the similar properties of noise gradients and noisy image gradients, and then added the noise to unpaired clean images to form paired training data. Waan et al. (2022a) constructed a new way of unsupervised denoising through optimal transport theory. It is worth noting that although [Lin et al. (202T); Waag et al. (2022a) no longer subject the end-to-end learning approach to pairing clean-noisy images, they still need to collect many clean images.
In order to solve the above problems, we propose new denoising mode that achieves unsupervised denoising without requiring noise prior, noise assumptions and any clean images. In one epoch of our training network, between network input and output, and use a random mask image to reduce the influence of nat- ural image information in the residual image. The generative noise model can be obtained by the above operation. In the second step, we put the model into the Pseudo Supervised module and the coarrupted-toRRecoruup
first obtain residual image containing noise through the difference
module to train the same network. At this point, one epoch of the entire training ends. Eventually after many iterations of network training, we will gradually generate more realistic noise model and perfect denoiser. More detail of our proposed Generate Recorrupted-to Recorrupted framework can be found in Figure 1.
The remainder of the paper is organized as follows. In Section 2, we introduce the related work. Then the details of our method is given in Section 3, followed by the experiments in Section 4, with the conclusions drawn in Section 5.
2
PROBLEM STATEMENT
2.1 SUPERVISED TRAINING
With the rapid development of deep learning, many supervised learning methods are applied to image denoising. DNCNN Zhang et al. (2017) successfully applied a 17-layer deep neural network to image denoising tasks by introducing residual learning. After that, a series of more efficient and complex neural networks succeed in denoising tasks. Unlike DNCNN[Zhang
al. (2017). FFDNET
Zhang K. & Zhang (2018) was more efficient in demoising, and CBDNET Guo et al. (2019) can handle more complex real noise. Without considering constraints, the above supervised denoising methods can be expressed by the following formula:
argminI (fo (x).
(1)
0
V. x. and T. are noisy images, clean images, denoising model and loss function respectively. However, these methods all require clean images as the target of training the neural network, and then optimize the parameter by calculating the gap between the network output and the target, so as to obtain a better denoising model.
The N2N Lehtinen et al. (2018) revealed that the noisy/true image pairs used train the DNN can be replaced by noisy/noisy images pairs. The corrupted pairs are represented by the y and where ni and n2 are uncorrelated. There are two main principles for N2N to successfully train network with a paired noisy image: the first is that the optimal solution obtained by network training is a mean value solution; the second is that the mean value of most noise close to zero. So the gradient generated by the network for one corrupted target is incorrect, but the gradient corresponding to the average of all corrupted images is correct, which can be expressed by the following formula:
argminI (So (v)
0
(2)
y = x + n1
z = x + 122-
2

---

Under review as a conference paper at ICLR 2023
Although the N2N alleviated the dependence on clean images, pairs of noisy images are still difficult to obtain.
2.2 SULF-SUEERVISED TRAINING
To eliminate N2N Lehtinen et al. 2018) restrictions in dynamic scene denoising, some methods utilized the original features of noisy images to construct auxiliary tasks to make self-supervision as effective as supervision. The N2V Krull et al. (2019) offered blind-spot network which can denoise using only a noisy image, and its main principle is to use the correlation among pixels to predict missing pixels. In this way, some pixel information is lost and the denoising effect is not ideal. The Self2Self Quan et al. (2020) can realize image denoising by dropout in the case of only one noisy image. Unlike N2V, it used lost information as targets, but its training time was extremely long and test image needed training model, which limit its practical application.
The NBR2NBR Huang et al. 2021)) was the most recent a self-supervised denoising method, which can be viewed as advanced version of the N2N Lehtinen et al. (2018) via novel image sampling scheme to get rid of the requirement of paired noisy images.
The following equation shows the reason why denoising is effective without clean targets under the condition that the noise mean is 0:
Eny [[ - =r12 En. (I,o(y) al2 const +Cov((fe(y) - x), (a (oa=x)) x)).
(3)
In N2N Lehtinen et al. (2018), a = 2, since n2 and n2 are independent covariance terms will disappear. Similarly in NBR2NBR Huang et al. (2021). a and y replaced by two adjacent noisy sub-images, covariance terms disappears under the local correlation of pixels and assumption of uncorrelated noise. Also in N2V Krull et al. (2019), it is denoised by blind spot network, which removes the covariance term by assuming that the noise is uncorrelated on adjacent pixels.
2.3 UNSUPERVISED TRAINING
category of unsupervised denoising requires unpaired clean-noisy images to train the denoiser. [2G[Lin  et al. (2021) obtained an approximate noise model by measuring the distance between the noise gradient generated after denoising and the original noisy image gradient, and then added it to the unpaired clean images for training. However, the approach requires not only clean images but also a lot of time to approximate the real noise distribution. Another approach Wang et al. (2022a) utilized WGAN-gr gg Guirajani al. (2017) to measure the distance between the denoised image output by the generator and the unpaired clean image to optimize the denoiser.
Another category of unsupervised denoising is when certain assumptions about the noisy model are required. AmbientGAN Bora et al. (2018), a method of training generative adversarial networks also disposed the reliance on clean images, and used measurement function generate noisy image, which was then fed into the discriminator for comparison with the original noisy image. The Noiser2Noise (Nr2N) Moran et al. (2020) re-destroyed the original noisy images through the noise prior as the input targets. Then, R2R Pang et al. (202I) overcame the disadvantage of N2N Lehtinen et al. (2018) and generated paired noisy images by introducing prior noise model. In particular, it used known noise levels form pairs of noisy images for a single noisy image. Although R2R close to N2N Lehtinen et al. (2018) in denoising effect, it is limited in applications with real noisy images because the level and type of noise from real noisy images are difficult to measure.
y + 7 to the network, and the original noisy images y were used as training
THEORETICAL FRAMEWORK
Our framework consists of three modules: Generate Noise (GN), Pseudo Supervised (PS) and Recor- rupted2Recorrupter (R2R). These three modules share neural network parameter during training, which means they are trained simultancously, but in different order in a network training iteration. First, we generate simulated noise through GN. In the second step, the noise is put into PS and R2R, and the network parameter is updated at the same time. In the next round of iterative training, GN will use the new network parameter to generate more realistic noise.
3

---

Under review as a conference paper at ICLR 2023
Generative Noise
Lg
Mask
y
y
Network fo(-)
fy(y)
n
vilo gradie
Pseudo Supervised
5
fo(fo()) +
Network
Network fa
fol
Network (a(-)
(0)
L
so (v)
Recorraptts-o-RRcoorntttee
(5) Trmining Phase
Infecunce Phase
Figure The framework of Generate Recorrupted-to- -orrupptddto-Reccrrppteddd Re-orrucrrred (GR2R). (a) Overall training process. y is the observed noisy image, the generate noise á in Pseudo Supervised (PS) and Recor- rupted2Recorrupeed (R2R) is obtained by glement-wise multiplication of random mask m and resid- ual map - (yy). The neural networks in the three modules update the same parameter in one network. Moreover, the regular loss Lg is used to stabilize the training phase, the supervised loss Lg avoids randomly generated noise from affecting unsupervised denoising, the Ly represents un- supervised loss. (b) Inference using the trained denoising model. The denoising network generates denoised images directly from the noisy images y of the test set without additional operations.
3.
RECORRUPTED2
MODULE
R2R Pang et al. (2021) is an unsupervised denoising method using training scheme that does require noisy image pairs or clean target images. The approach destroys a single noisy image through the noise prior to form the paired noisy images required in N2N Lehtinen et al. (2018).
Given the noisy observation y and noise prior n, R2R aims to minimize the following empirical risk:
argminL (ū) (8).g)
8
1 = y n' ÿ = y n'.
(4)
where fe (+) denotes the denoising model with parameter 0. This method assumes that noise n' is known before training network, however, it is difficult to accurately estimate the noise model in real noisy image denoising, which affects the performance the denoiser. The loss of equation is closely related to the one used in supervised learning:
argminL (So xx).
(5)
Proofs are as follows:
Bn,.' {150((0 0113] = Bn (I5o(() n 1+mm1) ml1122)
En,y' (lfo(() x12]
2Bn,n' { (n- - n') (fo(i) - x²)
(6)
Enn (n n') (n n'))
En,'' [ - x12
The n, nt', are the noise in the observed image, the noise prior and the clean image, respectively.
In equation 6, long and n' are independent, it can be inferred that + n' in fo(ū) and n

---

design the Pscudo Supervised module to reduce this Under review as a conference paper at ICLR 2023
independent. According the assumption of N2N Lehtinen et al. 2018 that the noise mean is O, can be found that equation 6 is equal to O. In equation 7, since the variance of the noise prior n' equal to the variance of the noise n, equation 7 is exactly equal to O. Next, going step further, how to achieve unsupervised denoising without noise prior?
3.2 GENERATE NOISE MODULE
A natural idea is generate the noisy model at training time. First we generate residual image by following equation fo (z), which contains real noise. However, since the residual image also contains a large amount of image feature information, it is difficult accurately model noise. So in the second step we introduce random mask map m, which is a vector of the same size and dimension as the noisy image, and obeys the following distribution:
1 p= 0.5
0.5
m
-1 p 0 0.5.
(8)
Then the generated noise during training can be represented as follows:
= (v - fo()) O m.
where represents element- wise multiplication. It can be seen from equation 8 that m is randomly generated and has a mean value of O. From equation 9, it can be deduced that the mean value of is Even if the original real noise n does not satisfy the assumption that the noise from Lehtinen et al. 2018) has a mean of O, the generated noise can be forced to satisfy the assumption of zero mean during network training iterations by equation 9. In addition, the operation may produce slight error in estimating the real noise model, error.
The input and target of our Recorupted2Rccorupieed are represented by the following formulas:
ú y
y + ft - y st.
(10)
is easy to see that the generated á and n are irrelevant, so n + n and n - â are unrelated. At this time, under the generated noise ń scheme, equation 6 vanish in Recorupped22Rcoorppee Since the variance of m is 1, equation 7 can be transformed into the following formula:
En, {llfo()) -v12 const.
(11)
Therefore, an optimal denoising criterion to achieve the same effect of supervised denoising can be expressed as
argminI (fo (0). . argminL (fo (v)
(12)
3.3 PSEUDO SUPERVISED
Note that both equation and equation 12 are equivalent to supervised denoising equation 5. How- in equation 5, the extra noise introduced by á will affect the accuracy in the denoiser, so R2R Pang et al.] (202I) adopts the Monte Carlo approximation to solve above problem. However, the averaging of multiple forward processes in R2R will not only greatly reduce the denoising speed on the test set but also affect the denoising accuracy. In addition, the noise we generate will have certain error, we design supervised-like approach to address the effect of the generate noise. Specifically, we use the generated fo(y) in GN as the 'clean' target and fo (y) + á as the input to train the denoiser. During training, since GN stop the update of 0, fo(y) gradually approaches the clean image without affecting the stability of training. The pseudo supervised loss as follows:
argminT (Jo (Jo (v) + n So (v)).
(13)
The total loss function of our method can be expressed as:
= Ly, + Lg + La
= Il (0) 7 Il (v) v1l - - 1IJo (Ja (v) + n) So (v)l
(14)
where fo denoising network with arbitrary network design, and 7 is a the strength of the pseudo supervised term.
controlling
5

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we evaluate our GR2R framework, which can bring significant improvement to the denoising quality of previous work.
Training Details. We use the same modified U-Net Ronneberger et al. (015) architecture as Lehtthhi nen et al. (20T8); Huang et al. (20TT): Wang al. (2022b). The batch size is 10. We use Adam Kingma & Ba 2013) as our optimizer. The initial learning rate is 0.0003 for synthetic denoising in sRGB space and 0.0001 for woaldworld denoising. All models are trained on a server using Python 3.8.5, Pytorch 1.6, and Nvidia Tesla K80 GPUs.
Datasets for Synthetic Denoising. Following the setting in Lehtinen et al. 2018); Huang et al. 2021), we select 44,328 images with sizes between 256 256 and 512 x 512 pixels from the L.SVRC2012/Deng et al. (2009) validation set as the training set. To obtain reliable average PSNRs, also repeat the test sets Kodak Franzen| 1999), , BSD300 Martin al. (2001) and Set14 Zeyde et al. (010) by 10, 3 and 20 times, respectively. Thus, all methods are evaluated with 240, 300, and 280 individual synthetic noise images. Specifically, consider two types of noise in sRGB space:
(I) Gaussian noise with 0 = 25. (2) Gaussian noise with o E 50).
Datasets for Real-World Denoising Following the setting Huang et al. (022), take the SIDD Abdelhamed et al. (2018) dataset collected by five smartphone cameras in TO scenes under different lighting conditions for real-world denoising in raw-RGB space. We use only raw-RGB images in
SIDD Medium Dataset for training and use SIDD Validation and Benchmark Datasets for validation
and testing.
Details of Experiments. For the baseline, we consider two supervised denoising methods (N2C Ronneberger et al. (2015) and N2N [ehtinen al. (2018)). We also GR2R with traditional ap- Dabov et al. (2007)) and eight self-supervised denoising algorithms (Self2Self Quan proach Daboy
ct al. (2020). Noise2 Void (N2V) Krull et al. (2019). Laine19 Laine et al. (2019), Noisier2Noise
Moran et al. (2020), DBSN Wu et al. (2020) R2R Pang al. (2021). NBR2NBR Huang et al.
(2021) and B2UB Wang et al. (2022b))
4.1 RESULTS FOR SYNTHETIC DENOISING
The quantitative comparison results of synthetic denoising for gaussian can be seen in Table 1. Whether the gaussian noise level is fixed variable, our approach significantly outperforms the traditional denoising method BM3D and all self-supervised denoising methods on BSD300 dataset, even beyond supervised learning methods. On the other two small test sets (KODAK and SET14), our method is also close to Laine 10mee19ppnn 19-pme| Laine et al. (2019), which is method that requires the same explicit noise modeling as R2R |ang al. (2021). However, explicit noise modeling means strong prior, leading to poor performance on real data. The following experiments real-world datasets also illustrate this problem. addition, figure shows that our method retains more natural image features while denoising compared to other denoisers.
4.2 RESULTS FOR WORLD DENOISING
In real raw-RGB space, Table 2 shows the quality scores for quantitative comparisons on SIDD Benchmark and SIDD Validation. Note that the online website of SIDD evaluates the quality scores for the SIDD Benchmark. Surprisingly, the proposed method outperforms the state-of-the-art (NBR2NBR) by 0.28 dB and 0.23 dB for the benchmark and validation, but even outperforms N2C and N2N about 0.1 dB. is worth noting that the unsupervised methods et al. (2019) and Pang et al. (021) relying on model prior are significantly less effective in dealing with real noise, and we even surpass Pang al. (202T) by 4.05 dB and 4.09 dB for the benchmark and validation. Obviously this type of model prior based approach is not advisable. The raw-RGB denoising performance in the real world demonstrates that our method is able simulate complex real noise distributions.
4.3 ABLATION STUDY
This section conducts ablation studies on the pseudo supervised module. Table lists the perfor- mance of different 7 values in unsupervised denoising. It can be seen that the denoising effect is

---

Under review as a conference paper at ICLR 2023
Cleun
Noisy (20.65dBB
N2C (27. 88dB)
N2N (28 05dB)
N2V (27.60JBB)
NBR2NBR 14dB)
B2UB (27.774B)
Ours (28.21dB)
BSD300-008
Clean
Noisy (20.27dB)
N2C (29. (29.43dB)
N2N (29.52dB)
N2V (28.78dB)
NBR2NBR (29.51dB)
BZUB (29.58dB)
Ours (29.748B)
(b) KODAK-000
Figure 2: Visual comparison of denoising sRGB images in the setting of 25.

---

Under review as a conference paper at ICLR 2023
Table 1: Quantitative denoising results synthetic datasets in sRGB space.
The highest
DSNR(dBySSII among unsupervised denoising methods is highlighted in bold.
Noise Type Method
KODAK
BSD300
SET14
Baseline. Ronneberger et al. (2015)
32.43/0.884
31.05/0.879 31.40/0.869
Baseline N2N Lehtinen et al. (0118)
32.41/0.884
31.04/0.878
31.37/0.88
CBM3D Dabov al.
2007)
31.87/0.868
30.48/0.861
50.88/0.854
Self2Self Quan et al. 2020)
31.28/0.864
29.86/0.849
30.08/0.88
N2V Kruil al. (2019)
30.32/0.821
29.34/0.824
28 28.84/0.802
La-ne19-mu Lame et al. (2019)
30.62/0.840
28.62/0,803
29.93/0.830
Gaussian
Laine19-pme 19-pme Lanne et (2019)
32.40/0.883
30.99/0.877
31.36/0.866
25
Noisier2Noise Moran et al.
2020
30.70/0.845
29.32/0.833
29.64/0.88 832
DBSN Wu al. (2020)
31.64/0.856
29.80/0.839
30.63/0. 846
NBR2NBR
et al.
(2021)
32.08/0.879
30.79/0.873
31.09/0.864
B2UB Wang el al. 2022b
32.27/0.880
30.87/0.872
31.27/0.866 864
R2R Pang et al.
32.25/0.880
30.91/0.872
31.32/0.80 865
Ours
32.34/0.882
31.08/0.879
31.20/0.866
Baseline, N2C Ronneberger al. (2015)
32.51/0.875
31.07/0.866
31.41/0.863
Baseline N2N Lehtinen et al. (2018)
32.50/0.875
31.07/0.866
31.39/0.866
CBM3D Dabov
2007)
32.0270.860
30.5670.847
30.9470.849
Self2Self Quan et al.
2020)
31.37/0.860
29.87/0.841
29.97/0.88 849
N2V Krull al. (2019)
30.44/0.806
29.31/0.801
29.01/0.792
Gaussian a E [5, 50]
Laine Lamne et al.
2019)
30.52/0.833
28.43/0.794
29.71/0.822
Laine 19-pme Laine et al.
(2019)
32.40/0.870
30.95/0.861
31.21/0.855
DBSN Wu et al. (2020)
30.38/0.826
28.34/0.788
29.49/0.88 814
NBR2NBR Huang cf al
(021
32.10/0.871
30.73/0.861
31.05/0.855
B2UB Wang ct al. (202bb)
b)
32.34/0.872
30.86/0.861
31.14/0.857
R2R Pang et al. (2021
31.50/0.850
30.56/0.855
30.84/0.850
Ours
32.46/0.875
31.13/0.867
31.02/0.856
Table 2: Quantitative denoising results SIDD benchmark and validation datasets in raw-RGB space.
Method
Network
SIDD Benchmark
SIDD Validation
Baseline, N2C Ronneberger et al. (2015)
U-Net
50.60/0.999
51.19/0.991
Baseline,N22 N2N Lehtinen et al. (2018)
U-Net
50.62/0.99 991
51.21/0.991
BM3D Dabov al. (2007)
48. 18.60/0.986
48.92/0.986
N2V Krull et al. 2019)
U-Net
48.01/0.983
48.55/0.984
Laine19-o 1 Laine
(2019)
U-Net
49.82/0.988 989
50.44/0.999
Laine 19 9-pme] Laine et al.
(009
U-Net
42.17/0.935
42.87/0.939
DBSN Wu
2020
DBSN
49.56/0.98.
50.13/0.988
NBR2NBR Huang et al.
2021
U-Net
50.47/0.990
51.06/0.991
R2R
Pang et al. 202
U-Net 46.70/0.977 978
47.20/0.988
Ours
U-Net 50.75/0.991
51.29/0.991
poor when no -spee-supe module participates training. When y = 5, our method achieves the highest accuracy on SIDD.
CONCLUSION
We propose Generative Recorupted2RRecorupee a novel unsupervised denoising framework, which achieves greatly denoising performance without noise prior, and it surpasses methods that require noise prior. The proposed method generates random dynamic noise in the process of train- ing the neural network, so as to solve the problem of requiring noise model prior in unsupervised denoising. In addition, the Pseudo supervised module improve the performance of unsupervised de- noising. At last, extensive experiments have shown the superiority our approach against compared methods.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Quantitative denoising results of different y on SIDD validation datasets.
5
10
15
30
PSNR/SSIM 49.41/0.989 51.23/0.991 51.29/0.991
51.23/0.991
51.20/0.991
51.06/0.99I
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1692-1700, 2018.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pp. 524-533, PMLR, 2019.
Ashish Bora, Eric Price, and Alexandros G Dimakis. Ambientgan: Generative models from lossy measurements. In International Conference Learning Reprexentations, 2018.
Tim-Oliver Buchholz, Mareike Jordan, Gaia Pigino, and Florian Jug. Cryo-care: content-aware im- age restoration for cryo-transmission electron microscopy data. In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 502-506. IEEE, 2019.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domalt collaborative filtering. IEEE Transactions image processing. 16 (8):2080-2095, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Thibaud Ehret, Axel Davy, Jean-Micher Morel, Gabricle Facciolo, and Pablo Arias. Model-blind video denoising via frame-to-frane training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11369-11378, 2019.
Rich Franzen. Kodak lossless true color image suite. source: http://rk.. us/graphics/Koduk 4(2), 1999.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo, and Lei Zhang. Toward convolutional blind denois- ing of real photographs. In Proceedings the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2019-June, 2019. doi: 10. 109/CVPR.2019.00133
Sai Gokul Hariharan, Christian Kaethner, Norbert Strobel, Markus Kowarschik, Shadi Albarqouni, Rebecca Fahrig, and Nassir Navab. Learning- based x-ray image denoising utilizing model-based image simulations. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 549-557. Springer, 2019.
Tao Huang, Songjiang Li, Xu Jia, Huchuan Lu, and Jianzhuang Liu. Neighbor2Neighbor: Self- Supervised Denoising from Single Noisy Images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14781-14790. 2021.
Diederík Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015.
Alexander Krull, Tim Oliver Buchholz, and Florian Jug. Shise2void-Learning denoising from single noisy images. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June:2124-2133 2019. ISSN 10636919. doi: 10.1109/CVPR.2019, 00223.
Alexander Krull, Tomáš Vičar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic noise2void: Unsupervised content-aware denoising. Frontiers in Computer Science, 2:5, 2020.

---

Under review as a conference paper at ICLR 2023
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image denoising. Advances in Neural Information Processing Systems, 32, 2019.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miia Aittala, and Timo Aila. Noise2Noise: Learning Image Restoration without Clean Data. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2965-2974, PMLR, 2018
Huangxing Lin, Yihong Zhuang, Yue Huang, Xinghao Ding, Xiaoqing Liu, and Yizhou Yu. Noise2grad: Extract image noise to denoise. In Zhi-Hua Zhou (ed.), Proceedings of the Thirlieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 830-836. International Joint Conferences Artificial Intelligence Organization, 8 2021. Main Track.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statisties. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pp. 423..22..2222 IEEE, 2001.
Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. Noisier2noise: Learning to denoise from unpaired noisy data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12064-12072, 2020.
Tongyao Pang, Huan Zheng, Yuhui Quan, and Hui Ji. Recorrupteddoo-Rcooruppe Unsupervised Deep Learning for Image Denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2043-2052, 2021.
Yuhui Quan, Minggin Chen, Tongyao Pang. and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1887-1895, 2020. ISSN 10636919, doi: 10. 1 1109CCVP442000 202.000090.220
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Wei Wang, Fei Wen, Zeyu Yan, and Peilin Liu. Optimal transport for unsupervised denoising learn- ing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022a.
Zejin Wang, Jiazheng Liu, Guoqing Li, and Hua Han. Blind2unblind: Self-supervised image de- noising with visible blind spots. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2027-2036, 2022b.
Dufan Wu, Kuang Gong, Kyungsang Kim, Xiang Li, and Quanzheng Li. Consensus neural network for medical imaging denoising with only noisy training samples. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pp. 741-749. Springer, 2019.
Xiaohe Wu, Ming Liu, Yue Cao, Dongwei Ren, and Wangmeng Zuo. Unpaired learning of deep image denoising. In European conference on computer vision, pp. 352-368. Springer, 2020.
Roman Zeyde, Michael Blad, and Matan Protter. On single image scale-up using sparse- representations. In International conference on curves and surfaces, pp. 711-730. Springer, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-315) 2017.
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A poisson-gaussiat denoising dataset with real fluorescence microscopy images. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11710- 11718, 2019.
Zuo W Zhang K. and Zhang. FFDNet: Toward a fast and flexible solution for enn-based image denoising. IEEE Trans. Image Process., 4609)4608-44 2018.
10


Reviews:
[{"review_id": "wXvLV9sEv5U", "reviewer": "Reviewer_L537", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an unsupervised deep learning method for single image denoising, which only takes a set of noisy images for model training. The basic idea is combining the Recorrupted-to-Recorrupted loss with a sign flipping scheme and a pseudo supervised loss on self-generated samples. The experiments on both synthetic Gaussian noise corrupted images and the real-world SIDD dataset showed that the proposed method has certain improvement over the recent unsupervised methods such as Neighbor-to-Neighbor in some settings.", "summary_of_the_review": "This paper showed the combination of two existing techniques for unsupervised image denoising can lead to certain improvement. However, there are severe errors and many unclear points in the mathematical proof and statements. Such problems are difficult to solve in a revision.", "strength_and_weaknesses": "Strengths\n1. The proposed method showed improvement over unsupervised methods such as Neighbor-to-Neighbor on the real-world dataset SIDD, though the improvement is somehow marginal.\n2. The combination of Recorrupted-to-Recorrupted loss with the sign flipping scheme leads to the improvement.\n\nWeaknesses\n1. Novelty is quite limited as the work is the combination of two established works.\n2. There are severe errors in the mathematical proof and logic gaps in the explanation of the proposed method.\n3. The writing needs improvement. In addition to many grammar errors and non-standard language usage, the technical part is very unclear.\n\nSee the comments below for the details about the weaknesses.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nPXa-oDcQBK", "reviewer": "Reviewer_c6Mm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem of self-supervised image denoising and present an method which is an extension of the existing work R2R (Recorrupted2Recorrupted [Pang et al. CVPR'21).  The basic idea is to synthesize the pair from a noisy image that can simulates the supervised pair (noisy, clean) in supervised learning, where the noise is simulated by randomly flipping the sign of residual. The experiments are conduced in several datasets.", "summary_of_the_review": "The paper studied an interesting problem, and presents an extension of existing works for solving it. However, the extension is based on some  procedure, and there is severe errors in their theoretical argument. The performance gain is also limited. Thus,, I don't think this paper meets the standard of ICLR for acceptance.", "strength_and_weaknesses": "**Strength**:\n\n1. The paper is quite accessible for catching the main idea.\n \n**Weakness**:\n 1. The novelty of the work is limited, it is essentially an extension of R2R with un-justified noise simulation\n2. The key theoretical argument for discussing R2R is incorrect.  In the last sentence, the author claim as long as $n+n'$ inside $f$ and $n-n'$ outside out $f$ are independent, the R2R works. It is incrrect, the noise $n+n'$ and $n-n'$ cannot be independent. \n3. The key theoretical argument for discussing sign-flipping is wrong. Firstly, the paper claim that the simulated $\\hat n$ defined in  (9)  is independent from $n$. It is not true as there is $n$ in the definition of $\\hat n$. Secondly, the claim after (1) that $n-\\hat n$ and $n+\\hat n$ is independent, which is incorrect by the same reason in 2.\n4. The performance improvement over existing ones is quite marginal.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jeiWxyWpUgo", "reviewer": "Reviewer_CpLn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an unsupervised learning framework for denoising, without assuming noise priors. Three stages are designed, i.e., generating noise, psedo supervised learning and recorrupted2recorrupted. The results on synthetic Gaussian noise and real noise datasets show the proposed method is comparable with existing methods. ", "summary_of_the_review": "The proposed method is effective, but it is not well explained, and the effect of each component is not well verified. The writing needs more effort to polish. ", "strength_and_weaknesses": "Strength: A new unsupervised learning framework for image denoising. The noise prior is not assumed, and the proposed method may be applicable to wider noise types. \n\nWeaknesses:\n1. The proposed method is somewhat complicated and is not well explained why it can work. Let us assume the denoiser is perfect, i.e., y-f(y) is exactly the true noise. Then second step does nothing since f can perfectly remove \\hat{n}, and third step forces f can remove stronger noise since y+\\hat{n} shoulbe be close to y-\\hat{n} by L_u. So the main denoising effect may come from the third step. But the contriton of these steps are not validated in ablation study.\n\n2. What is regular loss Lg? And in Fig. 1, network f is w/o gradient and the lines are dashed on Lg. I cannot understand what they mean, and there is no explanations in main context. \n\n3. The English is not inapproriate, e.g., \"we will ... perfect denoiser\", \"PSEUDO SUPERVISED\" is not suitable for section title, \"the generate noise\". Just list a few.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "P7gGyfjlEO", "reviewer": "Reviewer_XJfe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an unsupervised method for image denoise. The noise model is generated by a residual image and a random mask. With the noise model, the input and target of the network are generated from a single noisy image. Experiments are conducted on both synthetic and real-world datasets.", "summary_of_the_review": "I think this paper is below the acceptance threshold. The novelty is limited as they just combine several existing ideas. ", "strength_and_weaknesses": "Strength:\n1. This paper is easy to follow and well-organized.\n2. The paper is abundant with experiments. The ablation study somewhat justifies the design choices of the method.\n\nWeaknesses:\n1. The main idea of this paper is similar to [1] and [2]. The proposed method seems like a combination of these two methods. I cannot find new technical or theoretical insights in this work.\n[1] Lin H, Zhuang Y, Zeng D, et al. Self-Verification in Image Denoising. arXiv preprint arXiv:2111.00666, 2021.\n[2] Pang T, Zheng H, Quan Y, et al. Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. CVPR 2021: 2043-2052.\n2. The performance improvement is slight according to Fig.2 and Tab.1.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject