An Unsupervised Deep Learning Approach for Real-World Image Denoising
Authors: Dihan Zheng
Sia Huat Tan
Xiaowen Zhang
Zuoqiang Shi
Kaisheng Ma
Chenglong Bao
Venue: ICLR 2021 Poster (2021)
Keywords: Real-world image denoising
unsupervised image denoising

Abstract: Designing an unsupervised image denoising approach in practical applications is a challenging task due to the complicated data acquisition process. In the real-world case, the noise distribution is so complex that the simplified additive white Gaussian (AWGN) assumption rarely holds, which significantly deteriorates the Gaussian denoisers' performance. To address this problem, we apply a deep neural network that maps the noisy image into a latent space in which the AWGN assumption holds, and thus any existing Gaussian denoiser is applicable. More specifically, the proposed neural network consists of the encoder-decoder structure and approximates the likelihood term in the Bayesian framework. Together with a Gaussian denoiser, the neural network can be trained with the input image itself and does not require any pre-training in other datasets. Extensive experiments on real-world noisy image datasets have shown that the combination of neural networks and Gaussian denoisers improves the performance of the original Gaussian denoisers by a large margin. In particular, the neural network+BM3D method significantly outperforms other unsupervised denoising approaches and is competitive with supervised networks such as DnCNN, FFDNet, and CBDNet.

Published as a conference paper at ICLR 2021
AN UNSUPERVISED DEEP LEARNING APPROACH FOR REAL- WORLD IMAGE DENOISING
Dihan Zheng', Sia Huat Tan?, Xiaowen Zhang², Zuoqiang Shi KKiihhng Ma², Chenglong Bao 1,5m
1 Yau Mathematical Sciences Center, Tsinghua University Instiittte for Interdisciplinary Information Science, Tsinghua University Hisilicon
*Department of Mathematheal Sciences, Tsinghua University
"Yanqi Lake Beijing Institute of Mathematical Sciences and Applications
(zhengdh1s 9, csf19) 9919)e emails ts tsinghua. edu. en, 2hangxxlaoveneehssllicc.. (zashi, kai sheng, c clbao) @ma i 1 tss nghua .eeaaaedd...om edu.cn cn
ABSTRACT
Designing an unsupervised image denoising approach in practical applications is a challenging task due to the complicated data acquisition process. In the real- world case, the noise distribution is so complex that the simplified additive white Gaussian (AWGN) assumption rarely holds, which significantly deteriorates the Gaussian denoisers' performance. To address this problem, we apply a deep neu- ral network that maps the noisy image into latent space in which the AWGN assumption holds, and thus any existing Gaussian denoiser is applicable. More specifically, the proposed neural network consists of the encoder-decoder struc- ture and approximates the likelihood term in the Bayesian framework. Together with Gaussian denoiser, the neural network can be trained with the input image itself and does not require any pre-training in other datasets. Extensive exper- iments on real-world noisy image datasets have shown that the combination of neural networks and Gaussian denoisers improves the performance of the original Gaussian denoisers by a large margin. In particular, the neural neMD method significantly outperforms other unsupervised denoising approaches and is competitive with supervised networks such as DrCNN, FFDNet, and CBDNet.
INTRODUCTION
Noise always exists during the process of image acquisition and its removing is important for image recovery and vision tasks, e.g., segmentation and recognition. Specifically, the noisy image y is modeled as y = x + n, where x denotes the clean image, n denotes the corrupted noise and image denoising aims recovering from y. Over the past two decades, this problem has been extensively explored and many works have been proposed.
Among these works, one typical kind of model assumes that the image is corrupted by additive white Gaussian noise (AWGN), i.e., n ~ (N o²I) where (0, 1) is the standard Gaussian distribution. Representative Gaussian denoising approaches include block matching and 3D fil-
tering (BM3D) (Dabov et al. 2007b). non-local mean method (NLM) (Buades et al.] 2005). K- SVD (Aharon et al. 2006) and weighted nuclear norm minimization (WNNM) (Gu et al. 2014). which perform well on AWGN noise removal. However, the AWGN assumption seldom holds in practical applications as the noise is accumulated during the whole imaging process. For example, in typical CCD CMOS cameras, the noise depends the underlying context (daytime night- time, static or dynamic, indoor or outdoor, etc.) and the camera settings (shutter speed, ISO, white balance, etc.). In Figure I two real noisy images captured by Samsung Galaxy S6 Edge and Google Pixel smartphones are chosen from Smartphone Image Denoising Dataset (SIDD) (Abdelhamed al. 2018) and three 40 40 patches are chosen for illustration of noisy distribution. is clear that real noise distribution is content dependent and noise in each patch has different statistical proper- ties which can be non- Go-Gasssian. Due to the violation of the AWGN assumption, the performance of
"Corresponding author.

---

Published as a conference paper at ICLR 2021
colorche
colorche
colorche
colorche
9.00
9.65
0.32
Cooacoo
(a)
(h)
(c)
(d)
(c)
Figure 1: Two real noisy images. (a) Clean images. (b) Noisy Images. (c) Noisy distribution in red, green and yellow patches. (d) BM3D results (PSNR: 26.55 (top) and 29.41 (bottom)). (e)
NN+BM3D (PSNR: 27.53 (top) and 30.05 (bottom)).
the Gaussian denoiser deteriorates significantly (Figure (d)). Thus, is crucial to characterize the noise distribution and adapt the noise models to the denoiser in real-world image denoising.
In recent years, deep learning based methods have achieved remarkable performance with careful ar- chitecture design, good training strategies, a large number of noisy and clean image pairs. However, there are two main drawbacks of these approaches from the perspective of practical applications. One is the high dependency on the quality and the size of the training dataset. Collecting such im- age pairs time-consuming and requires much of human efforts, especially when the labeling needs deep domain knowledge such medical or seismic images. The very recent deep learning methods including Noise2Noise (N2N) (Lchtinen et al., 2018). Noise2Void (N2V) (Krull et al., 2019a) and Noise2Self (N2S) (Batson & Royer 2019) have relaxed the dataset requirement and can be trained noisy and noisy image pairs. Nevertheless, to guarantee the performance, these networks need to be pre-traine pre-trained with a large number of images cover sufficiently many local patterns, and thus they are not cost-effective. Therefore, to reduce the dependency of the training set, single-image based image denoising approaches deserved be studied and have both practical and scientific value. It is worth mentioning that recent unsupervised learning work (Ulyanov et al.) 2018) uses a deep image prior to the general image recovery problem but its denoising results are inferior to some typical Gaussian denoisers, e.g., BM3D. The other drawback is the generalization ability of trained network. When the noisy distribution training set, the results of the deep learning method can be deteriorated significantly, even worse than non-learning based methods. To alleviate this problem, some recent works are proposed by further consideration of noise estimation in the network design, e.g.. Guo et al. (2019); Yue et al. 2019); Zhang et al. (017)). Despite their good performance in blind Gaussian denoising (Guo et al. 2019 Zhang et al. 2017) and real-world denoising problem (Yue al.) 2019), large number of noisy and clean image pairs are needed and the generalization problem remains when the imaging system is complicated. Very recently, single image based method has been proposed in (Quan) ctt al. 2020) by developing a novel dropout technique for image denoising. Thus, unsupervised deep learning approaches with accurate noise models are important for solving real-world image denois- ing problems, yet current solutions are unsatisfactory. Such approach deserves to be studied and is challenging problem as it needs good combination of traditional methods and deep learning based methods such that the benefits of both methods are fully explored.
on wwwpiiieddunoooranzziee
complicated and not contained in the
1.1 THE SUMMARY OF IDEAS AND CONTRIBUTIONS
Motivated by the above analysis, the goal of this paper is to propose an unsupervised deep learning method that boosts the performance of existing Gaussian denoisers when solving real-world image denoising problems. The basic idea is to find a latent image z associated with the input noisy image y such that z/x satisfies the AWGN assumption, and thus obtain the clean image
2

---

Published as a conference paper at ICLR 2021
Neural Network Training
Latent Space Denoising
Decoder
Gaussian Denoiser
Encoder
Noisy y
Latent
Clean x
Likelihood
Regularization
Figure 2: The workflow of our method. Middle arrows indicate the alternating optimization between network training and Gaussian denoising in latent space.
from by any existing Gaussian denoiser. To find the appropriate latent representation, propose a neural network (NN) based approach that builds up the mapping between the noisy image y and latent image with an encoder-decoder structure. By applying the Gaussian denoiser in the latent space, we alternatively update the NN and the denoised image which does not need the other training samples. Figure iillustrates the workflow of the proposed approach.
Also, this idea can be formulated under the classical maximum a posterior (MAP) framework which consists of likelihood term and prior term. Building a proper likelihood term requires accurate estimation of the noise distribution. Although the accurate noise distribution is difficult to get, an evidence lower bound (ELBO) can be analytically derived for approximating the likelihood from below using variational auto-encoder (VAE) (Kingma & Welling 2013). This ELBO term gives the loss function for the encoder and decoder networks that are maps between noisy image and latent image. From the above derivation, we arrive at a model
min f(x, F, F,9) + R(x), x
(1)
where is the clean image, y is the input noisy image, FF G are decoder and encoder maps param- eterized by NNs, f the loss from ELBO and R(x) is the regularization term. Model ( can be minimized by the alternative direction method of multiplier (ADMM) which alternatively updates networks and the clean image estimation x. Using Plug-and-Play technique (Venkatakrishnan et al.) 2013), updating x can be replaced by any Gaussian denoiser. Thus, by fully exploiting the benefits of deep neural networks and classic denoising schemes, the real-world image denoising can be im- proved by large margin as shown Feg More importantly, training the proposed networks only uses the noisy image itself and does not need any pre-training. In summary, we list our main contributions as follows.
We propose effective approach that combines the deep learning method with traditional methods for unsupervised image denoising. Thanks to the great expressive power of deep neural networks, the complex noise distribution is mapped into a latent space which the AWGN assumption tends to hold, and thus better results are obtained by applying existing Gaussian denoiser for latent images.
Instead of heuristic loss design, the proposed NN approximates the likelihood in the clas- sie Bayesian framework, which gives clear interpretations of each loss term. Meanwhile, compared to many existing deep learning methods, our model is only trained on a single image, which significantly reduces the burden of data collection.
Extensive numerical experiments on real-world noisy image datasets have shown that the NN boosts the performance of the existing denoisers including NLM, BM3D and DnCNN. In particular, the results of NN+BM3D are competitive with some supervised deep learning approaches such DrCNN+, FFDNet+, CBDNet.
3

---

Published as a conference paper at ICLR 2021
2
RELATED WORK
There are numerous works in image demoising. Here, we review the works related to the non- learning and learning approaches for real-world image denoising.
2.1 NON-LEARNING BASED APPROACHES
Non-learning approaches are mainly based on the MAP framework that contains a data fidelity and a regularization term. Many works have been proposed to improve the regularize term, e.g., spar- sity based methods (Rudin et al. 1992 Perona & Malik 1990), low rank prior (Dong et al. 2012b Gu ct al. 2014) and non-local methods (Buades et al. 2005 Dabov et al. 2007a). Among these methods, BM3D (Dabov et al. 2007a) is one of the top methods. There are several other works related to the construction of data fidelity by modeling the complex noise distribution, e.g., Lebrun et al. (2015a); Nam et al.] (2 Xu al. (2017): Zhu et al. (2016). The correlated Gaussian dis- tribution (Lebrun et al.) 2015a) and Mixture of Gaussian (Zhu et al.) [0016] Nam et al.) 2016) are used approximate the unknown noise distribution. In Xu al. (2017), different noise statistics are estimated in different channels without the consideration of the content dependent noise. Re- cently, (mmin al., (2020) proposed Gaussianization method for gray scale OCT images. However, this method is not applicable for real-world image denoising tasks as natural images are colorful and the noise distribution much more complicated than that OCT images. Overall, due to the com- plexity of real-world noise, the performance of these approaches is unsatisfactory and needs to be improved.
2.2 LEARNING BASED APPROACIIES
The learning based approaches can be classified into two groups: single-image based methods and dataset based methods. Typical aingle-image based approaches are sparse coding methods AAharon et al. (200)); Bao et al. (2015): Xu et al. (2018b). In Xu et al. (2018b), the noise in each channel is estimated and followed by a weighted sparse coding scheme. recent years, as the appearance of real image denoising datasets including CC (Nam al. 2016)), PolyU (Xu et al. (018a)) DND (Plotz & Roth (017) and SIDD (Abdelhamed et al. 2018). deep neural networks including Guo et al. (2019): Yu et al. (2019): Zhang et al.] (2017] Zhou al. (2019): Yue al. (2019) have shown promising results on these datasets. However, these networks require many noisy/clean training pairs which limit their practical applications especially when the labeling work needs domain experts.
Recently, the deep learning approaches (Krull et al.] [ Batson & Royer 2019 Laine et al.) 2019 Lehtinen al. 2018) are proposed and trained with organized or unorganized noisy image pairs. To guarantee a satisfactory performance, these methods still need many training pairs such that sufficiently many local patterns are covered. Compared the above deep learning approaches, our method is single image based method which does not need training samples or pre-training from other datasets.
3 OUR METHODOLOGY
This section starts with the derivation of model and is followed by the detailed optimization techniques that incorporate any existing Gaussian denoiser.
3.1 THE MODEL FORMULATION
Let R.V be the noisy image where N Height x Width 3. The classic MAP framework aims at maximizing the posterior distribution p(xly) formulated as the following optimization problem:
max In p(xly) max In p(y(x) + In p(x) = max ( XR(x).
The term p(x) is the prior which represents the internal statistics of natural images. One common choice is p(x) cxp(-XR(x)) where R(x) is regularization function. The term p(y[x) is the likelihood that models the uncertainty of the observed image, i.c., y x ~ p(n) where p(n) denotes the noise distribution. In practice, the noise distribution is complex and it is only possible find an
4

---

Published as a conference paper at ICLR 2021
approximation. Following the VAE approach (Kingma & Welling 2013), the likelihood p(y|x) has a lower bound, i.c.,
Inply x) = In [ f( p(yxz in g(zly) ply, z/xx
g(Aly) dz 2 / g(z|y) In p(y, = Eg(nty) In puyp(ye 2, x) - (L(((Z)
g(zly) dz
= ELBO,
where is the latent image, q is the distribution of the latent image conditioned the noisy image y, KL denotes the Kullback-Leibler (KL) divergence and the ELBO stands for the evidence lower bound. In practice, motivates the usage of NNs.
construct tractable ELBO with high expressive power, which
The proposed NN consists of an encoder net G and decoder net F that construct the mappings between the noisy image and the latent image z. Suppose the latent image z is the Gaussian corruption of the clean image with strength a, i.e., z/x ~ NNx,, T²I) where N(0, I) is the standard Gaussian distribution. By choosing p(y|z, x) and q(ll)) properly, the next proposition gives a closed form of the ELBO.
Proposition 1 Suppose z|x ~ NN(x 12). Choosing y/z, ~ M(F(z), I) and g(zly) = (G where F,9 are decoder and encoder respectively, the ELBO equal
Edl(F(g(y) + yl1 20 (19(y) - x1| + t
where (0, and a constant depends on the image size.
The proof is given in appendix A.1. Let FF and G be parameterized by 01 and 02 respectively, we replace the likelihood in (2) by which derives our model:
F((x, 01, 02) F.c Fo (Go +) - -y12 + 20 11 (y) x|12 XR(x).
(4)
There is an expectation term in (4), which can be estimated by the Monte Carlo method. Following the standard sampling approach in VAE (Kingma & Welling 2013). we resample every time before backpropagation in our experiments.
Remark In our model, the basic assumption is that the latent image z a white Gaussian per- turbations of clean image x, i.e. z|x oxx,x Since Go, (y), the second term in (4) can be seen as the transformed data fidelity. Moreover, the combination of those three terms can prevent the encoder from degenerating into zero mapping or identity mapping.
3.2 MODEL OPTIMIZATION
Since the loss function 4 is non-convex and the regularization R(x) can be complicated, we intro- duce an auxiliary variable p and apply the ADMM scheme for solving 44. Concretely, define
1
f(x, 02) E,liFo, (So, (y) + y/1² + 202 Gn, (y) xx|
the loss function is rewritten as
min f(x, &(pp), s.t. 11.p-x-1 x,0,
(5)
The augmented I. Lagrangian function
(
Lp(x, 01, p) = f(x, 0, 02) R(p) ( - - -lla/p1
where q is the dual variable and p 0 is chosen constant. iterates:
ADMM consists of the following
(x gk+1 g arg min L,(x, 01, Oz. pt g*))
x,0,,0,
p k+1 arg min C, k+1 gh+1 10. ,pq.))
P h+1 q" + -
5

---

Published as a conference paper at ICLR 2021
Subproblem 66) is solved by alternating minimization, i.e., update the clean image estimation and the network parameters 0r and 22 alternatively. More specifically, the networks and g are updated by back propagation method with fixed x. Then, 01 and 02 are fixed, is updated by solving the minimization:
min 20 x Go. (1) + 2 x p p --p/l//p1-
(8)
The least square problem
has a closed form solution. Besides, subproblem is equivalent to min R(p) + 111p-
9" /pl12
Then p PrOXPR (x* k+1 pp) where the proximal mapping is defined ProxpR (z) x|2. Motivated by this observation, the Plug-and-Pias method (Venkatakr- 2013) is proposed by generalizing this proximal mapping to any existing denoising T(x*+1 /²)p). In method, we choose the T as any existing Gaus- sian denoiser, e.g., NLM (Buades et al. 2005), BM3D (Dabov et al. 2007a). Overall, the detailed algorithm is given in Algorithm [ where X denotes the Gaussian denoiser.
arg ming R(z) +
ishnan et al.
scheme, i.e. p ++
We initialized randomly for network parameters 0 and 02 without the usage of a pre- trained model. In method, p is initialized y, and q is initialized to 0 that is linear combination of y and the latent space image at the early stage of this algorithm. Therefore, x is not far from y in the beginning, which will lead correct convergence for our algorithm.
Remark
Algorithm The Denoising Algorithm NN+X.
Input:
Noisy image y. p. o, r7: Output: Denoised image Initial x = y. pd = y. qu = 0, and network parameters 0,, 0z.
for k 0, 1,2,3, .... M do
for i 0, 2, 3 .... m do Sample the e from standard Gaussian distribution in 66).
Update of. 0f by using backpropagation (BP) algorithm for 6
Update k.i+1 [ (otn+s+) 90t (y) + pp* q²) (p ))
end for Set x k.m 0p+1 0hmmm 05+1
+1 = 0p.m.
Update p k+1 by Gaussian Denoiser algorithm X x + qt/ /p.
Update q f+11 = q" np(xx+1 )l))
end for return x N
4 EXPERIMENTS
We evaluate the performance of our method on real noisy images in this section. All experiments are evaluated in the sRGB space.
4.1 IMPLEMENTATION DETAILS
Both encoder network G and decoder network F are chosen as two standard 10 layers U-Nets (Ron-) neberger al., 2015) implemented in Pytorch using Nvidia 1080TI or Nvidia 2080TI GPUs. ADAM algorithm (Kingma& & Ba 2014) is adopted to optimize the network parameters and the learning rate set as 0.01. The number of epoch network training set as 500 and the parameters p, or and vj in ADMM are set as 1, and 0.5 respectively. The noise level is estimated from Donoho & Johnstone (194)); Chen et al. (2015a). For an image with size 512 x 512 x 3, our method needs about 15 minutes with single Nvidia 2080TI GPU.
4.2 EXPERIMENTS ON REAL-WORKL NOISE
We combine the NN with three existing Gaussian denoiser methods, including two traditional methods (Non-local Mean (NLM) (Buades al. 2005) and Block Matching and 3D Filtering

---

Published as a conference paper at ICLR 2021
Table 1: Averaged PSNR and SSIM CC, PolyU and FMDD.
CC
PolyU
FMDD
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
NLM
33.47
0.8463
35.80
0.9063
30.81
0.8051
VST+NLM
36.05
0.9285
37.56
0.9545
31.34
0.7633
NN+NLM
37.31
0.9522
37.87
0.9606
31.96
0.8213
BM3D
35.19
0.8580
37.40
0.9526
29.70
0.7516
VST+BM3D
36.44
0.9396
38.15
0.9638
32.71
0.7922
NN+BM3D
38.26
0.9606
38.74
0.9681
33.91
0.8872
DrCNN
33,86
0.8635
36.08
0.9161
30.64
0.7514
VST+DnCNN
33.20
0.8452
35.66
0.9066
32.19
0.7728
NN+DnCNN
35.02
0.9069
36.67
0.9435
32.41
0.7798
Noisy (29.86/0.78) 86/0.8874)
NLM (29.92/), 79.02)
BM3D (3..040..725))
DrCNN (0.1770.7967)
Clean
NN+NLM (33.491088979)
NN+BM3D (34.33/0.9190)
NN+DuCNN (1.0220.8356
Figure 3: Visual results and PNSR/SSIM of image from CC.
2007b)) and one pre-trained deep learning method (DnCNN) Zhang et al. 2017 from its official project website. We choose two nature real-world noisy image datasets named CC (Nam et al. 2016). PolyU (Xu al.) 2018a), and one real fluorescence microscopy dataset named FMDD (Zhang et al. 2019) for testing the performance of our method in terms of PSNR and SSIM. Besides, the Variance Stabilizing Transform (VST) method (Makitalo & Foi 2012). traditional noisy transformation method, is chosen for comparison. We employ the noise estimation method (Foi et al. 2008) estimate the Poisson-Gaussiar noise parameters for the VST method in FMDD. In CC and PolyU, we evaluate different Poisson noise parameters (peak value, range from 10 10000) and select the best for whole datasets. There are 15 and 100 images in CC (Nam al,, 2016) and PolyU (Xu et al.) (0188) datasets with the same cropped regions in their original papers. FMDD, we evaluate
(BM3D) Dabov et al.
performance on the mixed test set with raw images which the same setting as in Zhang et al. (2019).
Denoising results. The denoising results are evaluated using PSNR and SSIM, with built-in func- tions in Python skimage package. The results are listed in Table which we have the following observations: (a) Our model improves the performance margin on both two datasets. On average, the PSNR/SSIM values increased by 2.35/0.0588. 2.87/0.0844 and 1. 1.133/0.0331 for NLM, BM3D and DrCNN respectively, see Figure for one visual example from CC. (b) Comparing with the VST method, the existing Gaussian Denoisers benefit more from the proposed method.
both three Gaussian denoiser by a large
DISCUSSION
Validation of AWGN assumption of latent images. Let x, z be clean, noisy and latent images. are the noise in image space and latent space respectively.
Denote n1 = y and n, We visualize the distribution of n, and n, in Figure 4 using two images in Figure] [ is obvious that the noise distribution in latent space is more similar white Gaussian than image space.

---

Published as a conference paper at ICLR 2021
CC3
CHEB 3eT
colorche
0003
0023
6.05
50
25
544
6412
0000
680
100
223
40
6
(b)
(d)
Figure 4: (a) Noisy images with three selected patches. (b) Noisy distribution in image space. (c) Noisy distribution after VST transformation. (d-e) Noisy distribution in latent space by NN+NLM and NN+BM3D respectively.
Table 2: Comparison with superviieed/nssupeviisee methods CC.
Supervised methods
Unsupervised methods
Metric
NN+BM3D
DnCNN+
FFDNet+
CBDNet
N2N
NAC
DIP
PSNR
35.40
37.63
36.44
35.32
36.59
37.72
38.26
SSIM
0,9115
0.9555
0,9460
0,9160
0.9502
0.9531
0,9606
Moreover, define a; = Mcan(n,) and o? = Var(n;) for i = 1,2 where Mcan, Var are the mean and variance operators respectively. To quantify the distance of between n,, í = 1, and white Gaussian distribution, we calculate the KL divergence between In image space, the KL divergence 0.4701, while in latent space the KL divergence reduces to 0.3821, 0.2868 and 0.3636 for NN+NLM. NN+BM3D and NN+DnCNN respectively. We note the noise distribution in latent space is closer to the Gaussian distribution than its in image space.
and ((a, o.1) on CC dataset.
Comparison with cluding DrCNN+ et al., 2017). FFDNet+ (Zhang et al., 2018). CBDNet (Guo et al. 2019) and three unsupervised approaches including Noise2Noise (N2N) (Lehtinen al. 2018), Deep Im- age Prior (DIP*) (Ulyanov et al. 2018) (3000 iterations for 5 times average) and Noise-As-Clean (NAC) (Xu et al. (019) chosen for comparison
methods. Three blind image denoising networks in-
CC datasets. FFDNet+ is a multi-scale exten- sion of FFDNet (Zhang et al. 2018) and DrCNN+ is a color version of DnCNN and fine tuned with the FFDNet+ results The denoising results are listed in Table 2 shown that NN+BM3D is better than deep neural networks trained on a dataset and other unsupervised deep learning methods.
The capacity of decoder. The proposed model 44) has a trivial global minimum when assuming (a) the decoder is constant mapping, i.e. F(z) y for all z; (b) G(y) x 0. We test the capacity of the decoder used in our model by fitting the map between the constant image and the noisy image, and the training loss versus iteration is reported in Figure training loss on PolyU dataset 294.56, which shows that our decoder not large enough and the trivial minimum does not exist,
5 (a). Moreover, the average
explanation in appendix A.2.
Training stability. Four classic images including Kodim03 (red), Kodim02 (green), Lena (blue) and Peppers (yellow) with noise level 30 are used for testing the training stability of our method. The PSNR value versus iteration number is reported in Figure [5] (b) and shows that the PSNR keeps increasing and is stable after certain number of iterations.
Latent space evolution. One image in the introduction from SIDD dataset used to show the latent image evolution in our method. In particular, we evaluate NN+BM3D for 20 iterations and show
'The results of DnCNN+, FFDNet+ and CBDNet are from Hou al. 2019 and the results of N2N and NAC are from (Xu et al. 2019)
8

---

Published as a conference paper at ICLR 2021
35.4
18. 164
18
20'
16
100
08 62
0.2
Baratoo
Scation
(d)
(c)
Figure 5: (a) is the training losses of our decoder using constant input for five images from PolyU. (b) show PSNR value versus the number of iterations in NN+BM3D. (c), (d) and (e) are results for different hyperparameters p. a and r) respectively.
colorche
colorche
colorche
colorcha
colorche
colorche
colorche
colorche
Iteration 2,
Iteration 17, 18, 19, 20
Figure 6: Evaluation of the latent representation Algorithm with NN+BM3D.
8 latent images and their patch noise distribution evolution in Figure 6 It is shown that the noise distribution gradually changes from non-Gaussian to Gaussian.
Hyperparameter sensitivity. The sensitivity of hyperparameters are given in Figure (c), (d) and (e), where we change one p. a and yj each time and fix the others as p a and v = 0.5. can be observed that our method is not sensitive to different hyperparameters.
6 CONCLUSION
In this paper, we propose a NN based method that maps the complex noisy distribution in real-world images into latent space in which the AWGN assumption holds. Combined with any existing Gaus- sian denoising approaches, it improves the denoising results by large margin. More importantly, this method does not require any training sample except the input noisy image itself. Extensive results validate the rationale of the proposed network training scheme and show the advantages of our method compared with existing representative approaches including learning and non-learning based methods.
ACKNOWLEDGEMENTS
This work is supported by the National Natural Science Foundation of China (1 (11901338, 12071244), Technology and Innovation Major Project of the Ministry of Science and Technology of China un- der Grant 2020AAA0108400 and 2020AAA0108403 and Tsinghua University Initiative Scientific Research Program.
REFERENCES
AbdeIrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In CVPR, pp. 1692-1700, 2018.
Neatlab ABSoft. Neat image, 2017.
Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing over- complete dictionaries for sparse representation. JEEE Trans. Image Process., 54(11):411-4333 2006.

---

Published as a conference paper at ICLR 2021
Zahra Amini, Hossein Rabbani, and Ivan Selesnick. Sparse domain gaussianization for varii-v statistical modeling of retinal oct images. IEEE Trans. Image Process., 2020.
Chenglong Bao, Hui Ji, Yuhui Quan, and Zuowei Shen. Dictionary learning for sparse coding: Algorithms and convergence analysis. IEEE Trans. Pattern Anal. Mach. Intell., 336))1336-3369,, 2015.
Joshua Batson and Loic Royer. Noise2self: Blind denoising by self-supervision. arXiv:1901.110 190.::990..1655, 2019.
Antoni Buades, Bartomeu Coll, and J-M Morel. non-local algorithm for image denoising. In CVPR, volume 2, pp. 60-65. IEEE, 2005.
Guangyong Chen, Fengyuan Zhu, and Pheng Ann Heng. An efficient statistical method for image noise level estimation. In ICCV, pp. 477-485, 2015a.
Yunjin Chen, Wei Yu, and Thomas Pock. On learning optimized reaction diffusion processes for effective image restoration. In CVPR, pp. 5261-5269, 2015b.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Color image denoising via sparse 3d collaborative filtering with grouping constraint in luminance-chrominance space. In ICIP, volume 1, pp. I-313. IEEE, 2007a.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Trans. Image Process., 16(8):2080- 2095, 2007b.
sheng Dong, Lei Zhang, Guangming Shi, and Xin Li. Nonlocally centralized sparse representation for image restoration. IEEE Trans. Image Process., 22(4):1620-1630) 2012a.
Weisheng Dong, Guangming Shi, and Xin Li. Nonlocal image restoration with bilateral variance estimation: a low-rank approach. IEEE Trans. Image Process., 22(2):700-711, 2012b.
David L Donoho and Jain M Johnstone. Ideal spatial adaptation by wavelet shrinkage. biometrika, 81(3):225-455, 1994.
Alessandro Foi. Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian. Practical poissonian- gaussian noise modeling and fitting for single-image raw-data. IEEE Trans. Image Process., 17 (10):1737-1754, 2008
Shuhang Gu, Lei Zhang, Wangmeng Zuo, and Xiangchu Feng. Weighted nuclear norm minimization with application to image denoising. In CVPR, pp. 2862-2869, 2014.
Shi Guo, Zifei Yan, Kai Zhang, Wangmeng Zuo. and Lei Zhang. Toward convolutional blind de- noising of real photographs. In CVPR, pp. 1712-1722, 2019.
Yingkun Hou, Jun Xu, Mingxia Liu, Guanghai Liu, Li Liu, Fan Zhu, and Ling Shao. Nih: a blind lixel-level non-local method for real-world image denoising. 1rXiv:1906.08834 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
Alexander Krull, noisy images. In CVPR, pp. 2129-21111 2137, 2019a.
Buchholz, and Florian Jug. Noise2void-leariing denoising from single
Alexander Krull, Tomas Vicar, and Florian Jug. Probabilistic noise2void: Unsupervised content- aware denoising, arXiv:190600065) 2019b..
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-qualit self-supervised deep image denoising. In NeurlPS, pp. 6968-6978, 2019.
Mare Lebrun, Miguel Colom, and Jean Mrca- Morel. Multiscale image blind denoising. JEEE Trans. Image Process., 24(10):119-3111 2015a.
10

---

Published as a conference paper at ICLR 2021
Mare Lebrun, Miguel Colom, and algorithm. Image Processing On Line, 5:1-54, 2015b.
Morel. The noise clinic: blind image denoising
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. arXiv: 180300880 2018.
Florian Luisier, Thierry Blu, and Michael Unser. Image demoising in mixed poisson-gaussian noise. IEEE Transactions on image processing, 20(3):69--708, 2010.
Markku Makitalo and Alessandro Foi. Optimal inversion of the generalized anscombe transforma- tion for poisson-gauus gassssian noise. IEFE Trans. Image Process., 22(1): 2012.
Seonghyeon Nam, Youngbae Hwang, Yasuyuki Matsushita, and Seon Joo Kim. holistic approach to cross-channel image noise modeling and its application to image denoising. In CVPR, pp- 1683-1691, 2016.
Pietro Perona and Jitendra Malik. Scale- space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell., 12(7):629-633 639, 1990.
Tobias Plotz and Stefan Roth. Benchmarking denoising algorithms with real photographs. In CVPR, pp. 1586-1595, 2017.
Yuhui Quan, Minggin Chen, Tongyao Pang, and Hui Ji. Self2self with dropout: Learning self- supervised denoising from single image. In CVPR, pp. 1890-1898, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, pp. 234-241 Springer, 2015.
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-266 1992.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In CVPR, pp. 9446- 9454, 2018
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-play priors for model based reconstruction. In GlobalSIP, pp. 945-948. IEFE, 2013.
Jun Xu, Lei Zhang, David Zhang, and Xiangchu Feng. Multi-channel weighted nuclear norm mini- mization for real color image denoising. In ICCV, pp. 1096-1104, 2017.
Jun Xu, Hui Li, Zhetong Liang, David Zhang, and Lei Zhang. Real-world noisy image denoising: 1804..0603. 2018a.
A new benchmark.
Jun Xu, Lei Zhang, and David Zhang. A trilateral weighted sparse coding scheme for real-world image denoising. In ECCV, pp. 20-36, 2018b.
Jun Xu, Yuan Huang, Li Liu, Fan Zhu, Xingsong Hou, and Ling Shao. Noisy-as-clean learning unsupervised denoising from the corrupted image. arXiv:1906.06878, 068788 2019.
Ke Yu, Xintao Wang, Chao Dong, Xiaoou Tang, and Chen Change Loy. Path-restore: Learning network path selection for image restoration. arXiv: 1904. 10343, 2019.
Zongsheng Yue, Hongwei Yong, Qian Zhao, Deyu Meng, and Lei Zhang. Variational denoising network: Toward blind noise modeling and removal. In NeurIPS, pp. 1688-1699, 2019.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning of deep enn for image denoising. IEEE Trans. Image Process., 26(7):3142 3155, 2017.
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: Toward a fast and flexible solution for cnn- based image denoising. IEEE Trans. Image Process., 270)):6608-4622, 2018.
Yide Zhang, Yinhao Zhu, Evan Nichols, Qingfei Wang, Siyuan Zhang, Cody Smith, and Scott Howard. A CVPR, pp. 11710-11718, 2019.
denoising dataset with real fluorescence microscopy images. In
11

---

Published as a conference paper at ICLR 2021
Yuqian Zhou, Jianbo Jiao, Haibin Huang, Yang Wang, Jue Wang, Honghui Shi, and Thomas Huang. When awgn-based denoiser meets real noises. arXhv:1904.03485, 2019.
Fengyuan Zhu, Guangyong Chen, and Pheng-Ann Heng. From noise modeling to blind image denoising. In CVPR, pp. 420-429, 2016.
A PROOF OF PROPOSITIONS 1
We show the derivation of Proposition and give one theoretical explanation of the capacity of decoder.
A.1 PROPOSITION
The ELBO in Proposition 1 is defined as Ey(sly inp(y/z,x) KL(g(zly))l((zx)) Since z/x ~ N(x, o" ~ (2)(2), I) and g(xly) = (y 1). Assume the image size of x is mxxx1 for gray scale image, then the KL divergence of two Gaussian distribution is
([(((x)))ll((xx)) Eo(z)) In g In p(zix)
In o² Bu(elr))
1
(-1lz- -x11=
2
mn
In o
mn
+
1
Eq(alyilzz - x112
2
2
202
mn
0²
mn
+
Eg(alyil1 -- G(y) + ((y) - xx1
2
202
1
In
o
+
(mn + -1(9()) x11²)
20²
mn
In
1
+ g2
[[((y) x112.
2
Using the reparameterization in (Kingma & Welling 2013). zly g(y) + where ~ N/ I), the expectation term is
Eg(ely) Inp(y|z,x) x) mn
In 2T
1 y112
2
mn
In 2m
1
Ee EFl(F(g(()) + y11
2
2
Then the ELBO Piy(aly) In 1(ply2,, - (T((gzly))lp((H) is
F. F.( + - 21 20
1
1 x||
mn
In 2T + In In g2
1
1
and for RGB images the constant term becomes holds.
3mn
(In 2 + In -1+ at) thus Proposition
2
A.2 THE CAPACITY OF DECODER
Here we give an explanation regarding to capacity of decoder in the following proposition.
Proposition 2 Suppose x E mmxxxx is a constant image, i.e.
Xijjk Kr Vi,i' E (1,2, ..., m),j.j' (1,2.... n). (1,2,...c.,
and F. is a deep neural network that composted with convolution layers (with reflect/replicate padding), down-sampling layers (with maxlaverage pooling). and up-sampling layer (with bilin- ear/nearest point interpolation) then F(x) is constant.
To simplify notations, we denote convolution layers as C, down- dlwn--sapling layers as D. and up- sampling layer as U, then only need to show for constant input Rmxxxe C(x). D(x) and
12

---

Published as a conference paper at ICLR 2021
(x) are constants. Assume ((z) x m'''n''xe need to show x.J.k , From the
dimension of X, there are d' convolution kernels denote as Kt E Rpixxixc 1,2 ....... c', then +22.3
xij,k=
Σ
K Nan x P- 11 +
Σ h,ldd
Kc dadds - +h.r" 2/" +12ds
so C(x) is constant. For down-sampling layer, since ((x) Max R} for max pooling and ( शरे DiJeR Xi,j. for average pooling, where R denote support of pooling kernel. Since x is constant, then clear from above expression that P(x) constant. Finally, for up-sampling layer, since we assume bilinear/nearest point interpolation, then (x)),,,. depends on the convex combination of {x.,, E S}. where denote the corresponding up-sampling region of U U So if x is constant then the convex combination of S is constant, then we have (x) is constant. Thus proposition holds.
The above proposition shows that the latent image can not be constant image and thus the regular- ization takes effects. Besides, the minimization problem is a non-convex problem which depends on the initialization. In experiments, initialize to be the input noisy image y and it empirically converges to a reasonable good denoised image.
Table Average PSNR/SSIM of denoised results on Set9 and BSD 68.
Datasets
Methods
= 25 0 = 50
σ = 75
O = 100
NLM
25.40/0.6281
21.27/0.4120
19.39/0.3092
18. 18.52/0.2597
NN+NLM
27.19/0.7493
23.6870.5663
21.48/0.4734
19.65/0.4329
BSD68
BM3D
28.33/0.800 8029
24.86/0.6699
22.63/0.5808
20.77/0.5167
NN+BM3D
28.32/0.8040
24.91/0.6776
22.75/0.5896
20.90/0.5241
NLM
30.11/0.7876
25.98/0.6615
22.59/0.5620
20.35/0. 4930
NN+NLM
30.27/0.7930
26.29/0.6761
22.71/0.5728
20.4570.4959
Set9
BM3D
31.47/0.8406
27.85/0.7551
24.64/0.6834
$1.96/0.611
NN+BM3D
31.48/0.8415
27.98/0.7618
24.79/0.6937
22.06/0.6285
B EXPERIMENTS ON SYNTHETIC NOISE
Additive white Gaussian noise. Even though our method is designed for the real-world image denoising, we evaluate the performance of this method on synthetic noises. We test two different Gaussian denoising methods, NLM and BM3D. Two datasets are used, one is the Set (Ulyanov) et al. 2018) with 9 classic color images and the other BSD 68 (Krull et al. 2019a) with 68 gray- scale images. We test four noisy levels: 25, 50, 75, 100, and all noisy images
quantized into 8-bits for simulating the common JPG images. The results are given in Table 3 we find even though the noise distribution is Gaussian, two denoising methods benefit from our approach more or
less.
Poisson noise. For noise distribution which is significantly different from Gaussian, we test the synthetic Poisson noise on CBSD 68 dataset with 68 color images by combining our method with VST transformation. The results are given in Table 4 all noisy images are quantized into 8-bits. We find that even though VST is a reliable transformation for the pure Poisson noise case, the neural networks are still helpful for improving the VST method. For DrCNN, we find sensitive to Poisson noise, for Peak value equals to 5, the result of VST+DrCNN is 17.23/0.3767 and our method (NN+VST+DRCNN) improves 0.60/0.0629 for on average.
Table 4: Average PSNR/SSIM of Poisson denoising results CBSD 68.
Peak Value
5
20
23.85/0.6792 NN+VST+BM3D 24.07/0.6951
VST+BM3D
25.66/0.7490 24.86/0.7201 25.82/0.7610
27. 27.99/0.8252 28.01/0.8313
25.06/0.7340
13

---

Published as a conference paper at ICLR 2021
Table 5: Averaged PSNR/SSIM CC, PolyU, DND, SIDD.
Datasets
Methods
WNNM
NCSR
TNRD
DnCNN
BM3D
35.77/0.9388 9381
33.46/0.8591
36.61/0.9463
23.86/0.8635
35.19/0.8580
CC
MCWNNM
TWSC
NC
NI
NN+BM3D
37.71/0.9552
37.81/0.9586
36.43/0.9364
35.49/0.9126
38.26/0.9606
WNNM
NCSR
TNRD
DnCNN
BM3D
36.59/0.9247
36.40/0.9290
38.17/0.9640
36.08/0.9161
37.40/0.9526
PolyU
MCWNNM
TWSC
NC
NI
NN+BM3D
38.51/0.9671
38.60/0.9683
36.92/0.9445
37.77/0.9570
38.74/0.9681
WNNM
NCSR
TNRD
DnCNN
BM3D
34.67/0.866
34.05/0.8351
33.65/0.8306
32.4296/0.79
34.51/0.855
DND
MCWNNM
TWSC
NC
NI
NN+BM3D
97.379/0.9294
37.96/0.9416
35.434/0.8844
35.1 125/0.8778
37.10/0.9441
WNNM
NCSR
TNRD
DrCNN
BM3D
25.78/0.88 8899
24.73/0.645
23.66/0.583
25.65/0. 685
SIDD
MCWNNM
TWSC
NC
NI
NN+BM3D
33.18/0.895
C EXPERIMENTS ON REAL WORLD NOISE
In this section, compare the performance of NN+BM3D with other denoising method on four real-world noisy image datasets named CC (Nam et al. 2016), PolyU (Xu et al. 201 8a), DND (Plotz & Roth 2017) and SIDD (Abdelhamed et al.J 2018). we make a comprehensive comparison study of method with many existing representative image denoising methods, in- cluding Weighted Nuclear Norm Minimization (WNNM) (Gu et al. 2014). Nonlocally Central- ized Sparse Representation (NCSR) (Dong et al. 2012a), Trainable Nonlinear Reactive Diffusion (TNRD) (Chen et al. 20155). DrCNN (Zhang et al. 2017). Multi-channel Weighted Nuclear Norm Minimization (MCWNNM) (Xu et al. 2017). Trilateral Weighted Sparse Coding (TWSC) (Xu al.) 2018b), "Noise Clinic" (NC) method (Lebrun et al. 2015a) and a commercial software Neat Image (NI) (ABSofi 2017). See Table S for the averaged PSNR/SSIM results. The results of comparison methods in CC and PolyU datasets are from (Xu et al. 2018a) and their results of DND and SIDD datasets are from their official project website.
Table 6: Comparison with supervised methods.
Datasets
DrCNN+
FFDNet+
CBDNet
NN+BM3D
DND
37.90/0.943
37.61/0.9415
08.05/0.9421
37.10/0.9441
SIDD
33.28/0. 868
33.10/0.895
Table 7: Comparison with superviieednnssupervised methods on FMDD.
Unsupervised Methods
Supervised Method
Metric
VST+BM3D
PURE-LET TWSC NN+BM3D
DnCNN
N2N
PSNR
32.71
31.95
31.64
33.91
34.88
35.40
SSIM
0.7922
0.7664
0.7787
0.8872
0.9063
0.9187
In addition, we compare the performance of NN+BM3D with Three top blind image denoising networks including DrCNN+
al., (017), FFDNet+ (Zhang et al. 2018) and CBDNet (Guo al. 2019). According to the results in DND and SIDD benchmarks, the denoising results are listed in Table 6 The symbol """ is used when the result the corresponding method is not reported.
We also compare the performance of NN+BM3D with superviseddunssupervieed methods in FMDD dataset (Zhang et al.] 2019). We compared with three unsupervised methods named as VST+BM3D (Makitalo & Foi 2012). PURE-LET (Luisier et al.] 2010) and TWSC (Xu et al.
2018b), two supervised methods DnCNN (Zhang et al. (017)) and N2N (Lehtinen et al.) (2018)
which are retrained the FMDD training set. The denoising results are listed in Table Although
14

---

Published as a conference paper at ICLR 2021
Table 8: Averaged PSNR/SSIM of different architectures of encoder/decoder CC dataset.
UNet D4
UNet D33
UNet D)2
58.32/0.9609
$7.89/0.9577
37.98/0.9581
UNet Skip D4 36.65/0.9499
UNet Skip D3 37.37/0.9543
UNet Skip D2 37.69/0.9555
R(C(64)-C(64))
R(C((2)-CC(4)-CC C(32)) 38.34/0.9609
((C(64)-C(128)-C(6)) 38.35/0.9609
38.34/0.9610
Table Averaged PSNR/SSIM of NN+FFDNet on CC dataset.
Method
DrCNN NN+DnCNN
FFDNet FFDNet+ NN+FFDNet
PSNR
33.86
35.02
34.63
37.63
38.00 0.9574
SSIM 0.8635
0.9069
0.8551
0.9555
NN+BM3D is not good DrCNN and N2N which specially refined in the dataset, it is better than all unsupervised method in terms of PSNR and SSIM.
D ABLATION EXPERIMENTS
Different architectures of NN. Here we show the results of different choose of the architectures of neural network. We evaluate method with UNets of different downsample scales (denote by D), with or without skip connection (denote by wo skip), and different CNNs, denote C(n) as wwwvouutioonn--Bacchooore
R(model)(x)
model(x). The results CC dataset
given in Table g
NN+FFDNet. We replaced DrCNN with FFDNet, and the result on CC is given Table 9 it is shown that FFDNet benefits more significantly from our method and is better than the results of its multi-scale version FFDNet+.
VISUAL RESULTS ON REAL WORLD NOISE
We show visual results on real world noisy images in E.1 and real fluorescence microscopy images in E.2 Moreover, ten real noisy images are captured by consumer cameras with ISO=3200 or 320, similar to the CC dataset, we crop a 512 x 512 region in each image to evaluate the performance NN+BM3D, see F.3
E. VISUAL EXAMPLE OF CC, POLYU
We show visual results of ten noisy images from CC (Nam et al. 2016) and PolyU (Xu et al. 201 Sa
datasets. BM3D (Dabov et al. 2007a),, DrCNN (Zhang al. 2017). NC (Lebrun al. MCWNNM (Xu et al. 2017) and TWSC (Xu et al. 2018b) are evaluated for comparison. See CC's results in page 16. 17 and PolyU's results in page 18, 19.
2015b).
E.2
VISUAL EXAMPLE OF FMDD
Two images from FMDD Zhang et al. (2119) datasets are evaluated for visual comparisons. We compared our approach with VST method (Makitalo & Foi 2012). See page 20.
E.3 VISUAL EXAMPLES OF REAL IMAGE
Ten real noisy images evaluated for visual comparisons. The results of three traditional methods (BM3D (Dabov et al. 2007a). MCWNNM (Xu et al. 2017) and NC (Lebrun et al. (015b)) and three deep learning methods (VDN (Yue al. 2019), DnCNN (Zhang et al.) 2017) and FFDNet (Zhang ct al. 2018)) are shown here. See Figure 10 page 21, 22, 23.
15

---

Published as a conference paper at ICLR 2021
(a) Noisy 34.93/0.8406
(b) BM3D
(c) DnCNN
(d) NC
37.18/0.9113
35.73/0.8588
39.29/0.9536
Ground Truth
(I) MCWNNM
(g) TWSC
(h) NN+BM3D
41.15/0.9774
41.05/0.9831
42.63/0.9858
(a) Noisy 33.28/0.8996
(b) BM3D
(c) DrCNN 33.95/0.9112
(d) NC
34.60/0.9324
35.57/0.9479
(e) Ground Truth
(f) MCWNNM
(g) TWSC 35.34/0.9547
(h) NN+BM3D 35.80/0.9555
34.98/0.9469
(a) Noisy 33.88/0.894. 8930
(b) BM3D 35.38/0.9366
(c) DnCNN
(d) NC
34.21/0.9013
35.60/0.9399
(e) Ground Truth
(f) MCWNNM
(g) TWSC.
(h) NN+BM3D
35.37/0.9328
35.90/0.9328
37.32/0.9590
16

---

Published as a conference paper at ICLR 2021
(a) Noisy 34.81/0.8492
(b) BM3D
(c) DECNN
(d) NC
36.63/0.9065
35.39/0.8683
38.17/0.9360
(e) Ground Truth
(1) MCWNNM 38.30/0.9394
TWSC
(h) NN+BM3D 39.28/0.9515
38.77/0.9418
(a) Noisy 33.83/0.9175
(b) BM3D
(c) DnCNN
(d) NC
35.190.9520
34.11/0.9223
35.57/0.9497
Ground Truth
(1) MCWNNM
(g) TWSC 13.10/0
(h) NN+BM3D
34.91/0.9475
37.02/0.9679
Figure 7: Visual results and PNSR/SSIM of five noisy images from CC.
17

---

Published as a conference paper at ICLR 2021
(a) Noisy 36.79/0.9165
(b) BM3D
(c) DnCNN
(d) NC
37.68/0.9442
36.83/0.9199
37.70/0.9373
Ground Truth
(I) MCWNNM
(g) TWSC 37.82/0.9269
(h) NN+BM3D
38.42/0.9534
38.95/0.9590
(a) Noisy 36.42/0.9198
(b) BM3D
(c) DrCNN 36.58/0.9227
(d) NC
37.39/0.9422
37.34/0.940S
Ground Truth
(f) MCWNNM
(g) TWSC 37.47/0.9340
(h) NN+BM3D
37.94/0.9501
38.31/0.9538
(a) Noisy 36.88/0.92 9900
(b) BM3D
(c) DnCNN 37.06/0.9279
(d) NC 36.88/0.9240
37.87/0.9474
(e) Ground Truth
(f) MCWNNM
(g) TWSC.
(h) NN+BM3D
38.43/0.9552
37.37/0.9273
38.80/0.9582
18

---

Published as a conference paper at ICLR 2021
(a) Noisy 34.60/0.8739
(b) BM3D
(c) DrCNN
(d) NC
36.34/0.9325
35.08/0.8823
36.46/0.9349
(e) Ground Truth
(I) MCWNNM
TWSC 37.45/0.9730
(h) NN+BM3D 38.08/0.9734
36.39/0.9453
(a) Noisy 35.27/0.9125
(b) BM3D
(c) DnCNN
(d) NC
36.30/0.9309
35.55/0.9151
35.72/0.9021
(e) Ground Truth
(1) MCWNNM
(g) TWSC 35.09/0.8760
(h) NN+BM3D 37.02/0.9382
36.54/0.9366
Figure 8: Visual results and PNSR/SSIM of five noisy images from PolyU.
19

---

Published as a conference paper at ICLR 2021
(a) Noisy 23.89/0.3413
(b) NLM
(c) BM3D
(d) DrCNN 24.33/0.3624
31.53/0.7465
24.38/0.3659
(e) VST+NLM
VST+BM3D 24. 24.76/0.3859
(g) VST+DaCNN 24.56/0.3759
23.92/0.3429
(h) Clean
(i) NN+NLM
(i) NN+BM3D 32.73/0.8062
(k) NN+DnCNN 24.85/0.3849
32.04/0.7628
(a) Noisy 25.62/0.3634
(b) NLM 35.01/0.9128
(c) BM3D
(d) DrCNN 25.72/0.3699
25.97/0.3858
(e) VST+NLM 25. 25.45/0.3635
(1) VST+BM3D
(g) VST+DuCNN 25.73/0.3700
26.36/0.4096
(h) Clean
(i) NN+NLM 36.01/0.9186
(j) NN+BM3D
(k) NN+DnCNN 26.15/0.3901
32.87/0.8407
Figure 9: Visual results and PNSR/SSIM of two noisy images from FMDD.
20

---

Published as a conference paper at ICLR 2021
Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DrCNN
(g) FFDNet
(h) NN+BM3D
Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DrCNN
gg) FFDNet
(h) NN+BM3D
(a) Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DOCNN
(g) FFDNet
(h) NN+BM3D
(a) Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DnCNN
gg) FFDNet
(h) NN+BM3D
21

---

Published as a conference paper at ICLR 2021
Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DOCNN
(g) FFDNet
(h) NN+BM3D
Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DnCNN
(g) FFDNet
(h) NN+BM3D
1955
WEL
1911
1931
(a) Noisy
(b) BM3D
(C) MCWNNM
(d) NC
1955.
(e) VDN
(f) DOCNN
(g) FFDNet
(h) NN+BM3D
(a) Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DnCNN
gg) FFDNet
(h) NN+BM3D
22

---

Published as a conference paper at ICLR 2021
(a) Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(c) VDN
(f) DnCNN
(g) FFDNet
(h) NN+BM3D
(a) Noisy
(b) BM3D
(c) MCWNNM
(d) NC
(e) VDN
(f) DnCNN
(g) FFDNet
(h) NN+BM3D
Figure 10: Visual results for real-world image denoising.
23


Reviews:
[{"review_id": "z7QvpJZ8Qw-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an unsupervised approach for denoising for which works on a single noisy image. The central idea is to use a neural network to map the noisy image to a latent image space where the noisy distribution follows Gaussian (like Variance Stabliziation Transform) and then use a off-the-shelf Gaussian denoiser on this image. The paper shows good numerical results. \n\n1. The details of the algorithm is not clear to me. Suppose your noisy image, y has pure Poisson noise. When you use Algorithm 1 and try to apply Gaussian denoiser to $x^{k+1}+q^{k}/\\rho$ does it still work? In my experience, these algorithms, particularly DnCNN is very sensitive to the type of the noise and I wonder how much of this algorithm holds when the noise distribution is significantly different from Gaussian. The test cases (including the synthetic noise) here seems to be very close to Gaussian. Authors should produce evidence that the algorithm can actually work on noise distribution significantly different from Gaussian distribution. \n\n2. The paper does not compare to some relevant baselines including:\na) \"Self2Self With Dropout: Learning Self-Supervised Denoising From Single Image\" Yuhui Quan, Mingqin Chen, Tongyao Pang, Hui Ji; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 1890-1898\nThis paper introduces a denoising method which works on a single noisy image and is agnostic to the noise distribution. \nb) \"High-Quality Self-Supervised Deep Image Denoising\" Samuli Laine, Tero Karras, Jaakko Lehtinen, Timo Aila, NeurIPS 2019\nhas better empirical results that the N2N baseline used in the paper\n\n\nUpdate after rebuttal: I appreciate the authors response and have updated my score. Please see some lingering thoughts below:\n\n1. It looks like the experiments show that just the NN based Gaussianization of latent space is not effective when the noise distribution is significantly different from Gaussian. The authors had to  introduce an additional VST in the latent space to make it work. I think this is a limitation of the work - the method just by itself is not capable of handle of noise type significantly different from Gaussian. In my opinion this should be acknowledged in the paper. \n\n2.  I think it is unfair to use a pretrained HQS model for comparison. The authors should retrain the model on the particular datasets they are interested in.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "UCEFMBfOwal", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The key idea of this paper is to transform a real-world noisy image into a latent image space using an encoder neural network(NN), where the noise is hopefully white Gaussian in the latent image space so that existing Gaussian denoisers can be used in the optimization steps. The proposed methods are reported to have improved performance compared to original Gaussian denoisers combined in their methods. One specific proposed combination, i.e., NN+BM3D shows better results than other unsurprised denoising approaches and is competitive with a supervised network. The idea of this paper is interesting and the presentation of the major idea is relatively clear.\n\nHowever, the reviewer suggests that the authors address the following questions and issues to make the paper clearer:\n1. Near equation (1), please define $y$, i.e., an input noisy image.\n2. In (2), I think a $\\lambda$ is missing and the equal sign does not hold. There should be a constant. Further, seem that the dimension of an image, i.e., $N$, is also used as an iteration number in Algorithm 1. Please avoid this confusion, unless there is a reason to do so.\n3. Before Remark 1, the authors mention that the sampling number is equal to 1. I think it would be better to mention when this sample is taken in Algorithm 1. Currently, in my guess, I think $\\epsilon$ is re-sampled every time before backpropagation is made so that it looks like a \"stochastic\" step, however, there are also other possibilities. Please clarify this.\n4. For Algorithm 1, there is a significant concern for initializing network parameters $\\theta_1$ and $\\theta_2$. Are they randomly initialized for your optimization algorithm, or they are inherited from some pre-trained models? I have this concern because in my mind, when updating $x$, it should not go too far away from $y$, and it gradually converges to the final results. If $\\theta_1$ and $\\theta_2$ are randomly initialized, $x$ might be very different from $y$ in the first several steps and the convergence is not very straightforward. However, if it can still converge, then the robustness of this algorithm can be demonstrated. Please clarify how these parameters are initialized.\n5. I recommend that the authors show the latent image and denoised image in $k_{\\rm th}$ iteration step for different $k$ in Algorithm 1. The results for NN+BM3D and one noisy image could be enough. For example, the authors can show the latent image at the first several iteration steps and the latent image when the algorithm tends to converge.\n6. Are the results of DnCNN+, FFDNet+, CBDNet, etc., directly copied from other papers, for example, Hou et al. (2019)? If so, please highlight this, for example, with footnotes. Further, please clarify why the results for the proposed method in Hou et al. (2019) are not compared.\n7. It is better to mention the running time in section 4 or 5, rather than in the Appendix.\n\n-----\nAfter rebuttal:\n\nI think the authors well-addressed my comments. Thanks for their efforts. I would suggest that the authors include some results in their response in the paper or supplementary results if they have not done it.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HMz1qWpWo9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##Updated Review##\n\nI'd like to thank the authors for their comments, clarifications and modifications.  I still believe that this paper is a novel contribution with clear impressive results.\n\n# The main idea: \nDeveloping an unsupervised imagedenoising approach is difficult because the real world noise distribution is much more complex than the simplified AWGN assumptions. The authors address this by mapping the noisy image into a latent space in which. AWGN assumption holds, and thus any existing gaussian denoiser is applicable.  The authors claim that the network can be trained with the input image itself and does not require any pertaining in other datasets.\n\nThis is a clever idea, it essentially is MAP gaussian denoising with a variational auto encoder as the likelihood model.\n\nSince it is an image-specific network it reduces the need for wide-space sampling fo the data distribution.  How fast is it?  The authors optimize with ADMM iteratively updating network weights and image estimates (which can be time consuming).  No mention is made of compute time when compared to comparable methods.\n\nThe encoder and decoder network shapes and architectures are chosen without much reasoning given other than that they are standard choices.  Did the authors explore any hyper parameter sweeps (of the Unet architectures specifically) in this space to see if performance was further improved or impoverished by different choices?\n\nStrength:\nCan be utilized with other existing gaussian denoisers to improve their performance.\n\nVisual Results are high quality and show clear improvement for baseline methods.\n\nWeaknesses:\nThe visual improvement for NN+DnCNN is debatable though the other results show significant improvement. (At least in the chosen examples)\n\nThe following is stated as fact but is not backed up with data.\n\nIt is noted that there is an expectation term in (4) that can be estimated by the Monte Carlo method and there is no need for a large amount of samples in practice. In our experiments, the sampling number is equal to 1.\n\n\nPlease fix:\nOn page 1 you write 'Due to the violence of the AWGN assumption” I assume this is supposed to read “Due to the violation”.\n\nAlso, on page 5 you write :\n\"Remark 1 In our model, the basic assumption is that the latent image z is a white Gaussian per- turbations of clean image x, i.e. z|x ∼ N (x, σ2I). Since z = Gθ2 (y), the second term in (4) can be seen as the transformed data fidelity while the first term is a regularization such that the encoder avoids from the trivial mappings, e.g. identity map or zero map.\" \n\nThe first term, which is  1Eε||Fθ1(Gθ2(y)+ε)−y||2  does prevent a zero mapping, but does not on it's own prevent the identity map (it is in fact encouraging it).  However, the combination of the first, second ( 1 ||Gθ2(y)−x||2) and third (λR(x)) is what pushes against the degenerate identity mapping solution.  Please update this sentence to be more correct.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FJP454Fe-yn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to perform denoising using a single image. A UNet encoder is used to map the image to a space where a denoiser such as BM3D is applied. ADMM is used to update the weights by minimizing a cost function where one of the terms fits the noise image (using an additional UNet decoder trained jointly) and the other term is associated to the denoiser. The method yields promising results.\n\n*Quality*: \n\nThe methodology proposed by the paper is intriguing, but I don't think it is analyzed adequately. The Bayesian motivation for the method is too detached from the algorithm that is finally applied. For example, the authors explain the method using expectations and then at some point state the following:\n\n\"It is noted that there is an expectation term in (4) that can be estimated by the Monte Carlo method and there is no need for a large amount of samples in practice. In our experiments, the sampling number is equal to 1.\" \n\nThis makes no sense. In fact, in Algorithm 1 we see that the $\\epsilon$ term from (4) does not appear. Also, the authors state \n\n\"the proposed NN approximates the likelihood in the classic Bayesian framework, which gives clear interpretations of each loss term\"\n\nbut show no evidence that the terms are actually meaningful. Due to the lack of analysis, the method seems very ad hoc. For example, it is unclear what role the decoder has. How do the results change with different choices of encoder/decoder? It would be very helpful to add some ablation studies. \n\nAlso, the authors show that the method is stable with respect to the hyperparameters, but it seems to be that there should be a meaningful dependence with $\\sigma$. How should it be estimated if you want to train on a single image, where the noise is not Gaussian? In the experimental section, they state: \"The noise level is estimated from Donoho & Johnstone (1994); Chen et al. (2015a)\". Is this related to $\\sigma$? I don't understand what the authors mean.\n\n*Clarity*: \n\nThe paper is not written clearly. It took me a while to understand what the authors were actually doing.\n\n*Significance*:\n\nIt seems to me that the authors have designed a promising method, but unfortunately they do not provide enough insight and analysis to show why it may work, or what the limitations may be. This will limit the possible impact and usefulness of the paper significantly.\n\n*Pros*:\n\nThe method is interesting and novel to the best of my knowledge. It produces promising results.\n\n*Cons*: \n\nNot much analysis. Confusing motivation. Lack of clarity.\n\nUpdated review: I appreciate the authors' response and have updated my rating. However, I still believe that the clarity of the exposition could be improved. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: After reading the author’s response, all reviewers recommend accepting the paper. R2 and R3 strongly support the paper while R1 and R4 consider it borderline.

There is agreement that the idea of the work is interesting and novel. The experimental results look solid. 

The authors provided an extensive response addressing most of the concerns of the reviewers. In light of this feedback, the reviewers provided some additional comments (which the authors could not address, as the discussion period was over). The AC considers that the authors should incorporate this feedback to the final version of the manuscript. Specifically,

Responding to R1's first question regarding the noise distribution on the original image being significantly different from Gaussian. The authors provided detailed results, which is highly appreciated. As R1 points out, the authors had to introduce an additional VST for the method. These results should be added to the manuscript is important, to show the limitations of the approach.

R3 asks about the importance of the initialization of the weights of the encode and decoder. This is a natural question as this is a non-convex problem. The authors clarify in the manuscript the initialization of x, but do not comment on the weights. It would be good to add a sentence in this regard (as done in the discussion).

R4 mentioned, and the AC agrees, that the authors should try to improve the clarity of the exposition.

The AC considers it important to add in the appendix more visual examples to quantitatively show the performance of the method.
Decision: Accept (Poster)