Comparison of CNN based and self-similarity based denoising methods
Minchuan Zhou
Stanford University

mczhou@stanford.edu

Abstract
Convolutional networks and traditional self-similarity
based methods are both state-of-the-art denoising methods.
We implement the BM3D model and DnCNN network andshow their favorable performance for denoising a Gaussian
noise at a given noise level, compared to NLM. We also
make use of dilated convolution in DnCNN and show that inthis way we can achieve comparable denoising
performance while greatly reducing the computationalcost. We have also trained a DnCNN-b network for blinddenoising for all noise levels between 0 and 55, whichshows similar denoising results without knowledge of what
the actual noise level is in the noisy image.

1. Introduction
Denoising images has been an important research topic incomputational imaging, especially in low level vision as
denoising is an important step before such images areprocessed for further application. Many new deep learning
based methods are proposed as the rise of new deep learningtechniques, for example, convolutional neural network(CNN). In the meanwhile, traditional methods are stillplaying important roles in many aspects.
The general problem of image restoration can be written
in the form of y = Ax +  , where A is any linear operator,
x is the original image,  is the noise term, and y is theresulting image. In the case of denoising, A is an identity
operator, and y is the noisy image. In addition, we canassume that the noise is Gaussian noise with standard
deviation  . The purpose is to recover the original clean
image x from the noisy image y . In class, we have learnt
denoising techniques such as Gaussian filter, bilateral filter
and non-local means (NLM) [1], among which NLM gives
the best denoising result. NLM averages pixels with asimilar neighborhood. Like other traditional methods, NLM
can produce a high denoising quality but has the drawback
of low computational efficiency and requires manual choiceof some parameters, such as the patch size, search window

size, and cut-off distance. More advanced traditional
methods are also being developed, such as block-matching
and 3D filtering (BM3D) [2].
On the other hand, CNN is widely used in many
computational imaging aspects, such as denoising,demosaicing, superresolution, etc. As a discriminative
learning method, CNN models do not require choosing theparameters manually, also require no iterative optimizationin the test phase therefore is highly computationallyefficient. In particular, the denoising CNN (DnCNN) [3]separates the noise from a noisy image by feed-forward
CNN using residual learning and batch normalization. The
advantage of CNN is two-fold: first, the deep structure of
CNN is suitable for exploiting image characteristics; second,
CNN can be trained on GPUs, which improves the run time
performance significantly.
In this report, we implement the state-of-the-art denoisingtechniques, both the traditional method (BM3D) and deeplearning method (DnCNN), and compare the denoising
performance with the NLM method we learnt in class. In
particular, we compare the quantitatively the metrics ofpeak signal-to-noise ratio (PSNR) and structural similarity
index (SSIM) and also qualitatively the visual quality.

2. Related work
2.1. Traditional denoising method
We have learnt a variety of traditional denoising methods
in class. Among all the traditional models, nonlocal
self-similarity models are very important. The NLM is oneof the simplest and most effective of the self-similarity
models.
BM3D is one of the state-of-the-art denoising methods.
BM3D, first proposed in Ref. [2], is also a non-local
denoising method, exploiting self-similarity in the image.
Different from NLM, BM3D is a transfer-domain denoising
method. BM3D makes use of the sparsity of the image in thetransform domain, e.g. DCT transform. The sparsity isenhanced by blocking match, which is to group similar 2D
blocks to 3D arrays. In the paper, the denoising
performance of BM3D is shown to be better than otherdenoising methods such as BLS-GSM [4] and K-SVD [5].

The basic idea is to divide the image into square blocks of
fixed size and group similar 2D blocks to 3D arrays byblock matching and then apply 3D collaborative filtering.
Both BM3D and NLM works very well for images withregular and repetitive patterns.

3. Method
3.1. Non-local means (NLM)
The idea of NLM is a weighted average of all pixels in theimage to get a denoised image z :2.2. Deep neural network denoising
The other kind of state-of-the-art denoising method is
deep neural network denoising. Compared toself-similarity-based methods, these neural networks work
better for images with irregular structures. In 2017, B. Ahn
and N. I. Cho proposed a block-matching convolutional
neural network (BMCNN) method that combines BM3D
and CNN denoising [6].
A variety of neural network based denoising models aredeveloped recently. For example, H. C. Burger et. al,proposed multi-layer perceptron (MLP) [7] for image
denoising at specific noise level in 2012, which exhibits
comparable performance as BM3D. In 2017, K. Zhang et.
al, proposed the DnCNN model that can perform Gaussiandenoising, using techniques such as residual learning [8]and batch normalization [9] are used in the model. In
addition, to decrease the computational cost, we use dilated
convolution in the CNN [10], which enables similar
denoising performance with fewer number of convolution
layers. In the following we discuss these techniques briefly.
DnCNN model with residual learning [8] estimates theGaussian noise instead of the clean image. Suppose that therecovery of the clean image from the noisy image is denotedby the mapping( y) = y −  . Instead of estimatinginresiduallearning,
weestimate( y) ,( y ) = y − ( y ) =  . Residual learning can address thedegradation problem in training deep neural networks.
When training a CNN, the distribution of a layer’s inputs
changes with the parameters of its previous layers. This
internal covariate shift slows down the training. To
overcome this issue, we can use batch normalization [9],which normalizes the inputs in a mini-batch by subtracting
the mean and dividing by the standard deviation of thebatch. Batch normalization brings in two additionalparameters (scale and shift) to the layer.
The DnCNN model usually requires many layers. Using
dilated convolution in the layers [10] can enlarge thereceptive field and achieve comparable performance as themodel with more layers. This helps reduce thecomputational cost.

z=

y

j

ij

j

j

(1)

ij

and ij is the weights that measures the similarity between
the neighborhood around the i th pixel and the j the pixel:ij =  k [v( y ) − v( y ) ]2 
1
m m
i m
j m
,
exp 


Zi
h2



(2)

  k [v( y ) − v( y ) ]2 
m m
i m
j m

Z i =  j exp 


h2


Here v( yi ) is the neighborhood around the i the pixel in theimage, m denotes the pixels in the neighborhood, km is a
Gaussian kernel, and h controls the decay of the weight.
In the project, we use the restoration.denoise_nl_means
function in the skimage module of Python.

3.2. BM3D
In BM3D denoising, block matching is realized byfinding blocks that are similar to a reference block. The
similarity is determined by finding the distance d between
the two blocks. Given a threshold distance  th , two blocks
with a distance smaller than the threshold distance ( d   th )is said to be similar. The similar blocks are stacked to form a
3D array, denoted by Z . Then the collaborative 3D
filtering is carried out to suppress the noise – we first DCT
transform the 3D array, filter it using either hard-threshold
filter or Wiener filter, and finally perform inverse-DCT
transform. The idea is to shrink the coefficients in the DCTtransform domain and filter out the noise components.
Finally, an estimate of the clean image is obtained byweighted averaging the overlapping blocks.
When denoising color image with BM3D, we firsttransform the RGB image to YCrCb space, and thegrouping is based on the information in the luminancechannel only, since the luminance channel contains most of
the information. The transformation matrix from RGB to
YCrCb is: 0.299 0.587 0.114 
ARGB 2YCrCb =  0.500 −0.419 −0.081
(3)
 −0.169 −0.331 0.500 



WC =

T3 D ( Bbasic )2

T3 D ( Bbasic ) +  C2
2

.

(7)

Finally, we use weighted average to get the final estimate.
The use of Wiener filtering gives more accurate filteringthan hard-threshold filtering in Step 1. As a result, the final
estimate outperforms the basic estimate.
We implement the full BM3D model in Python for color
image denoising. The results are compared to other methods
in Section 4.
Figure 1: Flow chart of BM3D denoising algorithm.

Suppose we add independent Gaussian noise to RGB
channels with a standard deviation of  , then the varianceof the noise in YCrCb channels are:  Y2   0.299 0.587 0.114    2   0.669 2 
 2   2  2 2  Cr  =  0.500 −0.419 −0.081     =  0.657   (4)  2   −0.169 −0.331 0.500    2   0.6232   
 Cb  
We first slice the noisy YCrCb image in a set ofoverlapping blocks. Then the estimate consists of two steps– basic estimate and final estimate – as shown in the
flowchart in Figure 1.
In the first step (Step 1), the blocks are grouped using thesimilarity criteria based on the d-distance between two
blocks Bi and B j defined as

T2 D ( Bi ) − T2 D ( Bi ) 22d ( Bi , B j ) =

2
N blo
ck 1

(5)

where T2D denotes a 2D transform and Nblock1 is the block
size used in Step 1. Then we apply 3D transform to thematched noisy blocks and use hard-threshold filtering in thetransform domain, based on assumed sparsity there.
Denoised blocks can be obtained from inverse 3D
transform. Then the blocks are stitched together to form thebasic estimate using weighted average with a weight of(6)
1i =  C−2 Ni−1
where  C is the standard deviation for channels
C = Y , Cr , Cb as defined in Eq. (4), and N i is the numberof nonzero coefficients kept in the transform domain.
In the second step (Step 2), since we already have a roughestimate of the clean image, we use the basic estimate toperform the block-matching, and create 3D arrays ( Bnoisy

and Bbasic ) from the noisy image and the basic estimate,respectively. This improves the grouping compared to Step
1. Then we apply 3D filtering to Bnoisy in the transformdomain. The difference with Step 1 here is that we use
empirical Wiener filtering, where the Wiener shrinkage
coefficients is calculated using Bbasic :3.3. DnCNN methods
We implement two different structures of neuralnetworks: one is the DnCNN proposed in Ref. [3]; the otherone makes use of dilated convolution to decrease thecomputational cost, which we call DnCNN-d.
The first network (DnCNN) is schematically illustrated in
Figure 2 (a). A total of 17 layers are used here. We use 64
filters of size 3  3  c in the 1st layer, 64 filters of size3  3  64 in the 2nd to 15th layer, and finally c filters of size3  3  64 in the 16th layer. Zero padding are used in the
convolution so that the output has the same size as the input.
The receptive field of the model is 33  33 . Here c is thenumber of channels in the image. In our color image case,c = 3 . Batch normalization is used in the layers to help with
the issue of internal covariate shift that slows down the
training process.
The architecture of the second network (DnCNN-d) isshown in Figure 2 (b). We use dilated convolution in the
network [10] to increase the receptive field. A dilated
convolution with a dilation rate of n is denoted by DConv-n.
With dilated convolution, a DnCNN-d network with 7
layers has a receptive field of 33  33 , same as that of
DnCNN with 17 layers.
We use Keras API in Python for both models. The meansquared error between the CNN output and the target clean
image is used as the loss function for training. We use Adam
optimizer in the training, which is an adaptive learning rate
optimization algorithm. For training the networks, we use
the 400 images train and test images from BSDS500 [11],while the rest 100 images are left for evaluation.
To train the network for denoising at a given noise level,
we add Gaussian noise with standard deviation at the givennoise level, so we have a pair of clean and noisy images. The
noise in RGB channel are added independently. We thengenerate overlapping patches of size 50  50  c with a
10  10 stride (no overlap in the third dimension) for each
image. A total of 492,800 patches are generated for the setof clean images, as well as the set of noisy images, which arethen grouped into 3850 mini-batches of 128 patches. The
input of the training is patch pairs – a clean patch and anoisy patch.

DnCNNDnCNN-dNLM
BM3D

Training time
(ms/step)
803
423
-

Average testingtime (sec)
1.583
0.9528
32.78
243.5

Table 1: Comparison of training time and testing time with
DnCNN and DnCNN-d.

4.2. Number of epochs

Figure 2: The architectures of (a) DnCNN network and (b)DnCNN-d network.

The loss function decreases with the number of epoch. As
can be seen from Figure 3, the loss function is converging
around 25 epochs. Therefore, for the rest of the report, we
use 25 epochs in the training.

3.4. Comparison metrics
Two popular objective image quality metrics are PSNR
and SSIM. PSNR between two images x and y can becalculated from max( I original ) 2 
PSNR = 10 log10 
(8)
 .

MSExy


The SSIM can be calculated using( 2 x  y + c1 )( 2 xy + c2 )SSIM =(  +  + c )( +  + c )
2
x

2
y

1

2
x

2
y

.

(9)

2

where  and   are the mean and standard deviation ofimage  = x, y , and  xy is the covariance of x and y.
ci = (ki L)2 , i = 1, 2 with k1 = 0.01 , k2 = 0.03 , and L
being 255.

4. Results and discussion
4.1. Training time and testing time
The traditional methods (NLM and BM3D) does not
require training, while the two neural networks (DnCNN
and DnCNN-d) needs to trained first. As there are fewer
layers in DnCNN-d, the training time is almost half of the
training time for DnCNN, as shown in Table 1. The average
testing time is also less for DnCNN-d. In terms of testingtime, DnCNN-d requires shortest testing time, and the
traditional methods are much slower than neural networks.
We also notice that the BM3D method we implement isslower than the other methods. We will need to modify ourimplementation so that it is more computational efficient in
the future.

Figure 3: The architectures of (a) DnCNN network and (b)DnCNN-d network.

4.3. Gaussian denoising performance
For evaluation of denoising performance, we use the rest
100 images in CBSD500 dataset. We calculate the average
PSNR and SSIM, and also the average testing time of thedenoised images for each denoising method we discussed
above, the results of which are summarized in Table 2. In
terms of PSNR and SSIM, the deep neural network based
methods have better performance than traditional denoisingtechniques. In particular, DnCNN performs the best.
DnCNN-d has similar performance with DnCNN, which
confirms that dilated convolution can be used to reduce thecomputational cost of DnCNN denoising, given that thetraining and testing time are both reduced greatly as
discussed in Sec. 4.1. Comparing the two traditional
methods, we find that BM3D gives overall better denoising
results than NLM.

PSNR(dB)
SSIM

NLM
27.90
0.7611BM3D
28.20
0.8126

DnCNN
31.00
0.9324

DnCNN-d30.85
0.9318

Table 2: Comparison of average PSNR, SSIM, and testing time of
NLM, BM3D, DnCNN and DnCNN-d using test images in
CBSD500 dataset with noise level of 25.

PSNR(dB)
SSIM

NLM
24.25
0.6246

BM3D
24.54
0.6509DnCNN
27.56
0.8668DnCNN-d27.62
0.8658

Table 3: Comparison of average PSNR and SSIM of NLM,
BM3D, DnCNN and DnCNN-d using test images in CBSD500
dataset with noise level of 50.

NLM
Clay
pot

Stone
carvin
g

PSN
R
SSIM
PSN
R
SSIM

32.69

BM3
D
32.64

DnCN
N
35.09

DnCN
N-d
34.90

0.822
7
22.55

0.836
8
24.01

0.95740.9550

27.68

27.53

0.831
4

0.857
8

0.93570.9396

Table 4: Comparison of PSNR and SSIM for of two image
examples in the CBSD500 dataset with noise level 25.

Examples of testing images and denoised results areshown in Figure 4. The quantitative comparison is shown in
Table 4. As can be seen from both PSNR and SSIM, neuralnetworks perform the best. In the case of the clay pot image,which has less details and more regular patterns, the NLM
method works well, but lots of details in the image areremoved. The result of BM3D also loses some details. The
two neural networks give similar results. For the stone
carving, which has a lot of details and irregular patterns,
there is still significant amount of noise in the NLM result.
The BM3D result appears to be over-smoothed. NLM andBM3D has better performance when the image has regularand repetitive patterns.

4.4. Blind denoising
The network trained for denoising at noise level 25
cannot properly denoise images with noise level 50, as
shown in the Figure 5. Similar in BM3D, if we set  = 25 ,there are still significant amount of noise in the denoised
image. The PSNR of the denoised result of DnCNN-d andBM3D are 22.01 and 21.82, respectively. These methods do
not perform well if we do not know a priori what the noiselevel of the image is. In order to perform ‘blind’ denoising,
we train a DnCNN-b network using the same structure as
DnCNN-b. We set the noise level of the training images to a
uniform random distribution between 0 and 55. The
denoised image is shown in Figure 5 (c). In addition, weshow the results of the denoising performance of the
network at noise level 25 and 50 as summarized in Table 5
and Table 6, respectively. As can be seen, DnCNN-b has asimilar performance as the DnCNN-d network trained at thecorrect noise level, and gives much better results than the
DnCNN-d network trained at an incorrect noise level.
In Figure 6, we show the results of DnCNN-b denoising
applied to an example image with noise level 15, 25, and 50,
respectively. Our trained DnCNN-b works well for noiselevel below 50. We notice that When the noise level is 50,
the denoised images show some artefacts and the result isover-smoothed.

5. Discussion and future work
We have implemented two state-of-art denoising methods: a
deep learning method DnCNN and a traditional BM3D
method. We have shown that DnCNN performs better than
traditional denoising methods BM3D and NLM; and BM3D
has an overall better performance than NLM. For DnCNN,

Figure 4: Color image denoising results of two image examples from the CBSD500 with noise level 25.

training sets and access the performance of the networks totry to mitigate this issue.
Self-similarity based denoising methods works better forimages with regular patterns, but may over-smooth images
with a lot of details; DnCNN methods, on the other hand,works very well for images with irregular patterns.
Combining the two methods may give a denoising methodthat outperforms each of them. We would like to investigate
as future work as well.

Figure 5: Resulting of denoising images at noise level 50 using (a)DnCNN-d trained at noise level 25, (b) BM3D model with
 = 25 , and (c) DnCNN-b.

`
Figure 6: Resulting of denoising images at noise level 15, 25, and
50 using DnCNN-b.

DnCNN-b
PSNR
(dB)
SSIM

30.77

DnCNN-dtrained for σ=25
30.85

DnCNN-dtrained for σ=5027.77

0.92930.93180.8415

Table 5: Average PSNR and SSIM of DnCNN-d trained fornoise level 25 and DnCNN-d using 100 test images in CBSD500
dataset with noise level of 25.

DnCNN-b
PSNR
(dB)
SSIM

27.41

DnCNN-dtrained for σ=50
27.62

DnCNN-dtrained for σ=25
21.65

0.86410.86580.5992

Table 6: Average PSNR and SSIM of DnCNN-d trained fornoise level 50 and DnCNN-d using 100 test images in CBSD500
dataset with noise level of 50.

we have trained two different structures of networks(DnCNN and DnCNN-d) for noise levels 25 and 50,
respectively. DnCNN-d makes use of dilated convolution to
reduce the computational cost without compromising theperformance of the network. We have also trained a
DnCNN-b network for blind denoising, which applies to allnoise levels between 0 and 55.
We also note that DnCNN requires the shortest testingtime, while BM3D is the least computational efficient. In
the future, we will modify our implementation to reduce thecomputational cost of BM3D.
One thing we found in training DnCNN is that thedenoising performance is affected greatly by the training
data we chose. Also, we notice that the denoised image froma high noise level image shows some artefacts and isover-smoothed. In the future, we would like to use different

Acknowledgement
We would like to thank Hayato Ikoma for his support in thiswork.

References[1] A. Buades, B. Coll and J. -. Morel, "A non-local algorithm
for image denoising," 2005 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition(CVPR'05), San Diego, CA, USA, 2005, pp. 60-65 vol. 2.
[2] Dabov, Foi, Katkovnik, Egiazarian, “Image denoising by
sparse 3D transform-domain collaborative filtering”, IEEE
Trans. Im. Proc. 2007.
[3] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng,“Beyond a Gaussian Denoiser: Residual Learning of Deep
CNN for Image Denoising,” IEEE Transactions on Image
Processing, Volume: 26, Issue: 7, July 2017.
[4] J. Portilla, V. Strela, M. Wainwright, and E. P. Simoncelli,“Image denoising using a scale mixture of Gaussians in the
wavelet domain,” IEEE Trans. Image Process., vol. 12, no.
11, pp. 1338-1351, November 2003.
[5] M. Elad and M. Aharon, “Image denoising via sparse andredundant representations over learned dictionaries,” IEEE
Trans. on Image Process., vol. 15, no. 12, pp. 3736-3745,
December 2006.
[6] B. Ahn, N. I. Cho, "Block-matching convolutional neuralnetwork for image denoising", CoRR, vol. abs/1704.00524,
2017, [online] Available: http://arxiv.org/abs/1704.00524.
[7] H. C. Burger, C. J. Schuler, and S. Harmeling, “Image
denoising: Can plain neural networks compete with BM3D?”IEEE Conference on Computer Vision and Pattern
Recognition, 2012, pp. 2392–2399.
[8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learningfor image recognition,” IEEE Conference on Computer
Vision and Pattern Recognition, 2016, pp. 770–778.
[9] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating
deep network training by reducing internal covariate shift,”
International Conference on Machine Learning, 2015, pp.
448–456.
[10] T. Wang, M. Sun and K. Hu, "Dilated Deep Residual
Network for Image Denoising," 2017 IEEE 29th
International Conference on Tools with Artificial
Intelligence (ICTAI), Boston, MA, 2017, pp. 1272-1279.
[11] The Berkeley Segmentation Dataset and Benchmark,https://www2.eecs.berkeley.edu/Research/
Projects/CS/vision/bsds/