State‐of‐art analysis of image denoising methods using convolutional neural networks

IET Image ProcessingVolume 13, Issue 13 p. 2367-2380 Review ArticleFree Access State-of-art analysis of image denoising methods using convolutional neural networks Rini Smita Thakur, Corresponding Author Rini Smita Thakur rinithakur66@gmail.com Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this authorRam Narayan Yadav, Ram Narayan Yadav Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this authorLalita Gupta, Lalita Gupta Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this author Rini Smita Thakur, Corresponding Author Rini Smita Thakur rinithakur66@gmail.com Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this authorRam Narayan Yadav, Ram Narayan Yadav Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this authorLalita Gupta, Lalita Gupta Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopaln, MP, IndiaSearch for more papers by this author First published: 08 October 2019 https://doi.org/10.1049/iet-ipr.2019.0157Citations: 17AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract Convolutional neural networks (CNNs) are deep neural networks that can be trained on large databases and show outstanding performance on object classification, segmentation, image denoising etc. In the past few years, several image denoising techniques have been developed to improve the quality of an image. The CNN based image denoising models have shown improvement in denoising performance as compared to non-CNN methods like block-matching and three-dimensional (3D) filtering, contemporary wavelet and Markov random field approaches etc. which had remained state-of-the-art for years. This study provides a comprehensive study of state-of-the-art image denoising methods using CNN. The literature associated with different CNNs used for image restoration like residual learning based models (DnCNN-S, DnCNN-B, IDCNN), non-locality reinforced (NN3D), fast and flexible network (FFDNet), deep shrinkage CNN (SCNN), a model for mixed noise reduction, denoising prior driven network (PDNN) are reviewed. DnCNN-S and PDNN remove Gaussian noise of fixed level, whereas DnCNN-B, IDCNN, NN3D and SCNN are used for blind Gaussian denoising. FFDNet is used for spatially variant Gaussian noise. The performance of these CNN models is analysed on BSD-68 and Set-12 datasets. PDNN shows the best result in terms of PSNR for both BSD-68 and Set-12 datasets. 1 Introduction The image is corrupted by noise during its acquisition and transmission process. The noise is introduced into the image by an imperfection in the image capturing devices, from the noise sources present in the vicinity of image capturing devices, faulty memory locations, lossy compression, camera imaging pipeline (shot noise, amplifier noise and quantisation noise), scattering and other adverse atmospheric conditions [1]. Image denoising plays an important role in daily life applications such as satellite TV, magnetic resonance, computed tomography (CT), remote sensing, astronomical applications etc. The different kinds of noise description are given in Table 1 [2, 3]. Table 1. Types of noise in images Types of noise Description Probability density function Gaussian It occurs during film exposure and development of the image. , where , Impulse (salt and pepper noise) It occurs during data transmission and the image pixel value is replaced by a minimum or maximum pixel value of an image. where a and b are extreme image pixel values. Periodic This noise is produced from electronic interferences during image acquisition. - Poisson or photon It occurs due to the statistical nature of electromagnetic waves such as X-rays, visible light and Gamma rays. where k is the Poisson parameter. Gamma It is found in laser-based images. It follows Gamma distribution. Rayleigh It is found in radar range images. It follows Rayleigh distribution. White Power spectrum density of white noise is constant. Its autocorrelation is zero. — Uniform It is caused by quantising the pixels of the image to a number of pre-defined levels. It follows a uniform distribution. where distribution is between image pixel values a and Image denoising is the process of estimating latent clean image from its noisy observation. Image denoising as well as closely related operations like image impainting, blur and artefacts reduction, watermark removal are also recognised as preprocessing tasks for branches of computer vision such as image segmentation and pattern recognition. The noise is modelled as Gaussian, Poisson, Gamma etc. based on probability distribution. It is characterised as white noise and colour noise based on correlation, additive and multiplicative noise based on nature, quantisation and photon noise based on source [1]. If the image x is corrupted by additive white Gaussian noise (AWGN) model, it can be formulated as (1)where denotes the observed noisy image, denotes the input clean image and denotes the Gaussian noise vector with zero mean and covariance matrix (I is the identity matrix). The variety of image de-noising methods have been developed such as non-local means (NLM) filter-based methods, wavelet-based methods, diffusion-based methods, total variation based methods, block-matching and three-dimensional filtering (BM3D), sparse representation-based methods, Markov random field models, neural network-based methods etc. [4-16]. The convolutional neural network (CNN) is a kind of deep neural network, which is being used in image segmentation, image superresolution, image de-blurring, image de-noising, image deconvolution etc. [17]. The emergence of CNN took place in 1962 when research on visual cortex cells of the cat was done. The successful implementation of LeNet 5 indicates the advantage of this network. The CNN has local receptive fields and translational invariant connection matrices. The concept of weight sharing (or weight replication) in CNN reduces the number of free (trainable) network parameters, so the network complexity is reduced and generalisation is improved as compared to the artificial neural network (ANN) [18]. CNNs are easy to train with backpropagation as compared to other ANNs because they have sparse connectivity in each convolution layer [19]. They are widely used in the area of deep learning due to the availability of application-oriented large databases and efficient parallel computing in graphics processing units (GPUs) [20]. CNNs provide better performance in image resolution as compared to traditional sparse representation because it possesses higher representation capability [21]. In sparse representation, sparse dictionaries are constructed by vectorising the image matrices; thus 2D structural information is lost that is the dependency of pixels of local neighbouring regions. On the contrary, CNN is capable of maintaining 2D structural information both in the training and testing phases because the convolution operation considers the local neighbouring image pixels by using 2D masks [22]. The patch-based methods such as NLM and B3MD require a computationally heavy iterative optimisation algorithm, and the performance of these methods are sub-optimal if the images have a low number of self-similarity patches [23]. On the other hand, CNN optimises its weight of the convolution masks, through gradient-based training scheme, which inherently considers self-similarity in the entire set of patches available in a relatively large number of training images [24]. Moreover, if CNN is trained once, it's weights can be learned by another network by transfer learning [25]. The image denoising performance is measured by mean square error (MSE), peak-signal-to-noise ratio (PSNR) and structural similarity index (SSIM), multiscale structural similarity index (MSSIM), correlation coefficient (β), weighted distance (WD) (using , normalised cross-correlation (NK), signal-to-noise ratio (SNR), mean absolute error (MAE), image enhancement factor (IEF), universal quality index (QI), average difference (AD), maximum difference (MD), structural content (SC), correlation quality (CQ), image fidelity (IF), Laplacian MSE (LMSE), peak MSE (PMSE), normalised absolute error (NAE), normalised MSE (NMSE), and -norm [26-28]. The quality of a given image is evaluated jointly by several image quality assessment approaches in which joint models are optimised and the image indices are given by Spearman Rank order Correlation Coefficient (SRCC), Kendall Rank order Correlation Coefficient (KRCC), Pearson linear Correlation Coefficient (PCC) [29]. Table 2 shows some image quality measures of 2D images. In all of these equations, x represents original image, y represents denoised image, n is the noisy image, is the mean value of the image, is the mean value of denoised image, are image dimensions, is the standard deviation of the original image, is the standard deviation of denoised image, are variances of x and y, respectively, is the covariance of x and y and are pixel indices. In the SSIM formula, and are two variables to stabilise the division with the weak denominator, L is the dynamic range of pixel values, and . The function is defined in three ways: or or (in cosine transform domain). In the MSSIM formula, denotes the luminance comparison for scale M, is the contrast comparison at scale j, is the structure comparison at scale j. The exponents are constants which are used to adjust the relative importance of different components. Table 2. Image quality measures Image quality measures Mathematical description correlation coefficient (β) NK MSE PSNR MAE IEF SSIM QI AD MD SC CQ IF LMSE PMSE NAE NMSE MSSIM The denoising CNN models discussed in this paper are DnCNN [30], IDCNN [24], NN3D [22], deep shrinkage CNN (SCNN) [31], CNN for mixed noise removal [32], fast and flexible denoising CNN (FFDNet) [23], denoising prior driven deep neural network (PDNN) [33]. The DCNN [30] uses residual learning along with the batch normalisation, but it suffers from a problem of gradient explosion and fails to converge quickly. So, the IDCNN [24] overcomes this disadvantage by introducing a gradient clipping in the network architecture of [30] which clip the individual gradients in the pre-defined range in order to increase the convergence speed. In order to improve the denoising performance CNN can be used in cascade combination with other conventional denoising methods. In NN3D [22], CNN is used in cascade combination with the non-local filter (NLF). Some CNN models are designed for adaptive noise where a single model is used to remove noise with a wide range of standard deviation. SCNN [31] uses soft shrinkage activation function for adaptive noise reduction. FFDNet [23] is designed to handle spatially variant noise with a wide range of noise levels (i.e. [0, 75]) effectively with a single network. In PDNN [33], model-based methods which rely on the denoising prior are merged with the learning-based CNN model, which maps low-quality images to desirable high-quality images with deep learning. The remaining part of this paper is organised as follows. In Section 2, CNN architecture is discussed. Section 3 presents learning for denoising operation on CNN. Section 4 gives the architecture of different CNNs along with the dataset used for denoising. Section 5 briefly describes some other CNN models with their application. Section 6 gives a description of the datasets used in this review. Section 7 compares the denoising performance in terms of PSNR on BSD-68 and Set 12 datasets. Section 8 concludes the paper. 2 CNN architecture The CNN consists of an input layer, an output layer and a number of hidden layers of neurons which perform an alternating sequence of linear and non-linear transformations. The output of each hidden layer is called a feature map. In each layer convolution kernel is multiplied with the output of the previous layer. The input multiplied by the weights of the convolution kernel is further processed by the activation functions. The lth layer output is given by [24] (2)where represents the selection of the input feature map is the output of previous feature map, is the weight of the convolution kernel of the lth layer, f is the activation function which can be rectified linear unit, sigmoid function etc. and is the bias of the lth layer. In network in network model [34], the convolution operation is replaced by a shallow multilayer perceptron which can represent arbitrary complex functions. After the convolution operation and transformation with activation functions, the size of the feature map is reduced by pooling operation. The output size after convolution operation is given by [35] (3)where is the dimension of an input image, P is the padding depth, is the dimension of the convolutional filter, S is stride which specifies how many pixels a filter is translated horizontally and vertically. A typical CNN architecture provided in [17] is given in Fig. 1. Fig. 1Open in figure viewerPowerPoint Illustration of a typical CNN architecture for pixel RGB images, including the objective function used for training 3 Learning for denoising operation on CNN The different algorithms used for training the CNN are first-order hybrid training methods, conjugate gradient methods, quasi-Newton methods, Levenberg–Marquardt method and its variants and least squares method [36]. The training algorithm uses an objective function approach and learning function approach. In the objective function approach, the minimisation of the reconstruction function is the solution of a problem. In the learning function approach, the solution of a regularised minimisation problem is a parametric function that is used to solve the denoising problem. The loss function or cost function is minimised to find the optimal parameters of the neural network. In supervised learning, this parameter is taken as the MSE. (4)where M is the number of output neurons, P is the number of training patterns, is the number of training iterations, and are the actual and desired responses of the ith output neuron due to the jth input pattern, respectively. The stochastic gradient descent method is used for optimising this MSE. The gradient descent method iteratively updates the weight w replacing by using the following update equation: <Xml: Insert for N with 'ƞ' character in all the places> (5)where ƞ is the learning parameter. The general learning equation of CNN to solve the inverse problem in imaging given in [17] as (6)where a training data set of ground–truth images and their corresponding measurements are known. The cost function is denoted by is the function designed to extract denoised image , the regularisation function is denoted by g which promotes a solution that matches the prior knowledge of input image . Ɵ is the set of all possible parameters in CNN. A differential and tractable loss function is required to train the network with the back-propagation algorithm. The choices of the loss function for solving denoising problem includes norm, norm and Frobenius norm [32]. The Frobenius norm is defined for matrices, so it is used when the image is the input of CNN. The noisy images X are fed as the input of the CNN and the corresponding noise-free images is estimated; the loss function is given by (7)where ||.||F is the Frobenius norm. The weights are the elements of the convolution filter, which are updated using the gradient of loss function f with respect to w. The update is calculated in terms of the weighted average of the square of the gradient which is known as mean squared (MS) error given by [32] (8) (9)In the above equations, and are hyper-parameters known as learning and decaying rates, respectively, ƞ is the iteration number, C is the loss function given by (7) and is the numerical stability factor. In the model-based denoising methods [37-41], the optimisation problem is constructed from a Bayesian perspective by maximising the posterior probability which is formulated as (10)where and denote data likelihood and prior terms, respectively. Equation (10) is reframed for AWGN in which corresponds to the -norm data fidelity term, and the prior term P(x) characterises the prior knowledge of x in a probability setting. So, (10) becomes (11)where denotes the regulariser associated with the prior term and the image restoration problem is given by where y and x denote the degraded image and the original image, respectively, denotes the degradation matrix relating to an imaging/degradation system, and n denotes the AWGN. Then, the desirable solution is the one that minimises both the -norm data fidelity term and the regularisation term weighted by parameter λ. The learning of the CNN requires training dataset of images, i.e. the input and desired output images. In some datasets, training, testing and validation images are pre-defined separately, whereas in other datasets user can select images for training, testing and validation as per the application. The popular image datasets are given in Table 3. Table 3. Some benchmark image denoising datasets and their description Datasets Image description MNIST [42] 70,000 images of handwritten digits of size 28 × 28. RENOIR [43] Colour images corrupted by natural noise due to low-light conditions. The dataset contains over 100 scenes and more than 400 images, including both RAW formatted images and 8 bit BMP pixel and intensity aligned images. SIDD [44] Smart phone image denoising dataset of 30,000 (10 scenes × 5 cameras × 4 conditions × 150 images). DND [45] Darmstadt Noise Dataset. It consists of 50 pairs of real noisy images and corresponding ground truth images that were captured with consumer-grade cameras of differing sensor sizes. BSD-300 [46] Berkeley Segmentation Dataset consists of 300 greyscale and colour segmentation images of size 256 × 256. Training set = 200, Test set = 100. CIFAR [47] It has 60,000 natural images of size 32 × 32. BRATS-2013 [48] 65 MRI of Glioma patients. ILSVRC [20] 1.2 million high-resolution colour images of variable size. LSIP [49] Laser Stripe Image patch Dataset consists of 520 pairs of image patches, each of which is composed of an 8-bit noisy greyscale image and an uncorrupted image. 4 Different CNN models for image denoising 4.1 Deep CNN using residual learning Few authors have considered image denoising is treated as a plain discriminative learning problem in which noise is separated from the noisy image using feed-forward CNN. In [30], residual learning and batch normalisation is used to speed up the training process and boost the denoising performance. Residual learning was initially designed to solve the performance degradation problem, i.e. as the length of the network increases, training accuracy begins to degrade. In residual learning, the output is noise instead of a denoised image. The training efficiency of mini-batch stochastic gradient method is largely reduced by an internal covariate shift, i.e. changes in the distributions of internal non-linearity in inputs during training. Batch normalisation is used to improve the training efficiency, which alleviates the internal covariate shift by incorporating a normalisation step and a scale and shift step before the nonlinearity in each layer [50]. DnCNN model in [30] is used for blind Gaussian denoising that is it can remove Gaussian noise of unknown noise level. Fig. 2 shows the architecture of DnCNN. Fig. 2Open in figure viewerPowerPoint Architecture of deep DnCNN [30] This single CNN model can solve three general image denoising tasks, i.e. blind Gaussian denoising, single image super-resolution with multiple upscaling factors, and JPEG image deblocking with different quality factors. A typical CNN architecture of deep CNN (DnCNN) is given in Fig. 2. There are three kinds of layers in DnCNN of depth D. (i) Conv + ReLU: for the first layer, 64 filters of size 3 × 3 × c are used to generate 64 feature maps, and rectified linear units (ReLU, max(0,·)) are then utilised for non-linearity. Here c represents the number of image channels, i.e. c = 1 for grey image and for colour image. (ii) Conv + BN + ReLU: for layers 2 (D − 1), 64 filters of size 3 × 3 × 64 are used, and batch normalisation [21] is added between convolution and ReLU. (iii) Conv: for the last layer, c filters of size 3 × 3 × 64 are used to reconstruct the output. The training algorithm used in this deep CNN is Stochastic Gradient Descent and Adam's algorithm [51]. The averaged MSE between the desired residual images and estimated ones from noisy input is taken as the loss function to learn the trainable parameters Ɵ. This CNN presented in [24] also uses residual learning without batch normalisation. It consists of the input layer, hidden layers, and the output layer as depicted in Fig. 3. At each layer, convolution operation is performed, which is actually linear filtering followed by a non-linear activation function, which is a ReLU. The ReLU activation function allows a faster and effective training of deep neural architectures on the large and complex dataset as compared to sigmoid function or other activation functions. In this network, fixed and non-fixed noise masks are used. In the fixed noise mask, noise matrix is the same for training contaminated images with the same noise levels while in the non-fixed noise mask; the noise matrix is different for each noise level. Fig. 3Open in figure viewerPowerPoint Architecture of IDCNN [24] The modelling of function R is done to predict , where v is the residual noise. Here represents N noisy-clean training image (patch) pairs. Then we have . So, the loss function is given by , where Ɵ = [W, b] is the network parameter of CNN. The non-convex optimisation problem is solved to minimise . The stochastic gradient descent method is used as learning method to optimise . CNN cannot converge even if the learning rate is >0.01, it suffers from problem of the gradient explosion. Therefore, gradient clipping is used to clip the individual gradients to the predefined range Gradient clipping scheme is applied within a given threshold [52, 53]. The given threshold value is denoted by and the current value of the gradient is denoted by g. If , then (12)Thus, when the current gradient value exceeds a given threshold during training, this value is assigned to . The CNN converges quickly as gradient lies in the fixed range. Some popular CNN using residual learning with data specifications are listed in Table 4. Table 4. List of CNN with data and noise specifications using residual learning CNN Training Dataset Testing dataset Type of noise DnCNN-S [30] 400 images of size 180 × 180 from BSD-500 68 images of the BSD-68 and 12 natural images specific Gaussian noise level DnCNN-B [30] 400 images of size 180 × 180 from BSD-500 68 images of the BSD-68 and 12 natural images blind Gaussian denoising CDnCNN-B [30] 432 images of size 180 × 180 from BSD-500 Remaining 68 images of the BSD-500 blind Gaussian denoising Model_15, Model_20, Model_25, [24] 91 images from BSD-500 Cameraman, House, Lena specific Gaussian noise level Model_mix [24] 91 images from BSD-500 Cameraman, House, Lena blind Gaussian denoising IDCNN-1 [24] 91 images from BSD-500 18 natural images specific Gaussian noise level IDCNN-2 [24] 91 images from BSD-500 18 natural images blind Gaussian denoising 4.1.1 Algorithm for DnCNN-S The algorithm for DnCNN-S (see Fig. 4). Fig. 4Open in figure viewerPowerPoint Algorithm for DnCNN-S 4.2 Non-locality-reinforced CNN for image denoising Non-locality-reinforced CNN for image denoising is the combination of CNN based denoiser and non-local denoiser based on the NLF [22]. NLF exploits the mutual similarities between the group of patches. NLF results in superior noise removal in that part of the image where it exhibits strong self-similarity, such as on edges or on regular texture. The performance of NLF is inferior on pseudorandom textures or singular features, i.e. where the image exhibits weak self-similarity. The image denoising methods based on NLF are BM3D [12], NLBayes [54], and weighted nuclear norm minimisation (WNNM) [55]. The advantages of CNN and NLF are combined through a simple iterative modular framework, as CNN is biased by the learned mapping of the local features and the NLF is biased towards non-local self-similarity. The CNN cascaded with BM3D is termed as NN3D [22] (see Fig. 5). In NN3D, the CNN used are for AWGN removal are FFDNet [23], DnCNN [30], and WDnCNN [56]. Fig. 5Open in figure viewerPowerPoint Architecture of NN3D [22] At high noise levels, the local nature of CNN coupled with the training on external examples, often leads to hallucination. In this case, NLF plays an important role as it can smooth out hallucinations that fail to meet the self-similarity prior. The disadvantage of NLF is that it introduces excessive spatial smoothing. The NN3D counteracts this unwanted smoothening by proceeding iteratively. Specifically, at each iteration k input to NN3D is a convex combination of the original noisy signal y and of the previous estimate given by (13)In NN3D, a convex combination of input noisy image y and the previous estimate of the clean image is fed into CNN filter followed by NLF, which is 3D collaborative filter. Here k denotes the number of iterations. The NLF, i.e. B3MD requires weaker regularisation, i.e. smaller threshold as a fraction of the noise is already attenuated by CNN. Rate of progress of the iterative procedure is controlled by step parameter Therefore, both threshold and step parameter are positive and monotonically decreases with every iteration. After the estimation of image by CNN, block matching is done in which group of similar blocks is identified. The similar blocks are compiled into a look-up table of group coordinates . Each has the dimension , in which coordinates of mutually similar blocks of size is compiled. \; Each pixel in the image is at least covered by one block of S. Non-local self-similarity is enforced by group-wise processing of based on the look-up table S. Specifically, at each iteration k and for each , a group of size could be filtered by 3D transform-domain shrinkage. Shrinkage operation is performed along the third dimension of the group , i.e. 1D transform of length . The filtered group is given by (14)where is the shrinkage operator. The image estimate is obtained by returning the block estimates from the filtered group to their original locations , where they are aggregated with group-wise weights reciprocal to the energy of the shrinkage factor. 4.2.1 Algorithm of NN3D The algorithm of NN3D (see Fig. 6). Fig. 6Open in figure viewerPowerPoint Algorithm of NN3D 4.3 Deep SCNN for adaptive noise reduction Deep SCNN can reduce the time-varying noise, unlike other CNNs, which are designed for specific noise levels. If the noise level of the training data set is different from that of testing data set, then most of the CNNs which do not have adjustable parameters fail to restore images effectively. In deep SCNN [31], the soft shrinkage activation function is used. Soft shrinkage activation function has a threshold which is adjustable to the noise level and is given by (15)where x is the pixel intensity of the given co-ordinate in the image and τ is the threshold. 4.3.1 Architecture of deep SCNN SCNN architecture is the same as that of DnCNN [30]. It had D no. of layers and l is the identifier of layers. The first layer is the feature extraction layer, which consists of convolution filters of size and activation function . The next layers from is the feature conversion layer which has convolution filters of size , followed by a batch normalisation [50] and activation layer. The last layer is the residual image generation layer, which consists of a convolution layer whose size of the filter is . The loss function of SCNN is given by (16)Here represents the N noisy-clean training image (patch) pair, R represents the residual noise function and Ɵ denotes the learnable parameters. is the soft shrinkage activation function which applies to each element of the feature map , where p is the position in the image and is an identifier of a channel, as follows: (17)where is a threshold of soft shrinkage for the cth channel of feature map given lth layer under the condition that standard deviation of the input image is , where is a coefficient to be optimised by the gradient me