High‐density impulse noise detection and removal using deep convolutional neural network with particle swarm optimisation

IET Image ProcessingVolume 13, Issue 2 p. 365-374 Research ArticleFree Access High-density impulse noise detection and removal using deep convolutional neural network with particle swarm optimisation Hui Ying Khaw, Hui Ying Khaw orcid.org/0000-0002-7186-7686 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorFoo Chong Soon, Foo Chong Soon orcid.org/0000-0002-7593-5308 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorJoon Huang Chuah, Corresponding Author Joon Huang Chuah jhchuah@um.edu.my orcid.org/0000-0001-9058-3497 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorChee-Onn Chow, Chee-Onn Chow Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this author Hui Ying Khaw, Hui Ying Khaw orcid.org/0000-0002-7186-7686 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorFoo Chong Soon, Foo Chong Soon orcid.org/0000-0002-7593-5308 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorJoon Huang Chuah, Corresponding Author Joon Huang Chuah jhchuah@um.edu.my orcid.org/0000-0001-9058-3497 Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this authorChee-Onn Chow, Chee-Onn Chow Department of Electrical Engineering, Faculty of Engineering, University of Malaya, 50603 Kuala Lumpur, MalaysiaSearch for more papers by this author First published: 01 February 2019 https://doi.org/10.1049/iet-ipr.2018.5776Citations: 7AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Most of the impulse denoisers are either median filter-based or fuzzy filter-based, which can only perform well in low noise conditions. This study presents an efficient convolutional neural network (CNN) with particle swarm optimisation (PSO) model for high-density impulse noise removal. The proposed high-density impulse noise detection and removal model mainly consists of two parts: the impulse noise removal and impulse noisy pixel detection for restoration. The authors' model initially leverages the powerful ability of deep CNN architecture to separate noise from the noisy image, then adopts PSO to pinpoint the most optimised threshold values for detecting impulse noisy pixels. An ensemble of these algorithms is an intelligent and adaptive solution, producing a clean output while preserving significant pixel information. Targeting to solve high-density impulse noise problems, the authors have trained their model with a massive collection of natural images and 14 standard testing images are used for validation purposes. In order to validate the robustness of the proposed method, different levels of high-density impulse noise are considered. Based on the final denoised images, their model has proven its reliability, in terms of both visual quality and quantitative evaluation, on greyscale and colour images. 1 Introduction Digital images may be seriously contaminated by impulse noise during acquisition and transmission. This randomly distributed noise is the root of loss of image details. Image denoising is a fundamental role to recover the original image by making an estimation to preserve image information. It is a pre-processing step before any subsequent image processing operations such as segmentation, analysis or restoration, and has been widely studied for nearly half a century, but still remains as an unsolved problem in digital image processing [1]. The non-linear median filter is one of the most notable impulse denoising filters due to its fast computation [2]. Since then, many filters have been proposed such as switching and decision-based median filters [3, 4]. Despite their simplicity, these methods would simply filter all pixels even if they are noise-free, hence often blur out the image. To overcome this, some impulse detectors [5, 6] are designed to only alter the noise candidates while still keeping the clean pixels unchanged to boost denoising performance. Then, various fuzzy median filters [7-10] exploiting fuzzy reasoning are proposed to correct impulse pixels. Some other remarkable model proposed included [11] which presents particle swarm optimisation (PSO) to optimise the parameters necessitated by the fuzzy-based filter while [12] proposed a low computational complexity interpolation-based impulse noise removal. Bai and Tan [13] proposed a two-stage denoiser for high-density impulse noise images based on local mean and variance detector (LMVD) with Newton–Thiele filter (NTF), which can automatically identify the corrupted images before further processing. Despite presenting a solution, most of the aforementioned methods have obvious drawbacks. Firstly, most of these denoising techniques are customised only for low-density noise filtering purpose, resulting in poor denoising quality on high-density impulse noise removal. Secondly, some methods rely heavily on manual tuning of various parameters. Hence, these adjustments are not only tedious, but also requiring ad-hoc tricks to get a promising result. These parameter-dependent algorithms generally are not practical in real-time application. Therefore, we are motivated in finding a more robust solution for better noise detection and removal approach. In the past decades, several intelligent or automated systems have been designed for various applications. Recently, a convolutional neural network (CNN) stands out from other techniques thanks to their excellent ability to analyse visual imagery. This hierarchical deep learning network has been widely used in breast cancer diagnosis [14], vehicle model recognition [15], gender and age prediction [16], noise types recognition [17, 18] and more. It is also learnt that CNN has been utilised in restoring noisy images, showing its potential in solving the denoising problem. For example, the performance of a plain multi-layer perceptron (MLP) [19] is on par with the three-dimensional filtering (BM3D) denoising algorithm if the capacity, patch size and training set are sufficiently large. A deep convolutional architecture that learns an end-to-end mapping from a noisy image to a corresponding denoised image is proposed in [20]. More notably, a very deep denoising CNN (DnCNN) [21] based on residual learning strategy is able to tackle multiple tasks, i.e. Gaussian denoising, single image super-resolution, and JPEG image deblocking. In view of this successfully designed Gaussian denoiser model, it is of our interest to explore the unrealised ability of CNN in impulse noise removal application. On the other hand, noisy pixel detection has been commonly implemented to improve median filter so that only the detected noisy pixel will be filtered. PSO-based methods are heuristic techniques which mimic the biological phenomena such as information sharing and social behaviour in a fish school or a bird flock [22]. Inspired by this, we adopt PSO to perform high-density noisy pixel detection, which is still an open area in finding the best approach. By doing this, we are putting together the strength of CNN in dealing high-intensity image noise and the yet to be explored capability of PSO in noisy pixel detection for an end-to-end high-density impulse denoising model. The contributions of the proposed high-density impulse noise detection and removal (HINDR) model are summarised as follows: To the best of our knowledge, we are the first to implement deep CNN model to tackle the challenging high-density impulse noise problems, in contrast to the commonly found traditional median filters and morphological filters. To execute impulse noise pixels locating after the denosing procedure using PSO so as to fit the system to the characteristics of impulse noises that only affect certain pixels in accordance to its noise level. The integration of median filter as a final cross check procedure with the use of PSO algorithm on CNN to present a complete intelligent impulse noise detection and denoising model. The remaining of this paper is organised as follows. The architecture of an intelligent impulse noise detection and denoising model using deep CNN and PSO, integrating with a median filter (CNN-PSO-Med) is elaborated in Section 2. In Section 3, the dataset description, multiple experiments and their analysis are presented. Finally, the conclusion of this project is drawn in Section 4. 2 Deep CNN with PSO for a high-density impulse noise detection and removal model The undiscovered potential of deep CNN and PSO in removing impulse noise on images is explored in this work. A complete HINDR model is proposed in Fig. 1. Fig. 1Open in figure viewerPowerPoint Architecture of a HINDR model 2.1 Impulse noise model When an originally clean image, X is degraded by impulse noise, V, the noise is dispersed uniformly and randomly on a specific amount of image pixels. A noisy image contaminated by impulse noise is described as . Let the 8-bit noisy observatory image be Y, at any pixel location (i, j), the corrupted pixels can be digitised into two extreme values, either salt or pepper while the clean pixels remain as the original image's grey level. The noisy observatory distribution can be described as follows: (1)where is the grey level of the noisy image, corrupted by noise density or l percentage of pixels degraded by noise at vmin = 0 and vmax = 255 and is the clean grey level of the original image. 2.2 Proposed CNN-PSO model for impulse denoising and noisy pixel detection A novel hybrid model of CNN algorithm combined with a PSO technique for an efficient impulse denoising and noisy pixel detection is designed. The first aspect of the model emphasises on learning the latent clean image directly from the model and subsequently obtaining the noise distribution to detect noisy pixels. The very deep CNN architecture for impulse denoising task is based on the Gaussian denoiser [21] and is integrated with a PSO technique for its noisy pixel detection. In this section, we are to briefly explain this end-to-end impulse noise detection and denoising model in different point of views: (i) network architecture of the CNN, (ii) learning objective of the network and (iii) PSO for searching and optimising the best threshold values and finally (iv) median filter for correcting false detection. 2.2.1 Deep CNN architecture The substantial research effort is recently being directed towards designing a deep CNN architecture for image recognition and denoising tasks. Among the most popular deep architectures for recognition problems are AlexNet [23], VGGNet [24] and so on. AlexNet's results propose that deep model is essential to boost recognition accuracy. Similarly, VGGNet pushes architecture depth by adding convolutional layers while keeping filters' size small. The DnCNN [21], a remarkable very deep CNN, verified the model's ability to exploit image characteristic for an effective Gaussian denoising and image deblocking. Inspired by this, our work is structured towards the integration of the PSO technique with this very deep CNN. The deep CNN model is composed of three components, which are the convolutional layers (Conv), batch normalisation (BN) and rectified linear units (ReLU), as shown in Fig. 1. A. Convolutional Layers: Theoretically, the convolutional layers that perform convolution are the primary components in a CNN model. A 2D-kernel convolves with an input image, resulting in an output feature map. Mathematically, this operation can be explained as (2)where denotes the 2D convolution operation between input and square filter with size k by k. For all layers ℓ, filter size k of 3 × 3 is chosen. Through the repeating layers of convolution, a higher level of features is yielded to iteratively separate noise from the noisy images. Ultimately, the output image size should be kept the same as the input image. Therefore, zero padding is implemented to pad zero pixel values around the boundary of input patches without contaminating the output pixels. To speed up the convergence rate, the ReLU is employed as the non-linear activation function as computed by (3) B. Batch normalisation: After multiple convolution layers, the feature maps usually increase exponentially, resulting in a bulky model. One of the most commonly found practices in a traditional CNN model is to introduce pooling layer to downsize the spatial size of feature maps. However, pooling layer will inevitably remove the pixels that carry significant noise information which could possibly affect the denoising performance. Hence, in order to replace pooling operation, batch normalisation is a more logical approach to reduce training time, batch normalisation is placed alternatively after each convolutional layers to normalise the output and alleviate the internal covariate shift. 2.2.2 Network objective of learning clean image Given a noisy observation, , the objective of our HINDR model is to learn a mapping function as latent clean image prediction and the objective function is indicated as (4)where is the trainable CNN parameter, and is the N clean training image. It is worth noting that, because this model alters all pixels regardless of its initial state, the values of clean image pixels after denoising have been affected. To overcome this, the dependence of PSO upon originally clean pixels correction becomes very necessary. 2.2.3 Determining threshold for impulse noise detection using PSO In contrast to another commonly found Gaussian noise which has all the image pixels degraded, one characteristic of impulse noise is that only a certain number of pixels are corrupted. Depending on the noise density, the affected pixels can be quantised into two extreme values: '0' (pepper) indicating black pixels and '255' (salt) the white pixels. The unaffected clean pixels shall take their original pixel value. One shortcoming of CNN-based denoising approach is that, willingly or not, it alters all image pixels. In the aforementioned traditional filters, some efforts have been put in to design an impulse pixel detection prior to applying filtering techniques to keep the clean pixels unchanged. Similarly, to boost the overall denoising performance, we employ an impulse noise detection step using PSO as which has been shown in the PSO part in Fig. 1. Different from these methods, our detection technique is not compulsory to be seen as the very first step. This is because we first take advantage of the deep CNN model's ability to learn the denoised image, and hence the estimated noise distribution, can be easily obtained from (5)where the grey level of Y is explained in (1). Now, explains a denoised image that assumes every pixel is contaminated by salt-and-pepper noise since the model automatically alters all pixels based on its cognitive learning. Based on (5), we do know that theoretically a clean pixel, should take the original value of hence yielding a zero value , i.e. . In the CNN model that all will be changed, however, the originally clean pixel at positions has its very close to zero, but not exactly at zero. In contrast, the pixels that are further from zero value while closer to the two intervals [–255, 255] are originally noisy. This is because, after the CNN denoising step, the denoised pixel value will be corrected, making it has a bigger difference from its original '0' or '255' value . Alluding to the objective that we want to only adopt the denoised result on noisy pixels while keeping the initially clean pixels unchanged, so as to boost up the overall denoising performance. The complication now lies in accurately determining the boundaries that separate the original clean pixels from the initially noisy ones. This task is the impulse noise pixel detection. This can be seen as an extra optimisation process where the thresholds are possible to be searched in order to achieve the best final peak signal-to-noise ratio (PSNR). Therefore, PSO is adopted to find the threshold boundaries so that clean pixel can then be replaced by their initial pixel values. Eventually, the final HINDR model can exhibit better denoising property. The optimisation algorithm is a technique to achieve maximum benefit with minimum cost and has been extended over to enhance the training phase of deep neural network especially when the traditional network is not powerful enough to find the most favourable solution. Mathematically, this optimisation process is to search for the values of D parameters which optimise a parameterised function f [25] given by (6)Our objective here is to maximise function f, i.e. PSNR of the denoised image, representing the overall denoising performance by finding the most optimised threshold parameters. To do this, we make use of PSO to find a set of best solutions. The solution is among a population, or more commonly known as a swarm in PSO, and consists of a cumulus of particles with candidate solutions (7)A particle is referred to as i and its position is represented by a vector (8)where D is the number of parameters to be optimised within the searching space. Each particle is moving at its own velocity as depicted (9)At each iteration, its new position with respect to its latest velocity , is described as (10)where the is the subsequent iteration of the . The algorithm captures two best positions in every iteration: the personal best position obtained by each individual particle recorded as and the overall best position within the swarm called the global best defined by . The new velocity of each particle is updated as (11)where is the parameter to be optimised in the searching space, is the inertia weight usually vary from 1 to 0 in order to ensure the particles follow previous flow direction, real-valued and are the acceleration constants that direct the particle steps towards its best personal and global best direction and are typically in the range , and and are the random numbers generated in [0, 1]. The last two terms in (11) are called cognitive component and social component, respectively. As shown in these equations, velocity is a very important component that influences the particle position in finding the best solutions as an output after every iteration. Hence, a velocity clamping technique is applied to make sure the velocity of each particle at each iteration does not go beyond the limits. To execute this, a maximum velocity threshold is imposed (12)where is the velocity component of the particle along the direction at the direction; is the maximum velocity allowed and should be defined as (13)where and are the lower and upper boundaries of the position of the particle usually initialised to prevent them from traveling out of the boundaries. Accordingly, the initial position of a particle is described as (14)where U indicates a uniform distribution of N particles within and . The final outcomes of PSO are two threshold values to group the image pixels into initially clean or initially noisy, functioning as a noisy pixel locating step. 2.2.4 Median filter The very last step of our HINDR model is a simple traditional median filter to cross-check the output image after the CNN-PSO procedure. The algorithm is simple, a sliding window is passed through every pixel, and if a pixel is found to take '0' or '255' values, it will take the median value of its 8 neighbouring pixels as a replacement. 3 Experiment 3.1 Dataset descriptions The proposed HINDR model aims to handle all images degraded with high-density impulse noise. For the learning of the model, we utilise a large training set consists of 400 natural images with a variety of features such as human, animal, building, pattern, scenery, silhouette and more. The rich collection of datasets is to enhance the model's learning process in order to reliably perform denoising in all circumstances. Some examples of training images provided by Zhang et al. [21] are shown in Fig. 2. These images are segmented into patches with the size of 40 × 40 pixels for training purpose. For a fair comparison, we used 14 standard testing images that are most commonly utilised in the literature on image processing. They are Lena, Cameraman, House, and more as shown in Fig. 3. The testing images are in dimensions of 256 × 256 pixels and 512 × 512 pixels. To show the effectiveness of our model, training and testing images are non-overlapping. As we are targeting to solve high-density impulse noise, the levels being considered are 50, 60, 70, 80 and 90%. Each training and testing images, in grayscale format, are added with a certain level of impulse noise for learning and validation purposes. Fig. 4 shows the images 'Lena' that are highly degraded with different severity. The higher density the impulse noise is added, the image details and information are more severely reduced hence the poorer image quality especially on the 80 and 90% impulse noisy images. Fig. 2Open in figure viewerPowerPoint Some natural images for training purpose Fig. 3Open in figure viewerPowerPoint 14 standard test images used in this work: Lena, boats, Baboon, bridge, Barbara, pirate, couple, cameraman, house, pepper, starfish, Monarch, plane and parrot Fig. 4Open in figure viewerPowerPoint Original 'Lena' image and degraded 'Lena' images (a) Original clean 'Lena', (b) Degraded 'Lena' with 50% impulse noise, (c) Degraded 'Lena' with 60% impulse noise, (d) Degraded 'Lena' with 70% impulse noise, (e) Degraded 'Lena' with 80% impulse noise, (f) Degraded 'Lena' with 90% impulse noise 3.2 Training of HINDR model 3.2.1 Training of CNN model In each CNN model training for a specific noise level, we use Stochastic Gradient Descent (SGD) solver with a learning rate of 0.01, weight decay of 0.001, a momentum of 0.9, minibatch size of 128 for 25 epochs. The training time needed for 25 epochs is 16 h and 15 min. The training is carried out on a desktop in MATLAB (R2015b) with Intel Core i5 central processing unit and 12-GB random access memory and Graphic Processing Units (GPU) GeForce GTX 1050Ti. 3.2.2 PSO for impulse noisy pixel detection The algorithm of PSO for impulse noisy pixel detection has been explained in Fig. 1. The basic idea is straightforward if a pixel falls beyond the two threshold intervals, it is marked as noisy, and its pixel will take the value given by CNN denoising. In a different situation where a pixel is within the interval, it is reckoned as clean which shall not be changed by the CNN-denoised value but replaced by its original image value. Now we would like to take the optimisation ability of PSO in searching the two threshold values that will yield the best denoising performance. The goal of PSO in obtaining the optimised threshold values T1 and T2 can be visualised in Fig. 5. For the ease of categorising, we have converted the range of of [–255, 255] to [–1, 1]. The ultimate thresholds output of PSO should classify each pixel into 'originally clean' or 'originally noisy'. To initialise the search for these thresholds, a wide range of search space boundaries, and are defined. We then input the noise patterns of standard testing images degraded by the different severity of impulse noise levels to move the searching and training processes forward. The estimated noise distribution, is the pixel-by-pixel subtraction of CNN denoised image, from the highly degraded impulse noisy image, Y as visually explained in Fig. 6. The most optimised threshold values obtained that will give an overall best performance for all images at different noise levels are and . This set of global PSO values produces satisfactory results for pixel detection and hence the subsequent restoration regardless of noise levels and images. Therefore, this noisy pixel detection is called blind detection, because there is unnecessary to find a set of threshold values for an image or a specific noise level. Fig. 5Open in figure viewerPowerPoint Threshold values, T1 and T2 to be trained by PSO that takes V′ as an input in the HINDR model Fig. 6Open in figure viewerPowerPoint Cameraman's (a) Noise distribution, obtained from subtraction between, (b) 90% degraded noisy image, and, (c) CNN denoised image, that has been explained in (5) 4 Results and analysis There are two aspects of performance worth discussing in this work, which is the overall performance of the CNN-PSO-Med algorithm and the strength of PSO in finding the best threshold intervals. These experiments are to show that the algorithm which has incorporated various techniques are able to maximise the strength of each component in performing impulse denoising. PSNR is used to represent the image quality of the denoised images in all experiments. The results of the experiments are shown next. 4.1 Performance of high-density impulse noise detection and removal model 4.1.1 Comparisons of impulse denoising performances with other state-of-the-art methods The proposed HINDR model is the first CNN-PSO-based solution for high-density impulse denoising. To prove its efficiency, we have first compared our work with other renowned denoising techniques in terms of PSNR representation and visual quality. In this session, a wide comparison has been made with the other state-of-the-art methods such as adaptive median filter (AMF) [26], decision-based algorithm (DBA) [6], modified decision based unsymmetric trimmed median filter (MDBUTMF) [27], Cloud Model (CM) [28] and NTF [13]. Four images chosen for this denoising comparison are Lena, boats, Baboon and bridge. Table 1 shows the PSNR comparison for high-density noises of 50, 60, 70, 80 and 90%. Note that in our work, there are three levels of work presented, firstly, the use of CNN method which only performs image denoising, secondly the CNN-PSO that takes care of both noisy pixel detection and image denoising and lastly, the CNN-PSO-Med as one complete HINDR model. Table 1 shows that an image under the highly distorted condition will have relatively lower denoised quality, in terms of PSNR, as compared to the same image with low noise density. This is even obvious in most of the state-of-the-art median filter-based methods where the denoising performances are much lower, as these methods are not focusing on high level impulse noise removal. The performance of NTF is better than but not as high as ours. The experimental results explain that the emphasis of our HINDR model is to resolve the high-density impulse noise in image data, from the high denoised PSNR achieved. Table 1. Denoising performances of the three-stage model, namely CNN, CNN-PSO, and CNN-PSO-Med and the comparison to other state-of-the-art methods Filter Images PSNR, dB Noise density 50% 60% 70% 80% 90% AMF Lena 28.31 25.48 21.95 18.23 15.66 boats 27.28 24.49 21.21 17.85 15.47 Baboon 23.02 21.48 19.51 17.02 14.22 bridge 22.74 21.21 19.17 16.62 13.73 DBA Lena 29.32 27.30 25.22 21.69 15.61 boats 28.97 26.91 24.07 21.26 17.12 Baboon 23.95 22.67 21.34 19.77 17.43 bridge 24.34 22.72 20.54 18.39 15.44 MDBUTMF Lena 31.71 29.82 26.10 25.43 24.65 boats 29.80 28.59 26.88 25.65 23.67 Baboon 24.57 23.61 21.69 20.69 20.33 bridge 24.12 22.77 19.92 19.02 18.29 CM Lena 33.39 32.12 30.05 28.83 25.86 boats 30.34 29.18 27.28 26.15 24.13 Baboon 24.89 23.98 22.89 22.06 21.08 bridge 25.46 24.29 23.06 22.14 20.75 NTF Lena 35.03 33.39 31.73 29.51 26.91 boats 32.57 30.93 29.03 27.01 24.66 Baboon 25.43 24.39 23.39 22.45 21.29 bridge 26.34 25.25 24.03 22.75 21.06 CNN Lena 36.83 36.33 34.67 32.46 26.97 boats 33.54 32.72 31.15 29.16 24.74 Baboon 26.55 25.28 23.84 22.42 20.37 bridge 28.40 27.38 26.07 24.66 22.34 CNN-PSO Lena 37.43 36.49 34.78 32.53 27.17 boats 33.85 32.80 31.17 29.16 24.88 Baboon 26.62 25.32 23.86 22.44 20.43 bridge 27.42 27.41 26.09 24.66 22.02 CNN-PSO-Med Lena 37.43 36.49 34.78 32.53 27.18 boats 33.86 32.82 31.20 29.20 24.88 Baboon 26.62 25.32 23.86 22.44 20.43 bridge 28.54 27.42 26.10 24.68 22.34 *Bold values show the best result of each image using particular methods. The PSNR in Table 1 shows that CNN is feasible in carrying out denoising task as it has more excellent performances than other state-of-the art methods in most conditions presented. However, better performance can be achieved when noise detection and denoising are considered by implementing CNN with PSO (CNN-PSO). Incorporating noisy pixel detection is important as a model is only well-suited to the role of impulse denoising when the nature of impulse noise that only degrades particular pixels is taken care of. The unaffected one