Multi‐scale Cross‐path Concatenation Residual Network for Poisson denoising

IET Image ProcessingVolume 13, Issue 8 p. 1295-1303 Research ArticleFree Access Multi-scale Cross-path Concatenation Residual Network for Poisson denoising Yueming Su, Yueming Su School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorQiusheng Lian, Corresponding Author Qiusheng Lian lianqs@ysu.edu.cn School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorXiaohua Zhang, Xiaohua Zhang School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorBaoshun Shi, Baoshun Shi orcid.org/0000-0003-4643-3816 School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorXiaoyu Fan, Xiaoyu Fan School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this author Yueming Su, Yueming Su School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorQiusheng Lian, Corresponding Author Qiusheng Lian lianqs@ysu.edu.cn School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorXiaohua Zhang, Xiaohua Zhang School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorBaoshun Shi, Baoshun Shi orcid.org/0000-0003-4643-3816 School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this authorXiaoyu Fan, Xiaoyu Fan School of Information Science and Engineering, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of China Hebei Key Laboratory of Information Transmission and Signal Processing, Yanshan University, Qinhuang dao, 066004 Hebei province, People's Republic of ChinaSearch for more papers by this author First published: 21 May 2019 https://doi.org/10.1049/iet-ipr.2018.5941Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The signal degradation due to the Poisson noise is a common problem in the low-light imaging field. Recently, deep learning employing the convolution neural network for image denoising has drawn considerable attention owing to its favourable denoising performance. On the basis of the fact that the reconstruction of corrupted pixels can be facilitated by the context information in image denoising, the authors propose a deep multi-scale cross-path concatenation residual network (MC2RNet) which incorporates cross-path concatenation modules for Poisson denoising. Multiple paths are achieved by the cross-path concatenation operation and the skip connection. As a consequence, multi-scale context representations of images under different receptive fields can be learnt by MC2RNet. With the residual learning strategy, MC2RNet learns the residual between the noisy image and the latent clean image rather than the direct mapping to facilitate model training. Specially, unlike existing discriminative Poisson denoising algorithms that train a model only for the specific noise level, they aim to train a single model for handling Poisson noise with different levels, i.e. blind Poisson denoising. Quantitative experiments demonstrate that the proposed model is superior over the state-of-the-art Poisson denoising approaches in terms of peak signal-to-noise ratio and visual effect. 1 Introduction Poisson noise is a typical scatter noise, which follows the Poisson distribution. The removal of Poisson noise is of fundamental importance in many applications such as low-light photography, astronomy and medical imaging [1, 2]. Since noise variance is equivalent to the signal intensity value, Poisson noise is signal dependent. Such a property determines that Poisson denoising is different from the Gaussian denoising. Thus, Poisson denoising poses challenges in real applications, especially in low-count cases, generally resulting in low-quality recovery. In recent years, there have been many approaches to address the Poisson denoising problem [3-9]. Major contributions focus on using the variance stability transformation (VST) [10, 11]. The VST can convert Poisson noise to the approximated Gaussian noise, whose strength is independent on its signal intensity. Then, the approximated Gaussian noise can be removed by the well known block-matching and 3D (BM3D) Gaussian denoiser [12], and the estimated image can be obtained by using an inverse VST to the denoised image [5, 6]. However, these VST-based methods are effective only when the intensity of measured pixels is relatively high. Put another way, the recover quality employing VST is poor in the low-light regime, as noted in recent works [8]. To overcome the deficiency of VST, some impressive attempts without VST have been investigated, which eliminate the noise by analysing the statistics of Poisson noise including sparse prior-based methods and the trainable non-linear reaction diffusion-based approach [7-9]. Although sparse prior-based methods have shown desirable denoising performance, a complex iterative optimisation problem can be generally involved, resulting in the denoising procedure time-consuming and the applicability restricted in practise. To overcome the drawbacks of sparse prior-based methods, the trainable non-linear reaction diffusion-based approach was developed to learn the diffusion process through the training strategies. However, there are many unconventional influence functions in the trained diffusion model, which can result in an unstable diffusion process. Another non-negligible issue is that existing Poisson denoising models are usually optimised under a certain noise level. From a practical perspective, a single model is urgently needed to handle different levels of degradations. This goal can be accomplished by increasing model complexity to a certain extent. Recently, the convolutional neural network (CNN) enjoys great popularity attributed to its strong learning and representation ability. In the deep network structure, CNN is effective and flexible in extracting image features and has the powerful non-linear analytical ability in various vision tasks. In addition, the dilated convolution which is the variant of CNN can enlarge the receptive field without sacrificing the computational cost. Moreover, the graphics processing unit (GPU) parallel computation can speed up the feed-forward inference of CNN. Given the superiority of CNN, it is widely used to tackle various image restoration tasks, e.g. single-image super-resolution [13], Joint Photographic Experts Group deblocking [14] and image denoising [15]. As the classical image restoration task, the image denoising employing CNN has produced a favourable performance. For example, Tai et al. [16] proposed a persistent memory network relying on CNN for image denoising. With the depth increasing, the network can tackle the long-term dependency problem to ensure that former layers have an influence on subsequent layers. In [15], Zhang et al. constructed a single-path feed-forward network to learn an end-to-end mapping by using CNN for Gaussian denoising. Specifically, the batch normalisation [17] and residual learning principle [18] were utilised to accelerate the training and promote the denoising performance. More recently, a fast and flexible CNN-based denoising network termed as FFDNet was proposed in [19], which established a trade-off between the denoising performance and inference speed by employing the noise level map as the input. More elaborate architectures can be found in [20-22]. Motivated by the aforementioned well-designed architectures, we investigate a deep CNN-based network combining the multi-scale method and residual learning principle [18, 23]. Instead of optimising a model with specific image priors, we regard the Poisson denoising task such as a plain supervised learning process, which is directly learning the Poisson noise from noisy images by using a feed-forward CNN-based network. Ultimately, a modularised multi-scale cross-path concatenation residual network (MC2RNet) that contains recursive cross-path concatenation modules (C2Ms) is investigated. Multi-scale image priors can be extracted due to multi-scale paths. The experimental results confirm that the proposed MC2RNet can produce favourable denoising performance. Our contributions are summarised as follows: A deep multi-scale trainable CNN-based residual network containing recursive C2Ms is proposed for Poisson denoising. The recursive modules ensure the information flows through multiple paths, which can enrich the receptive field and increase the flexibility of the network for exploring new features to further boost the denoising performance. The C2M is designed as the key component of the network. Therein, the cross-path concatenation and skip connection are integrated into two major paths to ensure multi-scale information fusion and reuse features from prior states, whilst alleviating the optimisation difficulty during the gradient back-propagation phase. Unlike the off-the-shelf discriminative Poisson denoisers that train a model only with a specific noise level, we train a single MC2RNet model with a range of noise levels to address blind Poisson denoising. Compared to the up-to-date Poisson denoising techniques, our proposed model is more practical in real applications and achieves desirable performance at different noise levels. The remainder of the work is summarised as follows. Section 2 presents some preliminaries including the statistics property of Poisson noise and a brief review of relevant works on the proposed method. In Section 3, we propose the MC2RNet for Poisson denoising and formulate the network in detail. Comprehensive experiments are presented to evaluate the performance of the proposed model in Section 4. Subsequently, the concluding remarks are given in Section 5. 2 Preliminaries 2.1 Poisson noise Let be the Poisson noisy image from an electronic device. Each observed in is regarded as a Poisson distributed independent random variable, whose variance and mean are the underlying estimated intensity value. Definitely, the discrete Poisson distribution of each is denoted by –Poisson (), that is (1)where is the Kronecker delta function. Owing to that the strength of Poisson noise is proportional to the image intensity , Poisson noise is dependent on the signal. The goal of Poisson denoising is to recover the latent clean image from the observed In general, the noise level is measured using the maximum pixel value (peak value) in [9]. Note that the standard deviation of the Poisson noise equals , and it is reasonable that the lower image intensity value yields the stronger noise. 2.2 Representative CNN architecture With the remarkable self-learning ability, CNN-based architectures become popular to handle high/low-level vision issues through considerable data. Here, we present some representative CNN architectures relevant to the proposed MC2RNet. To increase the learning capacity and further push the representation learning, Chen et al. [24] proposed the dual path network dependent on DenseNet [25] and ResNet [18] for image classification. This dual path network can ensure the feature reuse and new features extraction enjoying the superiority of dual path topologies. Inspired by the persistency of human thoughts, a persistent memory network with cascaded memory blocks was introduced in [16]. Each memory block consists of the recursive unit and the gate unit. Owing to this elaborate architecture, persistent memory can be explicitly mined by the adaptive learning process. On the basis of multi-scale principle, Zhang et al. [26] designed an effective CNN denoiser to learn sophisticated deep priors for image restoration. Owing to the expansion capacity of the dilated convolution for the receptive field, the CNN denoiser made a trade-off between the computational burden and image restoration performance. In [27], Zhang et al. proposed a deep residual CNN-based network called image denoising model (IDCNN) for image denoising. Specially, the IDCNN predicted latent clear images by separating the noise from contaminated images. Meanwhile, a single Gaussian denoising model was trained to handle different corruptions (noise levels), which highlighted the flexibility in the practical application. Notably, Remez et al. [28] designed a simple yet powerful CNN architecture for Poisson and Gaussian image denoising. The nature of the denoising procedure can be exploited attributed to this architecture, in which local noise statistics can be explored by shallow layers, while the textures and edges are recovered by deeper layers. 2.3 Poisson denoising In Poisson denoising regime, the Poisson noise is highly correlated with the ground-truth image, which makes this problem challenging. Extensive methods have been exploited in recent years. As stated previously, some methods tackle the problem by using VST which converts the Poisson noise to the approximate Gaussian noise. In this regard, a well-refined BM3D-based algorithm combining with VST operation was presented to address the Poisson denoising issue [6]. Although it obtained significant performance as well as the high efficiency, some artefacts can be produced in the low-light regime. To further improve the denoising performance and suppress the artefacts of recovered images at very low counts, Azzari and Foi [5] proposed an iterative algorithm called I + VST + BM3D at the expense of nearly twice computational cost of the non-iterative scheme, and the I + VST + BM3D showed significantly better performance and outperformed many costlier alternatives. It is generally agreed that the methods employing VST are less realistic for cases of low counts [7]. Thus, several methods aiming to analyse the Poisson noise statistics have been investigated. For example, Salmon et al. [7] developed a patch-based algorithm to address the denoising problem, which combined an adaptation of the principal component analysis and the sparsity-regularised optimisation method into a non-local framework. Relying on the dictionary learning and the Poisson statistical model in [7], Giryes and Elad [8] proposed a sparsity-inspired Poisson denoising algorithm that used the boost-strapping with a stopping criterion. Although the aforementioned methods have shown promising results, they mainly focus on obtaining utmost performance without considering the computation efficiency. A prevailing trend in image denoising is to leverage deep neural networks to learn a non-linear mapping from the noisy image to its clean version. In this case, novel models have been exploited on the subject of deep neural networks in Poisson denoising. In [9], a discriminative learning-based trained reaction diffusion models for Poisson denoising (TRDPD) algorithm was proposed to solve the Poisson denoising problem, and it can be regarded as a feed-forward network inferred by the proximal gradient descent method. Specially, the model can deliver significant improvements in both denoising performance and computational efficiency. Although the TRDPD achieved desirable performance, it is restricted to capture global features of image structures [15]. Meanwhile, there are many handcrafted parameters that can be involved during the stage-wise greedy training and the joint training stage [9]. In the work of [3], Remez et.al proposed a CNN-based network called DenoiseNet which has the resemblance to the residual network. The DenoiseNet can implicitly remove the Poisson noise from the noisy image by feed-forward CNN and has presented a superior performance in Poisson denoising task. While the deep neural network-based models have shown some promise, different denoising models must be trained for different levels of noises, which can inevitably increase the computational expense on account of highly redundant model parameters, and limit the utility of the technology. Thus, how to train a model for the blind Poisson denoising is an issue that needs deeper study. 3 Proposed CNN-based MC2RNet 3.1 Network architecture In Fig. 1, the CNN-based MC2RNet structure is presented, which is constructed with stacked C2Ms. Apart from the last layer of the network, the rectified linear units (ReLUs) are used after each convolution operation. Owing to the effectiveness of residual learning in designing deep models [20], we adopt the residual design in the proposed network. Given the noisy image patch , a non-linear residual mapping can be learnt by the residual learning, where denotes all the trainable parameters in MC2RNet. The estimated output can be obtained by the following equation: (2) Fig. 1Open in figure viewerPowerPoint Architecture of MC2RNet for Poisson denoising. Conv and 2D Conv represent traditional convolution and dilated convolution whose dilation factor is s = 2 The proposed MC2RNet is composed of three types of units, i.e. feature extraction unit, feature mapping unit and feature reconstruction unit. Specially, a convolutional layer is utilised in the feature extraction unit to extract features from the noisy image patch, that is (3)where denotes the feature extraction function and is the output of the feature extraction unit. Assume that the non-linear mapping unit contains n stacked C2Ms, we can obtain (4)where denotes the ith major path, wherein i = 1 is the path employing the traditional convolution kernel and i = 2 is the other path employing the dilated convolution with dilation factor s = 2; and represent the feature mapping function and the output of the ith major path in the C2M. In the final layer, we employ the convolution operations to re-establish the residual which is expressed as the following form: (5)where and denote feature reconstruction functions in the first and second major paths, respectively. On the basis of (5), (2) can be rewritten as (6) 3.2 Cross-path concatenation module In this section, we elaborately present the details of C2M. As the core module of the MC2RNet, the C2M is presented in the dashed box of Fig. 1. Each C2M consists of a cross-path concatenation operation, two skip connections and three parallel convolution structures. Relying on the fact that the filter with the same size can only extract the same scale features, we utilise the traditional convolution which is highly efficient in the deep network and the dilated convolution with the dilation factor s = 2 in two paths, correspondingly. Note that the dilated convolution with the dilation factor s can be regarded as a sparse convolution operator of size , in which only nine values of fixed positions are non-zeros. To fuse multi-scale features and avoid mid-/high-frequency information attenuation, the cross-path concatenation and the skip connections are adopted in each module. Each C2M contains a cross-path concatenation operation and six convolution operators, and every convolution operation outputs 64 feature maps. Let denote the non-linear mapping function ReLU. Here, for succinctness, we introduce to represent the input of each C2M. The output of every convolutional layer in the nth C2M can be formulated as (7)where refers to the output of each convolutional layer; and denote the weight and bias terms in each convolutional layer, respectively; is the two-dimensional (2D) convolution operator; and and denote the ith major path as mentioned above and the jth layer in the C2M, correspondingly. To avoid information attenuation and mitigate the training difficulty, the skip connection is conducted to deliver the information from the previous module to the second non-linear mapping layer. Then, we can obtain (8)where and are skip connection outputs of two different paths in the nth C2M. Specially, a cross-path concatenation operation is applied to concatenate multi-scale features before the third layer. In the third layer, we adopt a convolution operation to extract multi-scale features from previous states, which can avoid the loss of previously acquired information and effectively enable new features exploration. Thus, the output of the nth C2M can be explicitly expressed as follows: (9)Thereon, the recursive C2Ms contribute to the denoising performance owing to the powerful context learning ability of multi-scale convolution topologies with alternating linear filters and non-linear transformation operations. In addition, cross-path concatenation and skip connection are integrated into each C2M to assist MC2RNet in exploiting comprehensive contextual features. 3.3 Analysis of the receptive field diversity It is well known that contextual information can boost the reconstruction of the corrupted features in image denoising [20]. In CNN, increasing the diversity of the receptive field can ensure that the network extracts abundant contextual information. Dilated convolution operator is popularly attributed to the expansion capacity of the receptive field [23, 29]. Without losing the resolution or coverage, the dilated convolution can achieve the exponential expansion of the receptive field. By means of the cross-path concatenation operation and the skip connection operation, multiple paths can be generated by connecting different types of kernels to capture multi-scale features. Both operations can create paths to access features element-wisely from the bottom layer directly to the top layer. It is worth mentioning that cross-path concatenation operation not only enables feature reuse, but also fuses multi-scale features from different paths. The ablation experiment was conducted to demonstrate the effectiveness of the cross-path concatenation. See Section 4.3.2 for details. To quantify the diversity of the receptive field, the feature mapping unit is presented using a different style in Fig. 2. Note that the feature mapping unit determines the number of the path in MC2RNet. As shown, each C2M has five alternative paths for data flowing. By analogy, there are paths for feed-forward inference while extending to the nth C2M. Evidently, the number of path is substantial in MC2RNet. It is worth noting that multiple paths contribute to the multi-scale receptive field of MC2RNet. Hence, multi-scale contextual information can be extracted owing to the abundant receptive field, which can further boost the denoising performance. Fig. 2Open in figure viewerPowerPoint Visual presentation of receptive field diversity in the MC2RNet 3.4 Optimisation procedure In other neural network training phase, the default optimisation method is to minimise the l1 loss or l2 loss. However, the l1 norm is not smooth when the value closes to zero, and the l2 norm is sensitive to the abnormal point. To overcome their deficiency, we select the Huber loss [30] as the loss function. We empirically find that the Huber loss function can make the model more robust and stable. Such loss function is piecewise defined by (10)where a denotes the input value of the loss function. Formally, the loss function of our MC2RNet can be then written as (11)where N denotes the number of training patches. We empirically set the parameter in our experiments. Here, and represent ith noisy image patch and the ground-truth image patch, respectively. The optimal trainable parameters of the network are automatically learnt by Adam optimiser [31] with the mini-batch size of 16. Apart from the learning rate, other hyper-parameters of Adam are set by default. 4 Experiments 4.1 Network training 4.1.1 Training data We exploited 8000 images from the PASCAL VOC dataset [32] which was also used in [3]. During the training phase, the images were randomly selected from the dataset and then cropped into the patches of size . The data augmentation technique was utilised to avoid over-fitting. More specifically, the images were randomly flipped and rotated to patches for training. The training data was preprocessed following the guidelines in [3]. More specifically, the image patches were converted from the red, green and blue colour mode to the red, green and blue (RGB) color spaceto itsformat conversion space (YCbCr) colour mode and the Y channel was selected as the input grey-scale image. Then, the input images were shifted to the range [ − 0.5, 0.5] and corrupted with the Poisson noise. To avoid convolution artefacts in borders of image patches, the outer 20 px of the image patch were symmetrically cropped and only the central part was retained for parameter optimisation in the training phase. 4.1.2 Training details There were two kinds of trained models for Poisson denoising. One was trained with specific noise levels and six noise levels were considered in our experiment, i.e. peak = 1, 2, 3, 4, 8 and 30, and we referred to this model as MC2RNet-S. The other was a single model which was trained with a range of noise levels, i.e. , and we referred to this single model as MC2RNet-B for blind Poisson denoising. All models were implemented using Tensorflow deep learning framework. The training was performed on a graphic workstation with the GPU NVIDIA Tesla K80 and the central processing unit Inter Xeon (R) E5–2650 v4 at 2.20 GHz. The weights were initialised using the method in [33], and the biases initialised to zero. It took about 50 h to train the MC2RNet-B model and each MC2RNet-S model. The learning rate was set as at the first five epochs and then fixed to between the fifth epoch and the eighth epoch. In the last epoch, it was fixed to . Fig. 3a presents the valid loss curves of four specific models on the validation set with 16 images sampled from the PASCAL VOC dataset, and the magnified version of the loss curve at peak = 8 as shown in Fig. 3b. Evidently, the loss has a sharp decline when the learning rate reduces to at the fifth epoch. Then, there is a slight decay when encountering the next decay of the learning rate. The loss tends to be stable after eight epochs, which indicates that the training has converged. Fig. 3Open in figure viewerPowerPoint Valid loss curves of four specific models are achieved by the designed MC2RNet at different noise levels (a) Valid loss curves with respect to epochs at peaks = 1, 2, 4 and 8, (b) Magnified version with respect to epochs at peak = 8 4.2 Baseline algorithms and testing data In this paper, we compared our proposed models with three representative Poisson denoising algorithms: DenoiseNet [3], I + VST + BM3D [5] and TRDPD [9]. The corresponding codes obtained from the homepage of the authors. Two popular benchmark datasets were used for evaluation. One is the standard dataset named Set10 containing ten common images as shown in Fig. 4. The other contains 68 images from Berkeley segmentation dataset (BSD68) [34] selected by [35]. In the test phase, the test data was symmetrically padded with 20 px, and then symmetrically cropped to their original size after output to avoid the effect of convolution artefacts. Fig. 4Open in figure viewerPowerPoint Standard test images in dataset Set10 4.3 Quantitative analysis In this section, we conducted three experiments to determine the number of C2M for obtaining the optimal denoising quality, evaluate the effectiveness of the cross-path concatenation and verify the generalisation ability of the pr