DRU‐net: a novel U‐net for biomedical image segmentation

IET Image ProcessingVolume 14, Issue 1 p. 192-200 Research ArticleFree Access DRU-net: a novel U-net for biomedical image segmentation Xuegang Hu, Xuegang Hu School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of China Center for Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of ChinaSearch for more papers by this authorHongguang Yang, Corresponding Author Hongguang Yang yhgstudy@126.com School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of ChinaSearch for more papers by this author Xuegang Hu, Xuegang Hu School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of China Center for Applied Mathematics, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of ChinaSearch for more papers by this authorHongguang Yang, Corresponding Author Hongguang Yang yhgstudy@126.com School of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing, 400065 People's Republic of ChinaSearch for more papers by this author First published: 01 January 2020 https://doi.org/10.1049/iet-ipr.2019.0025Citations: 3AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract With the wide applications of biomedical images in the medical field, the segmentation of biomedical images plays an important role in clinical diagnosis, pathological analysis, and medical intervention. Full convolutional neural networks, especially U-net, have improved the performance of segmentation greatly in recent years. However, due to their regular geometric structure, the standard convolutions that they use are inherently limited in dealing with geometric transformations while biomedical objects have huge variations in shape and size. In this study, the authors propose the DRU-net, which is a novel U-net with deformable encoder and reshaping upsampling convolution decoder, for biomedical image segmentation. First, deformable convolutional networks are applied and improved to enhance the learning ability of the encoder for geometric transformations. Second, a novel upsampling method named reshape upsampling convolution is proposed for better-restoring resolution and fusion features. Furthermore, focal loss is used to address class imbalance and model overwhelmed problems in biomedical image segmentation tasks. Theoretic analysis and experimental results have shown that the proposed algorithm not only reduces the number of parameters of U-Net, but also achieves produces competitive results compared with the state-of-the-art algorithms in terms of various quantitative measures on Drosophila electron microscopy dataset and Warwick-QU dataset. 1 Introduction Biomedical image segmentation is an active area of research in medical analysis, which can support surgical planning, medical diagnosis, and treatment by extracting different tissues, lesions, organs, and biological structures. Normally, the segmentation is done manually by pathologists. This process is not only time-consuming, but also cumbersome. With the increase in the number and type of medical images, the cost of manual segmentation is also increasing. Therefore, automatic segmentation of biomedical images has become a crucial task. However, this task is challenging for various reasons, for example, the complex variations in the structure of the biomedical objects, the low contrast, noise, and other effects caused by various medical imaging modalities and techniques [1]. In past years, pixel-based methods [2, 3] and structure-based methods [4, 5] have made great progress in biomedical image segmentation. These methods apply handcrafted features and prior knowledge to obtain desired results in some simple segmentation tasks. However, the handcrafted design of invariant features and algorithms could be difficult or infeasible when the methods are applied to the objects with complex variations. Recently, convolutional neural networks (CNNs), especially fully CNN (FCN) [6], have been very effective for biomedical image segmentation and require little handcrafted features or prior knowledge. Thus, FCN is considered to be a baseline model for segmentation tasks [6]. The network achieves the purpose of segmentation by performing pixel-wise classification and can be trained in an image-to-image way. This method contains a downsampling path and an upsampling path. The downsampling path consists of convolutions and max-pooling layers for feature extraction. The upsampling path consists of convolutions and deconvolution (transposed convolution) layers for restoring resolution. U-net is developed based on FCN. The model integrates fine detail by connecting the intermediate decoder (downsampling path) and the coder (upsampling path) to improve the segmentation performance [7]. The model won the International Symposium on Biomedical Imaging (ISBI) Cell Tracking Challenge 2015. Since then, a series of biomedical image segmentation networks based on U-net [8-11] have been proposed and achieved promising results. In the segmentation networks based on U-net, the decoder restores resolution gradually, and finally exports the segmentation mask by using deconvolutions [12] or bilinear interpolation [7]. However, bilinear upsampling is not learnable, and zeros have to be inserted before the convolution operation in deconvolutions. None of these methods thus can recover the position information well during the upsampling process. In this paper, we propose a new upsampling method named reshape upsampling convolution (RUC), which can be learnt without filling with zeros. A key challenge in biomedical image segmentation is various poses and sizes of segmentation objects. There are two ways, in general, to overcome those variations. The first way is to use handcrafted features with spatial invariant such as scale-invariant feature transform [13]. However, it is difficult for this way to deal with overly complicated transformations [14]. The second way is to learn the spatial transformations from datasets by deep neural networks. Hence, it can be done from data augmentation and the networks with the ability of learning geometric transformations. Data augmentation is to build datasets with enough desired variations by some geometric transformations such as rotation, flip, zoom, and so on [14]. However, it is very time-consuming because of a need to generate a large number of images during the training. It also cannot deal well with the unknown geometric transformations, which cannot be properly modelled. Spatial transformer network (STN) [15] is the first work, which contains convolution layers and fully connected layers, to learn geometric transformation. It has the advantages of translation, expansion, rotation, perturbation, bending, and other space invariance, and has gotten great performance in small size image classification tasks [15]. STN learns a global parametric transformation such as affine transformation, to warp the feature maps. However, it is difficult for the parameters to be learnt well. The warping operation is also time-consuming. Deformable convolution generates new feature maps by sampling the feature maps locally and densely to accommodate geometric variations in the images [14]. Moreover, it has less computation complexity and can be trained easier than STN. Moreover, it is a common challenge for CNNs that biomedical images often have the problem of an unbalanced distribution of positive and negative samples, and similar samples are also difficult to distinguish. For example, the samples in the target edge region are more difficult to segment than those in the centre region. These two problems will make the loss function converge to a bad position, resulting in the reduction of generalisation ability of the model [16]. For this reason, the weighted cross-entropy is used as the loss function in [17]. A weight factor is added to each class of cross-entropy to balance the loss function. Ronneberger et al. [7] calculate a weight map in advance for each segmented label image to balance the proportion of classes in training and force the network to learn the difficult samples, for example, the boundary between two cells in contact, which causes good results in cell segmentation and cell tracking. However, the calculation of weight graph is very complex, which greatly increases the training time. Focal loss function [16] was originally used in dense target detection tasks to solve the problems of serious imbalance of samples generated by anchor mechanism, excessive contribution of simple samples to loss function, and poor learning of difficult samples. Inspired by the above work, we propose an improved U-Net for automatic segmentation of medical images. This method replaces the convolution in the encoder with deformable convolution based on U-Net network structure and forms deformable encoder to improve the ability of network to learn geometric deformation. At the same time, the decoder adopts a new learnable upsampling method without filling with zeros, namely RUC, whose parameters and computational complexity are less than the existing deconvolution and bilinear interpolation, and an improved U-Net network structure is obtained. Finally, focal loss is applied to realise automatic biomedical segmentation effectively. The main contribution of this paper can be summarised as follows: We present the DRU-net; the proposed method is based on a variant of U-net with deformable encoder and reshaping upsampling convolution decoder for biomedical image segmentation. We not only adapt deformable convolution for the strong ability to learn geometric variations, but also propose a new upsampling method named RUC. Experiments on Drosophila electron microscopy (EM) [18] data and Warwick-QU gland datasets [19] show that the proposed method has better generalisation ability than the existing main methods and improve the segmentation accuracy significantly. The remaining of this paper is organised as follows. Section 2 briefly introduces the related works. In Section 3, our method is described in detail. Experimental results and analysis are shown in Drosophila EM data and Warwick-QU gland datasets in Section 4. Finally, the conclusion is summarised in Section 5. 2 Related work Biomedical image segmentation is an important step for biomedical researches. Early studies of this topic need the prior knowledge of the radiologists and handcrafted features [20-22] to segment objects and then use post-processing methods such as watershed [23, 24] and graph-cut [25] algorithms, to refine performance. However, the methods tend to fail for objects with complex variations and ambiguous boundaries. With the rapid development of deep learning, CNNs have become popular and achieved desired results in different applications such as classification [26], object detection [27], and segmentation [6]. In terms of biomedical image segmentation, Ciresan et al. [28] applied a patch-wise classifier with the deep convolutional network in a sliding-window way to predict the class label of each pixel by providing a patch centred on the pixel itself as input. This method won the 2012 ISBI neuronal structure segmentation challenge. However, it is quite slow because the classifier has to be run for each patch. Long et al. [6] adapted the classification networks into FCNs and transfer the learnt parameters by fine-tuning to the segmentation tasks. It performs segmentation in an image-to-image way and leads to a breakthrough on semantic segmentation. U-net [7] has proposed by Ronneberger et al. and developed from FCN. It applies concatenation method to connect decoder and encoder for capturing multi-context information from multiple feature channels, which is especially suitable for small biomedical objects segmentation. With the development of CNNs, more and more improved U-net models have been proposed. Inspired by the deep residual network (ResNet) [29] which is designed for addressing the degradation problem, Quan et al. [9] applied residual connections to allow a much deeper U-net for more accurate segmentation. DenseUnet [30] includes dense connected blocks and U-net-like connections. There are both advantages of DenseNet [31] and U-net. One of them is that the dense connected blocks can ensure the maximum information flow between the layers and improve feature extraction ability; another one is that the U-net-like connections can learn context information when restoring resolution. A U-net architecture with three-dimensional (3D) convolutions has proposed to address the problem that 2D convolution ignores the volumetric contexts in [10], but 3D convolutions suffer from high computational cost and GPU memory consumption. Dou et al. [30] combined 3D DenseUnet with 2D DenseUnet effectively and proposed a hybrid densely connected U-net for liver and tumour segmentation. Recently, an attention gate (AG) model has proposed for medical imaging that automatically learns to focus on target structures of varying shapes and sizes [32]. AG model can be easily integrated into standard CNN architectures such as the U-Net model with minimal computational overhead while increasing the model sensitivity and prediction accuracy. This enables us to eliminate the necessity of using explicit external tissue/organ localisation modules of cascaded CNNs. Experimental results show that AGs consistently improve the prediction performance of U-Net across different datasets and training sizes while preserving computational efficiency. Another new method, named nnU-net [33], has received wide attention. This model can automatically design and execute network training process and has achieved the most advanced performance on several well known medical segmentation datasets. To the best of our knowledge, however, little methods are focusing on the networks with strong transformation modelling capability and improved the upsampling method in the above segmentation network. The loss function, which can solve model overwhelmed problem, is also scarce in the methods of biomedical image segmentation. This work is the first attempt of solving these problems. 3 Proposed method In this section, we describe the DRU-net with deformable encoder and RUC decoder for biomedical image segmentation. We start this section with the architecture of the DRU-net and then introduce the RUC and deformable convolution of the network in detail. The loss function used for dealing with model overwhelmed is described finally. 3.1 DRU-net In U-net architecture, the encoder and decoder mainly perform feature extraction and resolution recovery, respectively. We replace the standard convolutions in the encoder by deformable convolutions and the upsampling method in the decoder by RUC to produce the DRU-net architecture. The deformable convolution and RUC can be applied not only to U-net, but also to other segmentation network-based FCNs. As shown in Fig. 1, the DRU-net architecture is also an U-shaped structure. The left part of the network is the encoder, which consists of deformable convolution layers and max-pooling layers. The right part is the decoder, which is composed of conventional convolution layers and RUCs. The encoder and decoder are connected by splicing method, and the detail features extracted from the previous convolution layer are fused while the resolution is restored. Unlike the conventional U-Net network, we do padding during convolution, so the convolution operation does not reduce the resolution of the feature map, nor does it need to tailor the feature map of the encoder when splicing. At the same time, the group normalisation (GN) [34] operation is added before the activation function to improve the convergence speed of the model. GN can avoid batch normalisation [35] requiring large batch (size) problem, thereby reducing the memory limit of GPU [34]. Fig. 1Open in figure viewerPowerPoint DRU-net architecture. The purple blocks correspond to the feature maps produced by encoder and the blue blocks correspond to the feature maps produced by the decoder With deformable convolution, our model not only has strong feature learning ability, but also requires fewer convolution cores, which greatly reduce the number of parameters and increase the feasibility of porting the network to mobile devices. Moreover, RUC can directly learn from the feature map to restore the resolution, and the segmentation accuracy is higher. The following experimental results show our proposed U-net architecture has obtained desired segmentation performance in Section 4. As the focus of the network structure, RUC and deformable convolution are introduced in detail in Sections 3.2 and 3.3, respectively. 3.2 Reshape upsampling convolution In the decoder of U-net architecture, the resolution of the feature map is expanded by two times and the number of channels is reduced by two times by one time upsampling to get the same resolution and number of channels as the convolution layer. As a result, the two layers that are subsequently concatenated have the same size [7]. The upsampling process of RUC is shown in Fig. 2. For the input feature map with resolution and channel number C, the number of channels is enlarged twice first by convolution with 2C filters. Then, a new feature map of is obtained by GN operation and rectified linear unit (ReLU) activation function. Finally, the new feature map is divided into parts (represented by cubes of different colours in Fig. 2). Each part is reconstructed (the blue block in the yellow feature map shows the way of reconstructing). The feature map of is changed to . That is, the upsampling feature map of is generated, which completes the upsampling process with the resolution expanded by two times and the number of channels reduced by half. Fig. 2Open in figure viewerPowerPoint Illustration of RUC. The blue blocks in the two yellow feature maps show the way of the reshaping The main idea of RUC is that the feature map to be sampled has a large sense field. It can predict the value of each position in the feature map by convolution directly. Especially, RUC is learnable and does not require zero padding. Therefore, RUC is more effective than deconvolution and bilinear interpolation in theory. Furthermore, it is also verified by the performance of in Section 4. The convolution in RUC is lightweight, so the parameters and computational complexity of RUC are less than the other two methods, and the resolution and the number of channels of the sampling feature map can be controlled by the convolution, which has strong flexibility. 3.3 Deformable convolution The classic U-net is an end-to-end image segmentation model based on the FCN. The conventional convolutions used in this method have a fixed acceptance field because of the fixed geometry structure, which limits the ability to learn geometric deformation [14]. The deformable convolution can adaptively change the acceptance domain according to the data. In the conventional convolution, the size and shape of the convolution core are fixed, and the input feature graph x is sampled through a regular grid. For example, the grid R of a 3 × 3 convolution can be expressed as , for each pixel on the output feature map y, so the normal convolution can be expressed as (1)where is the value on the location in the output feature map and denotes the value on the location in the input feature map. Deformable convolution adds 2D offsets to the regular grid R, (1) becomes (2)The is typically fractional. The is implemented via bilinear interpolation as (3)where () denotes the coordinate of , () enumerates all integral spatial locations on the input feature map . is the 1D bilinear interpolation kernel, which is defined as It can be seen that only some values of bilinear interpolation g(a, b) are non-zero in the two nearest integral coordinates, (3) can be calculated quickly. The offset field keeps the same for different channels. Exceptionally, deformable convolution is equivalent to dilated convolution [36] with rate 2 when . So, it is a generalisation of dilated convolution. As shown in Fig. 3, two additional convolutions are applied in the input feature map to generate the new feature map, which has the same spatial resolution with the input. Its channel is that corresponds to N () 2D offsets. Therefore, in a deformable convolution, the number of channels of the new feature graph is 18. Then, the feature map is reconstructed into a offset field of dimension . The two channels correspond to the x-coordinate offset and the y-coordinate offset. As shown in the blue and yellow grids in Fig. 3, each area of 3 × 3 represents the offset of a convolution (corresponding to in (2). Adding offset on the basis of the grid, and bilinear interpolation operation [corresponding (3) is carried out on the input feature map accordingly, which enlarges the resolution of the input feature map by three times]. Finally, d convolution operations with size and step length 3 are used to output the feature map. During the training process, to learn the offset, the gradient can be propagated backward through the bilinear operations. Fig. 3Open in figure viewerPowerPoint Illustration of 3 × 3 deformable convolutions. The blue and yellow blocks in the purple feature maps show the way of the reshaping It should be noted that the deformable convolution in this paper differs from [14]; the receptive field of additional convolutions in this paper is larger than that of the current convolution kernel. For example, if the convolution core of the current convolution layer is , the convolution of the learning offset consists of two convolution layers (see Fig. 3), and a acceptance domain is obtained on the input feature map. In this way, the offsets can also be effectively learnt for non-grid-centred points, and the two convolution layers also increase the non-linearity of the network and enhance the learning ability of spatial transformation. In addition, the activation function of the first convolution layer is exponential linear unit. Unlike ReLU [37], it still has output when the input is negative, so that more information can be retained and input to the second convolution layer. The activation function of the second convolution layer is replaced by tanh, which maps the output to (−1, 1) corresponding to the offset interval (−1, 1). In this paper, GN is added before tanh to ensure that its input is in unsaturated region, so as to avoid the problem that the gradient disappears when the gradient is too small. The deformable convolution can adaptively change the acceptance domain according to the data, thus adapting to the geometric change of the target. Moreover, the computational complexity of deformable convolution is less than that of conventional convolution. The number of the filters of the two convolutions is only 2N, so the increased calculated amount is negligible. In Section 4, the performance of the improved deformable convolution can be shown by comparing and or comparing and . 3.4 Loss function In image segmentation, the goal of the network is to output the probability that each pixel belongs to various categories. Cross-entropy can be used as a loss function in neural network (machine learning) to measure the similarity between predicted value and real value. Binary cross-entropy is a commonly used loss function in biomedical image segmentation that can be defined as (4)where is the probability that the pixel at position x is a positive sample in the predicted map and is the corresponding ground truth. For images with serious imbalance in positive and negative samples (as showed in Fig. 4, less black and more white pixels denote the negative and positive, respectively), binary cross-entropy biases the learning of the model toward positive samples, leading to a reduction in the generalisation ability of the model [16]. Then, weighted binary cross-entropy [17] changes the weight of the positive class in loss by giving a weight factor to it. It can balance the contribution of classes to loss function to prevent model overwhelmed. Samples from the same class in different locations are also different. As shown in Fig. 4, the positive samples are easily segmented in the cell centre, and the samples near the cell boundary are difficult to segment. So, it is necessary to give more weight to the hard samples in the loss function. Ronneberger et al. [7] propose a binary cross-entropy with weight maps, which have the same resolution with the ground-truth segmentation. This method not only takes into consideration the serious class imbalance, but also thinks that the samples at different positions should have different contributions to loss. The way can make loss converge to a good position. However, the weight maps are computationally intensive and increase the time of training greatly. Focal loss is originally designed for dense object detection tasks to solve the two problems, one of which is imbalance class generated by anchor, the other of which is that simple samples contribute significantly to loss, resulting in poor model generalisation ability [16]. In this paper, we apply focal loss to biomedical image segmentation (5)where w is a factor that deals with the imbalance of classes. , which controls the contribution of hard and simple samples to loss. For a positive sample, the larger indicates that it is a simple sample, the smaller the corresponding , thus reducing its contribution to the loss. Instead, the smaller the p(x) is, the harder the sample is, and the larger is, the greater the contribution to loss. Fig. 4Open in figure viewerPowerPoint Example in training data (a) Original image, (b) Corresponding segmentation annotation Focal loss not only solves the serious imbalance classes, but also adaptively changes the contribution of hard and simple samples in the loss. Moreover, it does not need to calculate complex weight maps, so that the model can learn better. By comparing the performances of and , or and , focal loss can greatly improve performance in biomedical image segmentation tasks. 4 Experiments and results The proposed method is implemented using TensorFlow open-source deep learning library [38] and a NVIDIA Tesla M40 GPU is used to accelerate training. We train the networks with Adam algorithm and initial learning rate of 0.001. L2 regularisation is also applied for reducing the overfitting (attenuation factor is 0.0005). Similar to [39, 40], we apply a 'poly' learning rate policy (the initial learning rate is multiplied by with ). In this section, our experiments are carried out on Drosophila EM and Warwick-QU gland segmentation datasets. To verify the validity of RUC, the upsampling method in U-net is replaced with RUC and the other part of the network remains unchanged. Binary cross-entropy is used as the loss function, which is denoted as . To verify the effectiveness of the new DRU-Net network structure proposed in this chapter, we adopt the network structure in Fig. 1. The binary cross-entropy is used as the loss function, which is recorded as . To verify the effectiveness of focal loss in partitioning tasks, we replace the loss functions in and with centralised loss functions (), which are recorded as and , respectively. Through the following comparative analysis, has the best effect, which is the final method given in this paper. 4.1 Segmenting Drosophila EM data Drosophila EM data is the benchmark dataset of 2012 ISBI segmentation challenge. The training dataset consists of 30 images (512 × 512 px2) captured from a first instar larva ventral nerve cord. Each image has a segmentation annotation corresponding to it (see Fig. 4, black pixels denote membranes and white pixels denote cells). The test set consists of other 30 images for which segmentation annotations are kept secret. Since deep learning requires a large amount of data for training, we use data enhancement methods such as random flip, rotation, and elastic distortion to increase the number of images in the training set. In this dataset, ISBI competition organisers put forward two evaluation indicators, one is , the other is [41]. To compute the two indicators, each boundary map is transformed into a segmentation by marking connected components of '1' pixel as a segment and every '0' pixel is regarded as a segment containing just one pixel. The following describes the two scores in detail. Vrand: Suppose S is the predictive segmentation and G is the ground-truth segmentation. Let be the number of pixels in both the individual i in S and the individual j in G. which denotes the total number of pixels in the segmentation image. and denote the number of pixels belonging to segment k in G and S, respectively. Rand split score and Rand merge score are, respectively, defined as (6)The closer the two indicators are to 1, the smaller the separation error and fusion error. To balance the separation and fusion er