# A Multiscale CNN-CRF Framework for Environmental Microorganism Image Segmentation

Authors: Jinghua Zhang; Chen Li; Frank Kulwa; Xin Zhao; Changhao Sun; Zihan Li; Tao Jiang; Hong Li; Shouliang Qi
Journal: BioMed Research International
 (2020)

---

## Abstract

To assist researchers to identify Environmental Microorganisms (EMs) effectively, aMultiscale CNN-CRF (MSCC) framework for the EM image segmentation is proposed in this paper. There are two parts in this framework: The first is a novel pixel-level segmentation approach, using a newly introduced Convolutional Neural Network (CNN), namely, “mU-Net-B3”, with a dense Conditional Random Field (CRF) postprocessing. The second is a VGG-16 based patch-level segmentation method with a novel “buffer” strategy, which further improves the segmentation quality of the details of the EMs. In the experiment, compared with the state-of-the-art methods on 420 EM images, the proposed MSCC method reduces the memory requirement from 355 MB to 103 MB, improves the overall evaluation indexes (Dice, Jaccard, Recall, Accuracy) from 85.24%, 77.42%, 82.27%, and 96.76% to 87.13%, 79.74%, 87.12%, and 96.91%, respectively, and reduces the volume overlap error from 22.58% to 20.26%. Therefore, the MSCC method shows great potential in the EM segmentation field.


---

## Body

## 1. Introduction

Environmental pollution is an extremely serious problem in many countries. Therefore, many methods to deal with environmental pollution are constantly being put forward. The methods of eliminating environmental pollution can be divided into three major categories: chemical, physical, and biological. The biological method is more harmless and well efficient [1]. Environmental Microorganisms (EMs) are microscopic organisms living in the environment, which are natural decomposers and indicators [2]. For example, Actinophrys can digest the organic waste in sludge and increase the quality of freshwater. Therefore, the research on EMs plays a significant role in the management of pollution [3]. The identification of EMs is the basic step for related researches.Generally, there are four traditional types of EM identification strategies. The first one is the chemical method, which is highly accurate but often results in secondary pollution of chemical reagent [4]. The second strategy is the physical method. This method also has high accuracy, but it requires expensive equipment [4]. The third is the molecular biological method, which distinguishes EMs by sequence analysis of genome [5]. This strategy needs expensive equipment, plenty of time, and professional researchers. The fourth strategy is the morphological observation, which needs an experienced operator to observe EMs under a microscope and give the EM identities by their shape characteristics [1]. Hence, these traditional methods have their respective disadvantages in practical work.The morphological method has the lowest cost of the above methods, but it is laborious and tedious. Considering that deep learning achieves good performance in many fields of imaging processing, it can be used to make up the drawbacks of the traditional morphological method. Thus, we propose a full-automatic system for the EM image segmentation task, which can obtain the EM shape characteristics to assist researchers to detect and identify EMs effectively. The proposed system has two parts: The first part is a novel deepConvolutional Neural Network (CNN), namely, “mU-Net-B3”, with a Conditional Random Field (CRF) based pixel-level segmentation approach; the second part is a VGG-16 network [6] based patch-level segmentation method. In the pixel-level part, high-quality segmentation results are obtained on most EM images but lose effectiveness on some details with under-segmentation problems in some images. Therefore, we propose the patch-level part to assist the system to obtain more details of EMs. Hence, our Multiscale CNN-CRF (MSCC) segmentation system can solve the EM image segmentation effectively.In the pixel-level part, mU-Net-B3 with denseCRF is used as the core step for the segmentation task, where mU-Net-B3 is an improved U-Net. Compared with U-Net, it effectively improves the performance of segmentation result and reduces the memory requirement. Because denseCRF [7] can obtain global information between pixels in an image, it is used as the postprocessing after mU-Net-B3, which further improves the performance of the segmentation results. In the patch-level part, the segmentation task is actually a binary classification task. Because of the outstanding classification ability of VGG-16 in ImageNet [6] and the significant performance of transfer learning with limited training data set, we use the limited EM training data to fine-tune the VGG-16 model pretrained by ImageNet, which provides hundreds of object categories and millions of images [6], in our patch-level part. This approach effectively generates good classification results, from which we reconstruct the patch-level segmentation results. The EM segmentation framework is shown in Figure 1.Figure 1
An overview of our MSCC EM segmentation framework.In Figure1, (a) denotes the “Training Images”: The training set contains 21 categories of EM images and their corresponding ground truth (GT) images. We unify the image size to 256×256 pixels. Considering the colour information is inefficient in EM segmentation [8], these images are converted into grayscale; (b) shows the “Patch-level Training”: Images and their corresponding GT images are meshed into patches (8×8 pixels). Then, the data augmentation operation is used to balance patch data. After that, the balanced data are used to fine-tune the pretrained VGG-16 to obtain the classification model; (c) is the “Pixel-level Training”: Data augmentation is applied to make up the lack of data. Then, the data are fed to the mU-Net-B3 to obtain the segmentation model; (d) is “Testing Images”: The test set only has original images. We, respectively, convert them into grayscale images and patches for pixel-level and patch-level tests; (e) denotes the “Pixel-level Post-processing”: The denseCRF is used to further improve the pixel-level segmentation results; (f) shows “Patch-level Post-processing”: The predicted labels of patches are used to reconstruct the patch-level segmentation results. For further optimization, the denseCRF results are used to create the buffers to help the patch-level results to denoise. (g) is the “Final Results”: The denseCRF results and buffer results are combined and plotted by different colours on the original images.The main contributions of this paper are as follows:(i)
We propose a novel automatic approach that segments EM images from pixel-level and patch-level to assist EM analysis work(ii)
We propose three different strategies to optimize the original U-Net from the perspective of the receptive field, which well improve the segmentation performance(iii)
The proposed mU-Net-B3 not only improves the segmentation performance but also reduces the memory requirement to less than a third of that of U-Net


## 2. Related Works

### 2.1. Existing Microorganism Segmentation Methods

In this section, related works about microorganism image segmentation techniques are briefly summarized, including classical and machine learning-based methods. For more details, please refer to our previous survey in [9].


#### 2.1.1. Classical Methods

Classical methods include three subcategories, which are threshold-based methods, edge-based methods, and region-based methods. Threshold-based methods: The related work [10] shows a comparison between threshold-based segmentation methods for biofilms. The last result shows that iterative selection method is superior; in [11], different algorithms that are based on Otsu thresholding are applied for the segmentation of floc and filaments to enhance monitoring of activated sludge in waste water treatment plants. Edge-based methods: A segmentation and classification work is introduced to identify individual microorganism from a group of overlapping (touching) bacteria in [12]. Canny is used as the basic step of the segmentation part in [12]; in [13], to be able to segment large size images of zoo-planktons, a segmentation (based on Active Contour) and preclassification algorithm is used after the acquisition of images. Region-based methods: In [14], the segmentation is performed on gray-level images using marker controlled watershed method; in [15], after converting the colour mode and using morphological operations to denoise, seeded region-growing watershed algorithm is applied for segmentation.


#### 2.1.2. Machine Learning Methods

Machine learning methods usually have two categories: unsupervised and supervised methods. Unsupervised methods: [16] evaluates clustering and threshold segmentation techniques on tissue images containing TB Bacilli. The final result shows that k-means clustering (k=3) is outstanding; In [17], a comparison between condition random fields and region-based segmentation methods is presented. The final result shows that these two kinds of methods for microorganism segmentation have an average recognition rate above 80%. Supervised Methods: In [18], a segmentation system is designed to monitor the algae in water bodies. Its main thought is image enhancement (sharpening) applied first by using the Retinex filtering technique, then segmentation is done by using support vector machine; in [19], a network for segmentation of Rift Valley virus is proposed. Because of the insufficient data set, data augmentation is used to assist U-Net, which is used for segmentation.


### 2.2. Machine Learning Methods

In this section, the methods related to our work are introduced, including U-Net [20], Inception [21], denseCRF [7], and VGG-16 [6].


#### 2.2.1. U-Net

U-Net is a convolutional neural network, which is initially used to perform the task of medical image segmentation. The architecture of U-Net is symmetrical. It consists of a contracting path and an expansive path [20]. There are two important contributions of U-Net. The first is the strong use of data augmentation to solve the problem of insufficient training data. The second is its end-to-end structure, which can help the network to retrieve the information from the shallow layers. With the outstanding performance, U-Net is widely used in the task of semantic segmentation. The network structure of U-Net is shown in Figure 2.Figure 2
The network structure of U-Net.


#### 2.2.2. Inception

The original Inception, which uses filters of different sizes (1×1, 3×3, 5×5), is proposed in GoogleNet [22]. Because of the use of these filters, Inception has the capacity to adapt objects that have various sizes in images. However, there are also some disadvantages with the different filters used, for instance, the increasing of parameters, overfitting, and vanishing gradient. To reduce the negative effects, Inception-V2 gives a novel method, which is combining two 3×3 convolution filters to replace one 5×5 convolution filter [21]. For further optimization, Inception-V3 proposes a better approach, which uses a sequence of 1×N convolution filter and N×1 convolution filter to replace N×N convolution filter [21]. Figure 3 also shows the 3×3 convolution filter replaced by 1×3 and 3×1 convolution filters. This strategy reduces more parameter count. Furthermore, with more convolution filters with ReLU used, the expressiveness is improved.Figure 3
The strategies used by Inception-V2 and Inception-V3 to replace the big filter.
(a)
The 5 × 5 convolution filter(b)
Two 3 × 3 convolution filter(c)
The 3 × 3 convolution filter(d)
1 × 3 and 3 × 1 convolution filter


#### 2.2.3. DenseCRF

Although CNNs can perform well on pixel-level segmentation, there are still some details that are not perfect enough. The main reason is it is difficult to consider the spatial relationships between different pixels in the process of pixel-level segmentation by CNNs. However, [23] shows that using denseCRF as postprocessing after CNNs can capture the spatial relationships. It can improve the segmentation results. In [7], the energy function of denseCRF model is the sum of unary potential and pairwise potential, which is shown in Eq. (1).
(1)Ex=∑iUxi+∑i,jPxi,xj.In Eq. (1), x is the label assignment of pixel. Uxi represents the unary potential, which measures the inverse likelihood of the pixel i taking the label xi, and Pxi,xj means the pairwise potential, which measures the cost of assigning labels xi, xjto pixels i, j simultaneously [24]. We use Eq. (2) as unary potential, where Lxi is the label assignment probability at pixel i.
(2)Uxi=−logLxi.The pairwise potential is defined in Eq. (3), where ∅xi,xj is a penalty term on the labelling [25]. As explained in [7], ∅xi,xj is given by the Potts model. If pixel i and pixel j have the same label, the penalty term is equal to zero, and if not, it is equal to one.
(3)Pxi,xj=∅xi,xj∑m=1Mωmkmfi,fj⏟.kfi,fjAs Eq. (3) shows, each km is the Gaussian kernel, which depends on the feature vectors fi, fj of pixels i, j,and is weighted by ωm. In [7], it uses contrast-sensitive two-kernel potentials, defined in terms of the colour vectors Ii and Ij and positions pi and pj. It is shown as Eq. (4).
(4)kfi,fj=ω1exp−pi−py22σα2−Ii−Ij22σβ2⏟appearancekernel+ω2exp−pi−pj22σγ2⏟smoothnesskernel.The first appearance kernel depends on both pixel positions (denoted asp) and pixel colour intensities (denoted as I). The second smoothness kernel only depends on pixel positions. And the parameters σα, σβ, and σω control the scale of Gaussian kernels. The first kernel forces pixels with similar colour and position to have similar labels, while the second kernel only considers spatial proximity when enforcing smoothness [23].


#### 2.2.4. VGG-16

Simonyan et al. propose VGG-16, which not only achieves the state-of-the-art accuracy on ILSVRC 2014 classification and localisation tasks but is also applicable to other image recognition data sets, where they achieve excellent performance even when used as a part of relatively simple pipelines [6]. The architecture of VGG-16 is shown in Figure 4.Figure 4
The architecture of VGG-16 network.


## 2.1. Existing Microorganism Segmentation Methods

In this section, related works about microorganism image segmentation techniques are briefly summarized, including classical and machine learning-based methods. For more details, please refer to our previous survey in [9].


### 2.1.1. Classical Methods

Classical methods include three subcategories, which are threshold-based methods, edge-based methods, and region-based methods. Threshold-based methods: The related work [10] shows a comparison between threshold-based segmentation methods for biofilms. The last result shows that iterative selection method is superior; in [11], different algorithms that are based on Otsu thresholding are applied for the segmentation of floc and filaments to enhance monitoring of activated sludge in waste water treatment plants. Edge-based methods: A segmentation and classification work is introduced to identify individual microorganism from a group of overlapping (touching) bacteria in [12]. Canny is used as the basic step of the segmentation part in [12]; in [13], to be able to segment large size images of zoo-planktons, a segmentation (based on Active Contour) and preclassification algorithm is used after the acquisition of images. Region-based methods: In [14], the segmentation is performed on gray-level images using marker controlled watershed method; in [15], after converting the colour mode and using morphological operations to denoise, seeded region-growing watershed algorithm is applied for segmentation.


### 2.1.2. Machine Learning Methods

Machine learning methods usually have two categories: unsupervised and supervised methods. Unsupervised methods: [16] evaluates clustering and threshold segmentation techniques on tissue images containing TB Bacilli. The final result shows that k-means clustering (k=3) is outstanding; In [17], a comparison between condition random fields and region-based segmentation methods is presented. The final result shows that these two kinds of methods for microorganism segmentation have an average recognition rate above 80%. Supervised Methods: In [18], a segmentation system is designed to monitor the algae in water bodies. Its main thought is image enhancement (sharpening) applied first by using the Retinex filtering technique, then segmentation is done by using support vector machine; in [19], a network for segmentation of Rift Valley virus is proposed. Because of the insufficient data set, data augmentation is used to assist U-Net, which is used for segmentation.


## 2.1.1. Classical Methods

Classical methods include three subcategories, which are threshold-based methods, edge-based methods, and region-based methods. Threshold-based methods: The related work [10] shows a comparison between threshold-based segmentation methods for biofilms. The last result shows that iterative selection method is superior; in [11], different algorithms that are based on Otsu thresholding are applied for the segmentation of floc and filaments to enhance monitoring of activated sludge in waste water treatment plants. Edge-based methods: A segmentation and classification work is introduced to identify individual microorganism from a group of overlapping (touching) bacteria in [12]. Canny is used as the basic step of the segmentation part in [12]; in [13], to be able to segment large size images of zoo-planktons, a segmentation (based on Active Contour) and preclassification algorithm is used after the acquisition of images. Region-based methods: In [14], the segmentation is performed on gray-level images using marker controlled watershed method; in [15], after converting the colour mode and using morphological operations to denoise, seeded region-growing watershed algorithm is applied for segmentation.


## 2.1.2. Machine Learning Methods

Machine learning methods usually have two categories: unsupervised and supervised methods. Unsupervised methods: [16] evaluates clustering and threshold segmentation techniques on tissue images containing TB Bacilli. The final result shows that k-means clustering (k=3) is outstanding; In [17], a comparison between condition random fields and region-based segmentation methods is presented. The final result shows that these two kinds of methods for microorganism segmentation have an average recognition rate above 80%. Supervised Methods: In [18], a segmentation system is designed to monitor the algae in water bodies. Its main thought is image enhancement (sharpening) applied first by using the Retinex filtering technique, then segmentation is done by using support vector machine; in [19], a network for segmentation of Rift Valley virus is proposed. Because of the insufficient data set, data augmentation is used to assist U-Net, which is used for segmentation.


## 2.2. Machine Learning Methods

In this section, the methods related to our work are introduced, including U-Net [20], Inception [21], denseCRF [7], and VGG-16 [6].


### 2.2.1. U-Net

U-Net is a convolutional neural network, which is initially used to perform the task of medical image segmentation. The architecture of U-Net is symmetrical. It consists of a contracting path and an expansive path [20]. There are two important contributions of U-Net. The first is the strong use of data augmentation to solve the problem of insufficient training data. The second is its end-to-end structure, which can help the network to retrieve the information from the shallow layers. With the outstanding performance, U-Net is widely used in the task of semantic segmentation. The network structure of U-Net is shown in Figure 2.Figure 2
The network structure of U-Net.


### 2.2.2. Inception

The original Inception, which uses filters of different sizes (1×1, 3×3, 5×5), is proposed in GoogleNet [22]. Because of the use of these filters, Inception has the capacity to adapt objects that have various sizes in images. However, there are also some disadvantages with the different filters used, for instance, the increasing of parameters, overfitting, and vanishing gradient. To reduce the negative effects, Inception-V2 gives a novel method, which is combining two 3×3 convolution filters to replace one 5×5 convolution filter [21]. For further optimization, Inception-V3 proposes a better approach, which uses a sequence of 1×N convolution filter and N×1 convolution filter to replace N×N convolution filter [21]. Figure 3 also shows the 3×3 convolution filter replaced by 1×3 and 3×1 convolution filters. This strategy reduces more parameter count. Furthermore, with more convolution filters with ReLU used, the expressiveness is improved.Figure 3
The strategies used by Inception-V2 and Inception-V3 to replace the big filter.
(a)
The 5 × 5 convolution filter(b)
Two 3 × 3 convolution filter(c)
The 3 × 3 convolution filter(d)
1 × 3 and 3 × 1 convolution filter


### 2.2.3. DenseCRF

Although CNNs can perform well on pixel-level segmentation, there are still some details that are not perfect enough. The main reason is it is difficult to consider the spatial relationships between different pixels in the process of pixel-level segmentation by CNNs. However, [23] shows that using denseCRF as postprocessing after CNNs can capture the spatial relationships. It can improve the segmentation results. In [7], the energy function of denseCRF model is the sum of unary potential and pairwise potential, which is shown in Eq. (1).
(1)Ex=∑iUxi+∑i,jPxi,xj.In Eq. (1), x is the label assignment of pixel. Uxi represents the unary potential, which measures the inverse likelihood of the pixel i taking the label xi, and Pxi,xj means the pairwise potential, which measures the cost of assigning labels xi, xjto pixels i, j simultaneously [24]. We use Eq. (2) as unary potential, where Lxi is the label assignment probability at pixel i.
(2)Uxi=−logLxi.The pairwise potential is defined in Eq. (3), where ∅xi,xj is a penalty term on the labelling [25]. As explained in [7], ∅xi,xj is given by the Potts model. If pixel i and pixel j have the same label, the penalty term is equal to zero, and if not, it is equal to one.
(3)Pxi,xj=∅xi,xj∑m=1Mωmkmfi,fj⏟.kfi,fjAs Eq. (3) shows, each km is the Gaussian kernel, which depends on the feature vectors fi, fj of pixels i, j,and is weighted by ωm. In [7], it uses contrast-sensitive two-kernel potentials, defined in terms of the colour vectors Ii and Ij and positions pi and pj. It is shown as Eq. (4).
(4)kfi,fj=ω1exp−pi−py22σα2−Ii−Ij22σβ2⏟appearancekernel+ω2exp−pi−pj22σγ2⏟smoothnesskernel.The first appearance kernel depends on both pixel positions (denoted asp) and pixel colour intensities (denoted as I). The second smoothness kernel only depends on pixel positions. And the parameters σα, σβ, and σω control the scale of Gaussian kernels. The first kernel forces pixels with similar colour and position to have similar labels, while the second kernel only considers spatial proximity when enforcing smoothness [23].


### 2.2.4. VGG-16

Simonyan et al. propose VGG-16, which not only achieves the state-of-the-art accuracy on ILSVRC 2014 classification and localisation tasks but is also applicable to other image recognition data sets, where they achieve excellent performance even when used as a part of relatively simple pipelines [6]. The architecture of VGG-16 is shown in Figure 4.Figure 4
The architecture of VGG-16 network.


## 2.2.1. U-Net

U-Net is a convolutional neural network, which is initially used to perform the task of medical image segmentation. The architecture of U-Net is symmetrical. It consists of a contracting path and an expansive path [20]. There are two important contributions of U-Net. The first is the strong use of data augmentation to solve the problem of insufficient training data. The second is its end-to-end structure, which can help the network to retrieve the information from the shallow layers. With the outstanding performance, U-Net is widely used in the task of semantic segmentation. The network structure of U-Net is shown in Figure 2.Figure 2
The network structure of U-Net.


## 2.2.2. Inception

The original Inception, which uses filters of different sizes (1×1, 3×3, 5×5), is proposed in GoogleNet [22]. Because of the use of these filters, Inception has the capacity to adapt objects that have various sizes in images. However, there are also some disadvantages with the different filters used, for instance, the increasing of parameters, overfitting, and vanishing gradient. To reduce the negative effects, Inception-V2 gives a novel method, which is combining two 3×3 convolution filters to replace one 5×5 convolution filter [21]. For further optimization, Inception-V3 proposes a better approach, which uses a sequence of 1×N convolution filter and N×1 convolution filter to replace N×N convolution filter [21]. Figure 3 also shows the 3×3 convolution filter replaced by 1×3 and 3×1 convolution filters. This strategy reduces more parameter count. Furthermore, with more convolution filters with ReLU used, the expressiveness is improved.Figure 3
The strategies used by Inception-V2 and Inception-V3 to replace the big filter.
(a)
The 5 × 5 convolution filter(b)
Two 3 × 3 convolution filter(c)
The 3 × 3 convolution filter(d)
1 × 3 and 3 × 1 convolution filter


## 2.2.3. DenseCRF

Although CNNs can perform well on pixel-level segmentation, there are still some details that are not perfect enough. The main reason is it is difficult to consider the spatial relationships between different pixels in the process of pixel-level segmentation by CNNs. However, [23] shows that using denseCRF as postprocessing after CNNs can capture the spatial relationships. It can improve the segmentation results. In [7], the energy function of denseCRF model is the sum of unary potential and pairwise potential, which is shown in Eq. (1).
(1)Ex=∑iUxi+∑i,jPxi,xj.In Eq. (1), x is the label assignment of pixel. Uxi represents the unary potential, which measures the inverse likelihood of the pixel i taking the label xi, and Pxi,xj means the pairwise potential, which measures the cost of assigning labels xi, xjto pixels i, j simultaneously [24]. We use Eq. (2) as unary potential, where Lxi is the label assignment probability at pixel i.
(2)Uxi=−logLxi.The pairwise potential is defined in Eq. (3), where ∅xi,xj is a penalty term on the labelling [25]. As explained in [7], ∅xi,xj is given by the Potts model. If pixel i and pixel j have the same label, the penalty term is equal to zero, and if not, it is equal to one.
(3)Pxi,xj=∅xi,xj∑m=1Mωmkmfi,fj⏟.kfi,fjAs Eq. (3) shows, each km is the Gaussian kernel, which depends on the feature vectors fi, fj of pixels i, j,and is weighted by ωm. In [7], it uses contrast-sensitive two-kernel potentials, defined in terms of the colour vectors Ii and Ij and positions pi and pj. It is shown as Eq. (4).
(4)kfi,fj=ω1exp−pi−py22σα2−Ii−Ij22σβ2⏟appearancekernel+ω2exp−pi−pj22σγ2⏟smoothnesskernel.The first appearance kernel depends on both pixel positions (denoted asp) and pixel colour intensities (denoted as I). The second smoothness kernel only depends on pixel positions. And the parameters σα, σβ, and σω control the scale of Gaussian kernels. The first kernel forces pixels with similar colour and position to have similar labels, while the second kernel only considers spatial proximity when enforcing smoothness [23].


## 2.2.4. VGG-16

Simonyan et al. propose VGG-16, which not only achieves the state-of-the-art accuracy on ILSVRC 2014 classification and localisation tasks but is also applicable to other image recognition data sets, where they achieve excellent performance even when used as a part of relatively simple pipelines [6]. The architecture of VGG-16 is shown in Figure 4.Figure 4
The architecture of VGG-16 network.


## 3. Multiscale CNN-CRF Model

### 3.1. Pixel-Level Training

In pixel-level training, our novel multilevel CNN-CRF framework is introduced. In our data set, there are many objects of various sizes. As Figure5 shows, we can easily find that the EM shapes in different categories are completely different. Considering the current U-Net is difficult to adapt to this situation, we propose novel methods to optimize the adaptability of U-Net.Figure 5
The variety of the object sizes in EM images.As the U-Net structure is shown in Figure2, we can find that the receptive field of U-Net is limited. To optimize the adaptability of U-Net, the direct way is using convolution filters of different sizes, just as Inception does. We propose BLOCK-I, which incorporates 1×1, 3×3, 5×5, and 7×7 convolution filters in parallel, as shown in Figure 6. Although this approach can help the network to improve the adaptability, it also makes more parameters.Figure 6
The architecture of BLOCK-I.Inspired by Inception-V2 [21], a 5×5 convolution filter actually resembles a sequence of two 3×3 convolution filters. Likewise, a 7×7 convolution filter can be replaced by a sequence of three 3×3 convolution filters. In [26], the concatenate operation is used to concatenate the outputs after the first convolution operation and the second convolution operation with the output of the third convolution operation in a sequence of three 3×3 convolution operations to obtain the result, which resembles the interaction result of 3×3, 5×5, and 7×7 convolution operations. Therefore, we apply this concept to optimize BLOCK-I, and we get a novel architecture called BLOCK-II. BLOCK-II is shown in Figure 7. Compared with BLOCK-I, this architecture can reduce parameters.Figure 7
The architecture of BLOCK-II.Although the parameters of BLOCK-II are quite less than BLOCK-I, there is still some room for improvement in this architecture. As we mentioned about Inception-V3, a3×3 convolution filter can also be replaced by a sequence of 1×3 and 3×1 convolution filters. We apply this concept in BLOCK-III, which is shown in Figure 8. The experiments show that this approach can effectively reduce the memory requirement and achieve well-performed results.Figure 8
The architecture of BLOCK-III.Finally, we provide the whole architecture of our network mU-Net in Figure9. Because of the least memory requirement of BLOCK-III, we deploy BLOCK-III in mU-Net architecture in our final method. Besides, we add a batch normalization layer [27] after each convolution layer and convolution transpose layer. For short, mU-Net with BLOCK-X is abbreviated as “mU-Net-BX”. The details of mU-Net-BXs are provided in Table 1. The details of hyperparameters used in the pixel-level training process are provided in the following subsection: Pixel-level Implementation Details.Figure 9
The architecture of mU-Net.Table 1
Details of mU-Net architecture with different BLOCKs.
BlockModelFilter numberBlockModelFilter numbermU-Net-B1mU-Net-B2mU-Net-B3mU-Net-B1mU-Net-B2mU-Net-B3Block 1 and Block 9Con2D (3,3)Con2D (3,3)Con2D (3,1)16Block 2 and Block 8Con2D (3,3)Con2D (3,3)Con2D (3,1)32Con2D (1,3)Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Block 3 and Block 7Con2D (3,3)Con2D (3,3)Con2D (3,1)64Block 4 and Block 6Con2D (3,3)Con2D (3,3)Con2D (3,1)128Con2D (1,3)Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Block 5Con2D (3,3)Con2D (3,3)Con2D (3,1)256Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)


### 3.2. Patch-Level Training

In our patch-level training, we use our data set to fine-tune the VGG-16 [6], which is pretrained on a large-scale image data set ImageNet [28, 29].


#### 3.2.1. Fine-Tune Pretrained VGG-16

It is proved that the use of VGG-16 pretrained on ImageNet can be useful for classification tasks through the concept of transfer learning and fine-tuning in [30]. In our framework, the patch-level segmentation is actually a classification task.To fine-tune the pretrained model, we mesh the training EM images into patches of8×8 pixels. The examples are shown in Figure 10. There are two reasons for using patches of 8×8 pixels. First, all the EM image sizes are converted into 256×256 pixels where 256 can only be divisible by 2, 4, 8, 16, 32, 64, 128, or 256. Second, the patches, which are too large or too small, make no sense for the patch-level segmentation, because small patches cannot obtain details of EMs and large patches will result in poor segmentation results. We provide some examples of patches of different sizes in the original EM images in Figure 11. As we can see, patches of 2×2 and 4×4 pixels are too small to cover the details of EMs, and patches of 16×16 pixels are too large for the images.Figure 10
Examples of patches for patch-level training (the top row shows grayscale image patches, and the bottom row shows their corresponding GT images).Figure 11
The examples of the image patches of different sizes. The yellow outlines show the regions of EMs in GT images. The red arrows point out the image patches of different sizes. (From left to right, the EMs areActinophrys, Epistylis, and K. Quadrala).After that, we divide these patches into two categories: (With Object) and (Without Object). The criterion for dividing is the area of the object in each patch. If the area is more than half of the patch, we will give the label of (With Object) to the patch. If not, the label will be (Without Object).Finally, we apply data augmentation to make the number of patches in two categories balanced, and use balanced data to train a classification model through fine-tuning the pretrained VGG-16. As we can see from Figure4, the VGG-16 is pretrained by ImageNet. The pretrained model can be downloaded from Keras [31] directly. Before fine-tuning the pretrained VGG-16, we freeze the parameters of the pretrained model. After that, we use the balanced patch-level data to fine-tune the dense layers of VGG-16. The details of hyperparameters used in the patch-level training process are provided in the following subsection: Patch-level Implementation Details.


### 3.3. Pixel-Level Postprocessing

In our pixel-level segmentation, after getting the segmentation results from mU-Net-B3, we convert the results into binary images, where the foreground is marked as 1 (white) and the background is marked as 0 (black), and use these binary images as the initial matrices of denseCRF. It can effectively obtain the global information of images to optimize the segmentation results.


### 3.4. Patch-Level Postprocessing

In our patch-level segmentation, we use the predicted labels generated by VGG-16 to reconstruct the segmentation results. To remove the useless portions of the patch-level segmentation results, we built up the buffers by using the pixel-level postprocessing (denseCRF) results. The process is shown in Figure12. The way to make buffers is applying dilate operation to the denseCRF results. After that, we use these images as weight matrices to apply to the patch-level results. Only the patch-level segmentation results in the buffers are retained, and the segmentation results outside the buffers are erased. This approach can effectively help to denoise.Figure 12
The workflow of the patch-level postprocessing.


### 3.5. Segmentation Results Fusion and Presentation

After obtaining the segmentation results of pixel-level and patch-level, respectively, the final segmentation results are generated by combining these two kinds of segmentation results. For the convenience of observation, the segmentation results of pixel-level and patch-level are plotted on the original images in the form of masks of different colours. The masks of pixel-level are red, the masks of patch-level are fluorescent green, and the overlapped parts of pixel-level and patch-level segmentation results are yellow. Examples are shown in Figure13.Figure 13
The combined results of pixel-level segmentation results and patch-level segmentation results. The red and fluorescent green masks are pixel-level and patch-level segmentation results. The yellow masks represent the overlap of pixel-level and patch-level segmentation results. The purple outlines are the outlines of GT images.


## 3.1. Pixel-Level Training

In pixel-level training, our novel multilevel CNN-CRF framework is introduced. In our data set, there are many objects of various sizes. As Figure5 shows, we can easily find that the EM shapes in different categories are completely different. Considering the current U-Net is difficult to adapt to this situation, we propose novel methods to optimize the adaptability of U-Net.Figure 5
The variety of the object sizes in EM images.As the U-Net structure is shown in Figure2, we can find that the receptive field of U-Net is limited. To optimize the adaptability of U-Net, the direct way is using convolution filters of different sizes, just as Inception does. We propose BLOCK-I, which incorporates 1×1, 3×3, 5×5, and 7×7 convolution filters in parallel, as shown in Figure 6. Although this approach can help the network to improve the adaptability, it also makes more parameters.Figure 6
The architecture of BLOCK-I.Inspired by Inception-V2 [21], a 5×5 convolution filter actually resembles a sequence of two 3×3 convolution filters. Likewise, a 7×7 convolution filter can be replaced by a sequence of three 3×3 convolution filters. In [26], the concatenate operation is used to concatenate the outputs after the first convolution operation and the second convolution operation with the output of the third convolution operation in a sequence of three 3×3 convolution operations to obtain the result, which resembles the interaction result of 3×3, 5×5, and 7×7 convolution operations. Therefore, we apply this concept to optimize BLOCK-I, and we get a novel architecture called BLOCK-II. BLOCK-II is shown in Figure 7. Compared with BLOCK-I, this architecture can reduce parameters.Figure 7
The architecture of BLOCK-II.Although the parameters of BLOCK-II are quite less than BLOCK-I, there is still some room for improvement in this architecture. As we mentioned about Inception-V3, a3×3 convolution filter can also be replaced by a sequence of 1×3 and 3×1 convolution filters. We apply this concept in BLOCK-III, which is shown in Figure 8. The experiments show that this approach can effectively reduce the memory requirement and achieve well-performed results.Figure 8
The architecture of BLOCK-III.Finally, we provide the whole architecture of our network mU-Net in Figure9. Because of the least memory requirement of BLOCK-III, we deploy BLOCK-III in mU-Net architecture in our final method. Besides, we add a batch normalization layer [27] after each convolution layer and convolution transpose layer. For short, mU-Net with BLOCK-X is abbreviated as “mU-Net-BX”. The details of mU-Net-BXs are provided in Table 1. The details of hyperparameters used in the pixel-level training process are provided in the following subsection: Pixel-level Implementation Details.Figure 9
The architecture of mU-Net.Table 1
Details of mU-Net architecture with different BLOCKs.
BlockModelFilter numberBlockModelFilter numbermU-Net-B1mU-Net-B2mU-Net-B3mU-Net-B1mU-Net-B2mU-Net-B3Block 1 and Block 9Con2D (3,3)Con2D (3,3)Con2D (3,1)16Block 2 and Block 8Con2D (3,3)Con2D (3,3)Con2D (3,1)32Con2D (1,3)Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Block 3 and Block 7Con2D (3,3)Con2D (3,3)Con2D (3,1)64Block 4 and Block 6Con2D (3,3)Con2D (3,3)Con2D (3,1)128Con2D (1,3)Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Con2D (1,1)Block 5Con2D (3,3)Con2D (3,3)Con2D (3,1)256Con2D (1,3)Con2D (5,5)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (7,7)Con2D (3,3)Con2D (3,1)Con2D (1,3)Con2D (1,1)Con2D (1,1)Con2D (1,1)


## 3.2. Patch-Level Training

In our patch-level training, we use our data set to fine-tune the VGG-16 [6], which is pretrained on a large-scale image data set ImageNet [28, 29].


### 3.2.1. Fine-Tune Pretrained VGG-16

It is proved that the use of VGG-16 pretrained on ImageNet can be useful for classification tasks through the concept of transfer learning and fine-tuning in [30]. In our framework, the patch-level segmentation is actually a classification task.To fine-tune the pretrained model, we mesh the training EM images into patches of8×8 pixels. The examples are shown in Figure 10. There are two reasons for using patches of 8×8 pixels. First, all the EM image sizes are converted into 256×256 pixels where 256 can only be divisible by 2, 4, 8, 16, 32, 64, 128, or 256. Second, the patches, which are too large or too small, make no sense for the patch-level segmentation, because small patches cannot obtain details of EMs and large patches will result in poor segmentation results. We provide some examples of patches of different sizes in the original EM images in Figure 11. As we can see, patches of 2×2 and 4×4 pixels are too small to cover the details of EMs, and patches of 16×16 pixels are too large for the images.Figure 10
Examples of patches for patch-level training (the top row shows grayscale image patches, and the bottom row shows their corresponding GT images).Figure 11
The examples of the image patches of different sizes. The yellow outlines show the regions of EMs in GT images. The red arrows point out the image patches of different sizes. (From left to right, the EMs areActinophrys, Epistylis, and K. Quadrala).After that, we divide these patches into two categories: (With Object) and (Without Object). The criterion for dividing is the area of the object in each patch. If the area is more than half of the patch, we will give the label of (With Object) to the patch. If not, the label will be (Without Object).Finally, we apply data augmentation to make the number of patches in two categories balanced, and use balanced data to train a classification model through fine-tuning the pretrained VGG-16. As we can see from Figure4, the VGG-16 is pretrained by ImageNet. The pretrained model can be downloaded from Keras [31] directly. Before fine-tuning the pretrained VGG-16, we freeze the parameters of the pretrained model. After that, we use the balanced patch-level data to fine-tune the dense layers of VGG-16. The details of hyperparameters used in the patch-level training process are provided in the following subsection: Patch-level Implementation Details.


## 3.2.1. Fine-Tune Pretrained VGG-16

It is proved that the use of VGG-16 pretrained on ImageNet can be useful for classification tasks through the concept of transfer learning and fine-tuning in [30]. In our framework, the patch-level segmentation is actually a classification task.To fine-tune the pretrained model, we mesh the training EM images into patches of8×8 pixels. The examples are shown in Figure 10. There are two reasons for using patches of 8×8 pixels. First, all the EM image sizes are converted into 256×256 pixels where 256 can only be divisible by 2, 4, 8, 16, 32, 64, 128, or 256. Second, the patches, which are too large or too small, make no sense for the patch-level segmentation, because small patches cannot obtain details of EMs and large patches will result in poor segmentation results. We provide some examples of patches of different sizes in the original EM images in Figure 11. As we can see, patches of 2×2 and 4×4 pixels are too small to cover the details of EMs, and patches of 16×16 pixels are too large for the images.Figure 10
Examples of patches for patch-level training (the top row shows grayscale image patches, and the bottom row shows their corresponding GT images).Figure 11
The examples of the image patches of different sizes. The yellow outlines show the regions of EMs in GT images. The red arrows point out the image patches of different sizes. (From left to right, the EMs areActinophrys, Epistylis, and K. Quadrala).After that, we divide these patches into two categories: (With Object) and (Without Object). The criterion for dividing is the area of the object in each patch. If the area is more than half of the patch, we will give the label of (With Object) to the patch. If not, the label will be (Without Object).Finally, we apply data augmentation to make the number of patches in two categories balanced, and use balanced data to train a classification model through fine-tuning the pretrained VGG-16. As we can see from Figure4, the VGG-16 is pretrained by ImageNet. The pretrained model can be downloaded from Keras [31] directly. Before fine-tuning the pretrained VGG-16, we freeze the parameters of the pretrained model. After that, we use the balanced patch-level data to fine-tune the dense layers of VGG-16. The details of hyperparameters used in the patch-level training process are provided in the following subsection: Patch-level Implementation Details.


## 3.3. Pixel-Level Postprocessing

In our pixel-level segmentation, after getting the segmentation results from mU-Net-B3, we convert the results into binary images, where the foreground is marked as 1 (white) and the background is marked as 0 (black), and use these binary images as the initial matrices of denseCRF. It can effectively obtain the global information of images to optimize the segmentation results.


## 3.4. Patch-Level Postprocessing

In our patch-level segmentation, we use the predicted labels generated by VGG-16 to reconstruct the segmentation results. To remove the useless portions of the patch-level segmentation results, we built up the buffers by using the pixel-level postprocessing (denseCRF) results. The process is shown in Figure12. The way to make buffers is applying dilate operation to the denseCRF results. After that, we use these images as weight matrices to apply to the patch-level results. Only the patch-level segmentation results in the buffers are retained, and the segmentation results outside the buffers are erased. This approach can effectively help to denoise.Figure 12
The workflow of the patch-level postprocessing.


## 3.5. Segmentation Results Fusion and Presentation

After obtaining the segmentation results of pixel-level and patch-level, respectively, the final segmentation results are generated by combining these two kinds of segmentation results. For the convenience of observation, the segmentation results of pixel-level and patch-level are plotted on the original images in the form of masks of different colours. The masks of pixel-level are red, the masks of patch-level are fluorescent green, and the overlapped parts of pixel-level and patch-level segmentation results are yellow. Examples are shown in Figure13.Figure 13
The combined results of pixel-level segmentation results and patch-level segmentation results. The red and fluorescent green masks are pixel-level and patch-level segmentation results. The yellow masks represent the overlap of pixel-level and patch-level segmentation results. The purple outlines are the outlines of GT images.


## 4. Experiments and Analysis

### 4.1. Experimental Setting

#### 4.1.1. Image Data Set

In our work, we useEnvironmental Microorganism Data Set 5th Version (EMDS-5), which is a newly released version of EMDS series [32], containing 21 EM classes as shown in Figure 14. Each EM class contains 20 original microscopic images and their corresponding GT images, thus the data set includes 420 scenes. Owing to the microscopic images having multifarious sizes, we convert all the image sizes into 256×256 pixels uniformly.Figure 14
Examples of the images in EMDS-5. (a) shows the original EM images and (b) shows the corresponding GT images.
(a)
Original Image(b)
Ground truth image


#### 4.1.2. Training, Validation, and Test Data Setting

Due to the different living conditions and habits of EMs, it is difficult to obtain a large number of EM images for our EMDS-5 [8]. To observe the improvements made by the optimized models, a large amount of testing images is needed. Therefore, we randomly divide each class of EMDS-5 into a training data set, validation data set, and test data set in a ratio of 1 : 1 : 2. Furthermore, because of the limitation of EMDS-5, data augmentation is used in our pixel-level training. Inspired by the strategy proposed in [19], we augment the 105 training images with rotations by 0, 90, 180, 270 degrees, and mirroring, which results in 840 images for training. In our patch-level training, we mesh 105 training images and their corresponding GT images into patches (8×8 pixels), and 107520 patches are obtained. These patches are divided into two categories: (With Object) and (Without Object). We find that the numbers of patches in these two categories are inconsistent. The first category (With Object) has 18575 patches, and another category (Without Object) has 88945 patches. To resolve this situation, we employ data augmentation to the first category (With Object). We augment the 18575 patches in the first category (Without Object) with rotations by 0, 90, 180, 270 degrees, and mirroring, which result in 148600 patches. Then, we randomly choose 88945 patches to replace the data in the first category (With Object).


#### 4.1.3. Experimental Environment

The experiment is conducted by Python 3. The models are implemented using Keras [31] framework with Tensorflow [33] as backend. In our experiment, we use a workstation with Intel(R) Core(TM) i7-8700 CPU with 3.20 GHz, 32GB RAM, and NVIDIA GEFORCE RTX 2080 8 GB.


#### 4.1.4. Pixel-Level Implementation Details

In our pixel-level segmentation, the task of the segmentation is to predict the individual pixels whether they represent a point of foreground or background. Actually, this task can be seen as a pixel-level binary classification problem. Hence, as the loss function of the network, we simply take the binary cross-entropy function and minimize it [26]. Besides, we use Adam optimizer with 1.5×10−4 learning rate in our training process. The models are trained for 50 epochs using Adam optimizer. As the average training loss and accuracy curve of the training process is shown in Figure 15, we can find that the loss and accuracy curves of training and validation tend to level off after 30-35 iterations. Therefore, considering the computing performance of the workstation, we finally set 50 epochs for training.Figure 15
The loss and accuracy curves of the training process.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3


#### 4.1.5. Patch-Level Implementation Details

In our patch-level training process, we employ the pretrained VGG-16 as the core and fine-tune the dense layers of VGG-16. As Figure4 shows, the last layer is softmax. The categorical cross-entropy function is the loss function of choice for softmax output units. Besides, Adam optimizer with 1.0×10−4 learning rate is used in VGG-16. The pretrained model is trained for 15 epochs.


#### 4.1.6. Evaluation Metric

In our previous work [3], Recall and Accuracy are used to measure the segmentation results. Besides that, we employ Dice, Jaccard, and VOE (volumetric overlap error) to evaluate the segmentation results in this paper [34]. The definitions of these evaluation metrics are provided in Table 2. Vpred represents the foreground that is predicted by the model. Vgt represents the foreground in a ground truth image. From Table 2, we can find that the higher the values of the first four metrics (Dice, Jaccard, Recall, and Accuracy) are, the better the segmentation results are. On the contrary, the lower the value of the final metric (VOE) is, the better the segmentation result is.Table 2
The definitions of evaluation metrics for image segmentation. TP (True Positive), FN (False Negative), FP (False Positive), and TN (True Negative).
MetricDefinitionMetricDefinitionDiceDice=2×Vpred⋂VgtVpred+VgtJaccardJaccard=Vpred⋂VgtVpred⋃VgtRecallRecall=TPTP+FNAccuracyAccuracy=TP+FNTP+FN+FP+TNVOEVOE=1−Vpred⋂VgtVpred⋃Vgt


### 4.2. Evaluation of Pixel-Level Segmentation

Because the pixel-level segmentation methods are discussed above, we mainly introduce comparisons between U-Net [20], the models we proposed, the existing segmentation methods mentioned in Related Works, and the segmentation result of our previous work [3] in this section.


#### 4.2.1. Evaluation of Different BLOCKs

In this part, we make comparisons between different mU-Net-BXs and U-Net on memory requirement, time requirement, and segmentation performance.Memory Requirement: The memory requirements of U-Net and mU-Net-BXs are provided in Table3. As we can see, the memory requirements of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 355 MB, 407 MB, 136 MB, and 103 MB, respectively. Obviously, mU-Net-B3 has the lowest memory requirement.Table 3
The memory requirements of U-Net and mU-Net-BXs.
ModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Memory requirement355 MB407 MB136 MB103 MBTime Requirement: For 840 training images and 210 testing images, the time requirements of U-Net and these improved models, which include training and average testing time, are provided in Table4. The training time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 35.756 minutes, 78.235 minutes, 28.53 minutes, and 36.521 minutes, respectively. The average testing time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 0.045 seconds, 0.134 seconds, 0.091 seconds, and 0.148 seconds, respectively. We can find that all these networks have a short test time that is less than 0.15 s, showing their feasibility in the practical EM image segmentation task.Table 4
The memory requirements of U-Net and mU-Net-BXs.
Memory requirementModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Training35.76 min78.24 min28.53 min36.52 minAverage testing0.045 s0.134 s0.091 s0.148 sSegmentation Performance: As the workflow is shown in Figure1, the evaluation indexes of all improved models are provided with denseCRF as the postprocessing. The overall segmentation performance of U-Net and these improved models are shown in Figure 16. As we can see, all the improved models make better performance than U-Net. Compared with U-Net, the average Dice values of all the improved models are increased by more than 1.8%, and in particular, the improvements of mU-Net-B1 and mU-Net-B2 are more than 2%. The average Jaccard values of mU-Net-B1, mU-Net-B2, and mU-Net-B3 make 2.89%, 2.75%, and 2.32% improvements, respectively. Likewise, the improvements of the average Recall values made by these improved models are 4.98%, 4.91%, and 4.85%, respectively, and for the average Accuracy values, the improvements of these improved models are 0.65%, 0.34%, and 0.15%, respectively. The average VOE values of the improved models are reduced by 2.89%, 2.75%, and 2.32%, respectively.Figure 16
The average evaluation indexes of U-Net and mU-Net-BXs with denseCRF.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3Summary: From the above, we can find that all the improved models make better segmentation performance than U-Net. Compared with mU-Net-B1 and mU-Net-B2, mU-Net-B3 has the lowest memory requirement, relatively low time requirement, and the similar performance, so it has a big potential in the EM image segmentation work.After evaluating the overall performance of these methods, we also provide the detailed indexes and segmentation result examples of each category of EM under these methods in Table5 and Figure 17, respectively.Table 5
The average indexes for each category of EM generated by U-Net, mU-Net-BXs. For short, in this table, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are abbreviated as B1, B2, and B3, respectively.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1U-Net71.857.4759.1397.5342.538U-Net92.7787.4693.7497.8412.5415U-Net93.0687.2391.3697.3812.77B171.356.7659.1997.4143.24B194.0589.3994.998.1910.61B192.9687.2692.2697.4612.74B272.0457.7759.4697.5642.23B294.5189.9895.898.2910.02B293.7188.392.1297.7211.7B372.1657.8659.5297.5742.14B394.599096.9798.2310B393.1287.2591.6297.4912.752U-Net94.8791.1892.5497.58.829U-Net86.5780.9883.5197.5519.0216U-Net89.882.2384.4197.5917.77B197.4795.2498.2898.634.76B188.9982.2388.7297.1117.77B191.985.4387.1898.0614.57B295.0491.1596.5297.198.85B292.0486.1789.0298.1513.83B292.6886.6286.8398.1713.38B397.6995.6998.6798.714.31B389.3882.4990.8897.0117.51B392.486.1887.9398.2213.823U-Net94.0688.8692.1999.711.1410U-Net91.3684.395.1398.9715.717U-Net89.2883.1694.2297.9516.84B193.0287.0993.1199.6212.91B193.9788.7995.5299.0211.21B186.7679.8795.4697.8420.13B294.2589.2593.4499.7110.75B294.1789.2395.8798.9410.77B285.7779.1595.1897.4820.85B394.389.3892.8299.7110.62B394.7190.0195.5299.349.99B384.9478.9995.2296.7921.014U-Net48.8338.2444.2496.6461.7611U-Net88.4882.4783.792.2717.5318U-Net93.0887.2788.1695.0712.73B158.1644.957.0297.1255.1B196.5193.4195.3197.196.59B194.8990.3591.6996.489.65B260.7647.6657.6897.3152.34B296.192.6394.6796.917.37B294.2389.2190.3596.0710.79B359.246.2960.4496.6853.71B392.0886.3587.4993.9513.65B394.1288.9990.1895.8211.015U-Net87.4678.8391.1997.2521.1712U-Net83.3273.2176.6396.9126.7919U-Net91.5685.285.4398.3714.8B186.3877.7495.298.0922.26B181.672.1179.5796.6527.89B193.6388.3788.6698.7211.63B285.4477.8894.7297.922.12B282.8973.4582.7396.8626.55B293.3787.6887.9598.4112.32B382.2872.9391.7497.2227.07B386.7877.6380.4997.5322.37B390.9784.3685.1297.9915.646U-Net55.4340.5650.048959.4413U-Net88.7680.6384.597.2519.3720U-Net80.0168.7270.0693.5731.28B169.2953.8972.6990.7446.11B192.2385.891.698.0214.2B189.7982.383.2296.1717.7B263.748.6275.0686.9351.38B287.1979.8691.1395.2220.14B289.4381.5983.2596.1918.41B364.0148.9476.7487.1151.06B389.8783.2492.6196.4416.76B387.3279.1180.8295.2720.897U-Net90.1182.9393.8298.4117.0714U-Net84.6274.5283.3197.7625.4821U-Net94.8690.3290.4997.529.68B188.5380.5997.2197.9919.41B183.674.8584.4597.625.15B194.7190.0890.9397.439.92B290.3382.9996.6698.4417.01B279.9773.2180.3697.8526.79B295.3891.2692.0297.88.74B388.9681.0498.2198.1618.96B385.6876.6984.5898.0623.31B395.2491.0191.9197.738.99Figure 17
An example of GT images and segmentation results for each category of EMs by U-Net, U-Net-B1, U-Net-B2, and U-Net-B3 (from the left to the right).


#### 4.2.2. Comparison with Other Methods

In this part, we conduct some comparative experiments on the segmentation of EM. During the experiments, we mainly adopt some representative segmentation methods mentioned inRelated Works, including Otsu, Canny, Watershed, MRF, and k-means. During the experiments, because the results are often insufficient, we need some postprocessing for the results. To show better segmentation results of these methods, we uniformly use the same postprocessing operations. To evaluate the overall performance of these methods, we provide the average evaluation indexes of these methods in Figure 18.Figure 18
The average evaluation indexes of Otsu, Canny, Watershed,k-means, and MRF.
(a)
Otsu(b)
Canny(c)
Watershed(d)
k-means(e)
MRFFrom Figure18, we can find none of the methods performs as well as the proposed methods. But we can find that the recall values in Figure 18 are higher than the recall values in Figure 16. This is because some of the segmentation results generated by these methods have a lot of background parts divided into the foreground. From Table 2, we can realize that as long as the foreground in the segmentation result contains the entire real foreground in GT images, the value of recall is 1 regardless of whether the oversegmentation problem is existing or not. Therefore, we should not judge the segmentation results by Recall alone.To better observe the performance of these methods, we provide the detailed indexes of the segmentation results of each category of EM under these methods in Table6. Besides, we also provide examples of the segmentation results under these methods in Figure 19.Table 6
The average indexes for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1Otsu31.1224.8881.9641.4175.128Otsu48.3738.5898.8845.0361.4215Otsu81.897291.8186.428Canny27.4219.3799.1636.5280.63Canny56.8445.7199.0459.0154.29Canny90.2482.6897.5596.3317.32Watershed32.0526.1383.5344.1273.87Watershed64.8555.685.3771.9244.4Watershed86.8777.4187.1795.1422.59k-means14.7310.8567.7440.3789.15k-means14.278.535050.6691.47k-means53.1645.8761.5677.8454.13MRF30.7924.1795.6933.575.83MRF49.840.6990.4448.8659.31MRF7868.0997.838331.912Otsu73.3263.2498.6471.4636.769Otsu70.7762.3791.1782.1737.6316Otsu58.8948.0577.6475.0251.95Canny76.2966.3399.1377.1833.67Canny59.8853.9468.7889.746.06Canny65.4654.8997.1777.6645.11Watershed67.0258.1185.3574.5441.89Watershed69.5357.4173.3591.2542.59Watershed83.9576.8689.0988.9723.14k-means52.4147.468.8870.1852.6k-means70.463.1879.4688.3236.82k-means35.7629.0652.8173.9770.94MRF65.457.3189.9763.642.69MRF62.2954.1996.4872.2445.81MRF72.8862.6586.6582.2337.353Otsu4.492.3189.4914.3197.6910Otsu40.0230.449249.1269.5617Otsu30.0423.5297.329.8476.48Canny13.910.1499.9625.3389.86Canny39.6729.8299.7851.9970.18Canny68.8160.5188.1987.9539.49Watershed12.229.8769.1144.0290.13Watershed52.7946.3381.2969.1953.67Watershed75.7170.4584.891.429.55k-means4.382.259011.9797.75k-means26.8321.0560.1158.3278.95k-means22.8816.5489.2326.3583.46MRF4.372.249012.897.76MRF39.9930.4999.7749.6669.51MRF71.5762.0693.7180.5737.944Otsu4.492.329011.3297.6811Otsu62.3347.798.3153.0852.318Otsu73.2563.8993.3168.7836.11Canny7.083.7696.7817.3196.24Canny96.3393.0898.6997.266.92Canny93.3487.7191.5795.4612.29Watershed7.114.1762.741.2895.83Watershed81.3170.8176.3888.3429.19Watershed90.5683.1690.2793.616.84k-means4.492.329011.3397.68k-means37.2326.1564.8146.3773.85k-means44.0935.6564.457.6464.35MRF4.672.419013.1797.59MRF75.8965.4991.3974.9934.51MRF71.0459.0774.3777.6540.935Otsu31.0924.6896.4529.6375.3212Otsu71.3859.378.3587.1540.719Otsu40.2833.4781.9948.4866.53Canny38.8433.1699.9150.2466.84Canny81.0170.994.0495.3429.1Canny57.1550.7997.1661.4449.21Watershed35.9428.9282.4458.5971.08Watershed58.0344.5649.5692.155.44Watershed63.6256.875.3281.5743.2k-means14.1710.1161.9241.0689.89k-means49.3542.7846.2892.7557.22k-means40.6433.9880.8349.7666.02MRF27.9522.3196.1830.0177.69MRF71.4162.595.1777.937.5MRF53.1548.0695.2952.4651.946Otsu24.7214.7590.0122.9885.2513Otsu51.6641.9293.1460.6258.0820Otsu45.8437.6266.3365.4962.38Canny36.8723.5699.2539.8176.44Canny72.9659.9498.689.640.06Canny65.6555.6695.174.5344.34Watershed30.7519.2581.7439.3480.75Watershed58.6645.7366.2686.7154.27Watershed53.9441.8464.5773.4558.16k-means24.7214.7590.0122.9985.25k-means32.1924.8866.8963.7675.12k-means21.6516.8643.6362.2783.14MRF23.9514.480.0132.3285.6MRF46.3437.6763.3470.3462.33MRF63.2554.975.3672.2945.17Otsu47.3941.7986.366.3758.2114Otsu50.5440.8896.866.859.1221Otsu83.7875.289.9686.0924.8Canny50.2940.1999.4964.5959.81Canny58.6648.9278.6291.0351.08Canny94.6189.9596.997.2910.05Watershed63.9655.6279.3979.3144.38Watershed50.6341.2462.4487.2158.76Watershed74.0969.2471.3790.1530.76k-means5.93.4626.1973.996.54k-means43.2635.9667.4881.2664.04k-means58.8452.8859.8784.8247.12MRF34.4926.4786.9550.0673.53MRF37.3928.7593.2948.9871.25MRF75.4967.6298.9371.0332.38Figure 19
An example of GT images and segmentation results for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF methods (from the left to the right).


#### 4.2.3. Comparison with our Previous Work

In our previous work [3], the EMDS-4 data set we used contains only 20 categories. The 17th category (Gymnodinium), which is used in this paper, is excluded from our previous work. Besides, we only use Average Recall and Overall Accuracy to evaluate the segmentation performance in our previous work. Therefore, we provide the evaluation indexes of the segmentation results obtained by mU-Net-B3 with denseCRF without the 17th category. Furthermore, in our previous work, there are six models for segmentation: Per-pixel RF (noEdges), CRF with Potts pairwise potentials (Potts), CRF with contrast-sensitive Potts model (PottsCS), fully connected CRF with Gaussian pairwise potentials (denseCRF), fully connected CRF on segmentation results by the original DeepLab method [23] (denseCRForg), and fully convolutional network (FCN). We provide the Average Recall and Overall Accuracy values of mU-Net-B3 with denseCRF as postprocessing and our previous models in Figure 20. It can be found from Figure 20 that compared with the previous models, the Average Recall is improved by more than 7% and the increase of Overall Accuracy is by at least 1%. From that, we can realize mU-Net-B3 with denseCRF we proposed in this paper performs better than the models in our previous work.Figure 20
The average recall and overall accuracy of mU-Net-B3 with denseCRF and our previous models.
(a)
Average recall(b)
Overall accuracy


### 4.3. Evaluation of Patch-Level Segmentation

Although mU-Net-B3 with Dense CRF performs well on the segmentation task for most categories of EM, there are still some shortages. For example, as the results ofColpoda shown in Figure 21, mU-Net-B3 is not able to segment the whole object, leading to an undersegmentation result. Therefore, we use patch-level segmentation to make up this shortage.Figure 21
Examples for proving the validity of combining pixel-level segmentation with patch-level segmentation. From top to bottom, images in each EM represent GT image, pixel-level segmentation result, patch-level segmentation result, and combined result, respectively.


#### 4.3.1. The Criterion for Assigning the Labels

In this part, we mainly discuss the criterion for assigning the labels to the patch in training and validation data sets and the determination of buffer size. As we mentioned above, we divide the patches into two categories: (With Object) and (Without Object). The criterion for assigning these two labels to the patch is whether the area of the object is more than half of the total area of the patch. There are two reasons for using the half area as the criterion. The first reason is that when we choose 0.25 area and 0.75 area as the criteria, the results do not make much difference. This is because when we, respectively, use these three criteria, the number of patches in the two categories varies so little. We provide detailed numbers of patches in the two categories under different criteria in Table7. It means that most patches that contain objects are divided into (With Object). The second reason is that it can show the lowest loss and the highest accuracy on the validation data set when compared with 0.25 and 0.75 areas, respectively. The loss values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 26.74%, 26.37%, and 27.38%, respectively. The accuracy values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 90.24%, 90.33%, and 90.12%, respectively. Besides, we provide some segmentation results under different criteria as examples in Figure 22.Table 7
The number of patches in different categories under different criteria.
CategoryCriterion0.250.50.75(With Object)206701857516823(Without Object)868508894590697Figure 22
Patch-level segmentation results under different criteria.


#### 4.3.2. The Determination of Buffer Size

From Figure22, we can find that the patch-level segmentation results contain a lot of noises around the objects we need to segment. We only want to retain the useful parts of the patch-level segmentation results and remove the useless parts. The direct way is establishing buffers near the pixel-level segmentation results. The challenge is how to set the size of the buffer. The solution we propose is combining the patch-level segmentation results under different buffer size settings with pixel-level segmentation results and comparing the combined results with GT images to determine the size of the final buffer based on the performance of evaluation indexes. Furthermore, we make a comparison between the buffers of different sizes. It starts with a buffer size of 2 pixels and gradually increases the buffer size by 2 pixels until the buffer size is 40 pixels. After that, the patch-level segmentation results after different buffer processing are combined with the pixel-level segmentation results. Finally, the combined results are compared with GT images to obtain relevant evaluation indexes, which are shown in Figure 23. We determine the buffer area size corresponding to the intersection point of Accuracy and Recall in Figure 23 as the final buffer size setting. The buffer size corresponding to the intersection point is 26 pixels. Besides, we provide the patch-level segmentation results in the form of fluorescent green masks in Figure 24.Figure 23
The evaluation indexes of the combined results under different buffers.Figure 24
Examples of patch-level segmentation results with buffer.


### 4.4. Evaluation of Combined Segmentation Results

To observe the advantages of combining patch-level segmentation with pixel-level segmentation better, we provide some examples and their corresponding evaluation indexes in Figures21 and 25, respectively. We can find that patch-level segmentation effectively helps to improve the shortage of pixel-level segmentation.Figure 25
The evaluation indexes for combined segmentation results.


### 4.5. Segmentation Result Fusion and Presentation

Finally, we provide the combined results of patch-level segmentation results and pixel-level segmentation results in Figure13. The yellow parts in the images are the overlapping areas of the patch-level segmentation results (fluorescent green parts) and pixel-level segmentation results (red parts). The purple outline plotted on the images is the GT images.


## 4.1. Experimental Setting

### 4.1.1. Image Data Set

In our work, we useEnvironmental Microorganism Data Set 5th Version (EMDS-5), which is a newly released version of EMDS series [32], containing 21 EM classes as shown in Figure 14. Each EM class contains 20 original microscopic images and their corresponding GT images, thus the data set includes 420 scenes. Owing to the microscopic images having multifarious sizes, we convert all the image sizes into 256×256 pixels uniformly.Figure 14
Examples of the images in EMDS-5. (a) shows the original EM images and (b) shows the corresponding GT images.
(a)
Original Image(b)
Ground truth image


### 4.1.2. Training, Validation, and Test Data Setting

Due to the different living conditions and habits of EMs, it is difficult to obtain a large number of EM images for our EMDS-5 [8]. To observe the improvements made by the optimized models, a large amount of testing images is needed. Therefore, we randomly divide each class of EMDS-5 into a training data set, validation data set, and test data set in a ratio of 1 : 1 : 2. Furthermore, because of the limitation of EMDS-5, data augmentation is used in our pixel-level training. Inspired by the strategy proposed in [19], we augment the 105 training images with rotations by 0, 90, 180, 270 degrees, and mirroring, which results in 840 images for training. In our patch-level training, we mesh 105 training images and their corresponding GT images into patches (8×8 pixels), and 107520 patches are obtained. These patches are divided into two categories: (With Object) and (Without Object). We find that the numbers of patches in these two categories are inconsistent. The first category (With Object) has 18575 patches, and another category (Without Object) has 88945 patches. To resolve this situation, we employ data augmentation to the first category (With Object). We augment the 18575 patches in the first category (Without Object) with rotations by 0, 90, 180, 270 degrees, and mirroring, which result in 148600 patches. Then, we randomly choose 88945 patches to replace the data in the first category (With Object).


### 4.1.3. Experimental Environment

The experiment is conducted by Python 3. The models are implemented using Keras [31] framework with Tensorflow [33] as backend. In our experiment, we use a workstation with Intel(R) Core(TM) i7-8700 CPU with 3.20 GHz, 32GB RAM, and NVIDIA GEFORCE RTX 2080 8 GB.


### 4.1.4. Pixel-Level Implementation Details

In our pixel-level segmentation, the task of the segmentation is to predict the individual pixels whether they represent a point of foreground or background. Actually, this task can be seen as a pixel-level binary classification problem. Hence, as the loss function of the network, we simply take the binary cross-entropy function and minimize it [26]. Besides, we use Adam optimizer with 1.5×10−4 learning rate in our training process. The models are trained for 50 epochs using Adam optimizer. As the average training loss and accuracy curve of the training process is shown in Figure 15, we can find that the loss and accuracy curves of training and validation tend to level off after 30-35 iterations. Therefore, considering the computing performance of the workstation, we finally set 50 epochs for training.Figure 15
The loss and accuracy curves of the training process.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3


### 4.1.5. Patch-Level Implementation Details

In our patch-level training process, we employ the pretrained VGG-16 as the core and fine-tune the dense layers of VGG-16. As Figure4 shows, the last layer is softmax. The categorical cross-entropy function is the loss function of choice for softmax output units. Besides, Adam optimizer with 1.0×10−4 learning rate is used in VGG-16. The pretrained model is trained for 15 epochs.


### 4.1.6. Evaluation Metric

In our previous work [3], Recall and Accuracy are used to measure the segmentation results. Besides that, we employ Dice, Jaccard, and VOE (volumetric overlap error) to evaluate the segmentation results in this paper [34]. The definitions of these evaluation metrics are provided in Table 2. Vpred represents the foreground that is predicted by the model. Vgt represents the foreground in a ground truth image. From Table 2, we can find that the higher the values of the first four metrics (Dice, Jaccard, Recall, and Accuracy) are, the better the segmentation results are. On the contrary, the lower the value of the final metric (VOE) is, the better the segmentation result is.Table 2
The definitions of evaluation metrics for image segmentation. TP (True Positive), FN (False Negative), FP (False Positive), and TN (True Negative).
MetricDefinitionMetricDefinitionDiceDice=2×Vpred⋂VgtVpred+VgtJaccardJaccard=Vpred⋂VgtVpred⋃VgtRecallRecall=TPTP+FNAccuracyAccuracy=TP+FNTP+FN+FP+TNVOEVOE=1−Vpred⋂VgtVpred⋃Vgt


## 4.1.1. Image Data Set

In our work, we useEnvironmental Microorganism Data Set 5th Version (EMDS-5), which is a newly released version of EMDS series [32], containing 21 EM classes as shown in Figure 14. Each EM class contains 20 original microscopic images and their corresponding GT images, thus the data set includes 420 scenes. Owing to the microscopic images having multifarious sizes, we convert all the image sizes into 256×256 pixels uniformly.Figure 14
Examples of the images in EMDS-5. (a) shows the original EM images and (b) shows the corresponding GT images.
(a)
Original Image(b)
Ground truth image


## 4.1.2. Training, Validation, and Test Data Setting

Due to the different living conditions and habits of EMs, it is difficult to obtain a large number of EM images for our EMDS-5 [8]. To observe the improvements made by the optimized models, a large amount of testing images is needed. Therefore, we randomly divide each class of EMDS-5 into a training data set, validation data set, and test data set in a ratio of 1 : 1 : 2. Furthermore, because of the limitation of EMDS-5, data augmentation is used in our pixel-level training. Inspired by the strategy proposed in [19], we augment the 105 training images with rotations by 0, 90, 180, 270 degrees, and mirroring, which results in 840 images for training. In our patch-level training, we mesh 105 training images and their corresponding GT images into patches (8×8 pixels), and 107520 patches are obtained. These patches are divided into two categories: (With Object) and (Without Object). We find that the numbers of patches in these two categories are inconsistent. The first category (With Object) has 18575 patches, and another category (Without Object) has 88945 patches. To resolve this situation, we employ data augmentation to the first category (With Object). We augment the 18575 patches in the first category (Without Object) with rotations by 0, 90, 180, 270 degrees, and mirroring, which result in 148600 patches. Then, we randomly choose 88945 patches to replace the data in the first category (With Object).


## 4.1.3. Experimental Environment

The experiment is conducted by Python 3. The models are implemented using Keras [31] framework with Tensorflow [33] as backend. In our experiment, we use a workstation with Intel(R) Core(TM) i7-8700 CPU with 3.20 GHz, 32GB RAM, and NVIDIA GEFORCE RTX 2080 8 GB.


## 4.1.4. Pixel-Level Implementation Details

In our pixel-level segmentation, the task of the segmentation is to predict the individual pixels whether they represent a point of foreground or background. Actually, this task can be seen as a pixel-level binary classification problem. Hence, as the loss function of the network, we simply take the binary cross-entropy function and minimize it [26]. Besides, we use Adam optimizer with 1.5×10−4 learning rate in our training process. The models are trained for 50 epochs using Adam optimizer. As the average training loss and accuracy curve of the training process is shown in Figure 15, we can find that the loss and accuracy curves of training and validation tend to level off after 30-35 iterations. Therefore, considering the computing performance of the workstation, we finally set 50 epochs for training.Figure 15
The loss and accuracy curves of the training process.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3


## 4.1.5. Patch-Level Implementation Details

In our patch-level training process, we employ the pretrained VGG-16 as the core and fine-tune the dense layers of VGG-16. As Figure4 shows, the last layer is softmax. The categorical cross-entropy function is the loss function of choice for softmax output units. Besides, Adam optimizer with 1.0×10−4 learning rate is used in VGG-16. The pretrained model is trained for 15 epochs.


## 4.1.6. Evaluation Metric

In our previous work [3], Recall and Accuracy are used to measure the segmentation results. Besides that, we employ Dice, Jaccard, and VOE (volumetric overlap error) to evaluate the segmentation results in this paper [34]. The definitions of these evaluation metrics are provided in Table 2. Vpred represents the foreground that is predicted by the model. Vgt represents the foreground in a ground truth image. From Table 2, we can find that the higher the values of the first four metrics (Dice, Jaccard, Recall, and Accuracy) are, the better the segmentation results are. On the contrary, the lower the value of the final metric (VOE) is, the better the segmentation result is.Table 2
The definitions of evaluation metrics for image segmentation. TP (True Positive), FN (False Negative), FP (False Positive), and TN (True Negative).
MetricDefinitionMetricDefinitionDiceDice=2×Vpred⋂VgtVpred+VgtJaccardJaccard=Vpred⋂VgtVpred⋃VgtRecallRecall=TPTP+FNAccuracyAccuracy=TP+FNTP+FN+FP+TNVOEVOE=1−Vpred⋂VgtVpred⋃Vgt


## 4.2. Evaluation of Pixel-Level Segmentation

Because the pixel-level segmentation methods are discussed above, we mainly introduce comparisons between U-Net [20], the models we proposed, the existing segmentation methods mentioned in Related Works, and the segmentation result of our previous work [3] in this section.


### 4.2.1. Evaluation of Different BLOCKs

In this part, we make comparisons between different mU-Net-BXs and U-Net on memory requirement, time requirement, and segmentation performance.Memory Requirement: The memory requirements of U-Net and mU-Net-BXs are provided in Table3. As we can see, the memory requirements of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 355 MB, 407 MB, 136 MB, and 103 MB, respectively. Obviously, mU-Net-B3 has the lowest memory requirement.Table 3
The memory requirements of U-Net and mU-Net-BXs.
ModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Memory requirement355 MB407 MB136 MB103 MBTime Requirement: For 840 training images and 210 testing images, the time requirements of U-Net and these improved models, which include training and average testing time, are provided in Table4. The training time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 35.756 minutes, 78.235 minutes, 28.53 minutes, and 36.521 minutes, respectively. The average testing time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 0.045 seconds, 0.134 seconds, 0.091 seconds, and 0.148 seconds, respectively. We can find that all these networks have a short test time that is less than 0.15 s, showing their feasibility in the practical EM image segmentation task.Table 4
The memory requirements of U-Net and mU-Net-BXs.
Memory requirementModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Training35.76 min78.24 min28.53 min36.52 minAverage testing0.045 s0.134 s0.091 s0.148 sSegmentation Performance: As the workflow is shown in Figure1, the evaluation indexes of all improved models are provided with denseCRF as the postprocessing. The overall segmentation performance of U-Net and these improved models are shown in Figure 16. As we can see, all the improved models make better performance than U-Net. Compared with U-Net, the average Dice values of all the improved models are increased by more than 1.8%, and in particular, the improvements of mU-Net-B1 and mU-Net-B2 are more than 2%. The average Jaccard values of mU-Net-B1, mU-Net-B2, and mU-Net-B3 make 2.89%, 2.75%, and 2.32% improvements, respectively. Likewise, the improvements of the average Recall values made by these improved models are 4.98%, 4.91%, and 4.85%, respectively, and for the average Accuracy values, the improvements of these improved models are 0.65%, 0.34%, and 0.15%, respectively. The average VOE values of the improved models are reduced by 2.89%, 2.75%, and 2.32%, respectively.Figure 16
The average evaluation indexes of U-Net and mU-Net-BXs with denseCRF.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3Summary: From the above, we can find that all the improved models make better segmentation performance than U-Net. Compared with mU-Net-B1 and mU-Net-B2, mU-Net-B3 has the lowest memory requirement, relatively low time requirement, and the similar performance, so it has a big potential in the EM image segmentation work.After evaluating the overall performance of these methods, we also provide the detailed indexes and segmentation result examples of each category of EM under these methods in Table5 and Figure 17, respectively.Table 5
The average indexes for each category of EM generated by U-Net, mU-Net-BXs. For short, in this table, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are abbreviated as B1, B2, and B3, respectively.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1U-Net71.857.4759.1397.5342.538U-Net92.7787.4693.7497.8412.5415U-Net93.0687.2391.3697.3812.77B171.356.7659.1997.4143.24B194.0589.3994.998.1910.61B192.9687.2692.2697.4612.74B272.0457.7759.4697.5642.23B294.5189.9895.898.2910.02B293.7188.392.1297.7211.7B372.1657.8659.5297.5742.14B394.599096.9798.2310B393.1287.2591.6297.4912.752U-Net94.8791.1892.5497.58.829U-Net86.5780.9883.5197.5519.0216U-Net89.882.2384.4197.5917.77B197.4795.2498.2898.634.76B188.9982.2388.7297.1117.77B191.985.4387.1898.0614.57B295.0491.1596.5297.198.85B292.0486.1789.0298.1513.83B292.6886.6286.8398.1713.38B397.6995.6998.6798.714.31B389.3882.4990.8897.0117.51B392.486.1887.9398.2213.823U-Net94.0688.8692.1999.711.1410U-Net91.3684.395.1398.9715.717U-Net89.2883.1694.2297.9516.84B193.0287.0993.1199.6212.91B193.9788.7995.5299.0211.21B186.7679.8795.4697.8420.13B294.2589.2593.4499.7110.75B294.1789.2395.8798.9410.77B285.7779.1595.1897.4820.85B394.389.3892.8299.7110.62B394.7190.0195.5299.349.99B384.9478.9995.2296.7921.014U-Net48.8338.2444.2496.6461.7611U-Net88.4882.4783.792.2717.5318U-Net93.0887.2788.1695.0712.73B158.1644.957.0297.1255.1B196.5193.4195.3197.196.59B194.8990.3591.6996.489.65B260.7647.6657.6897.3152.34B296.192.6394.6796.917.37B294.2389.2190.3596.0710.79B359.246.2960.4496.6853.71B392.0886.3587.4993.9513.65B394.1288.9990.1895.8211.015U-Net87.4678.8391.1997.2521.1712U-Net83.3273.2176.6396.9126.7919U-Net91.5685.285.4398.3714.8B186.3877.7495.298.0922.26B181.672.1179.5796.6527.89B193.6388.3788.6698.7211.63B285.4477.8894.7297.922.12B282.8973.4582.7396.8626.55B293.3787.6887.9598.4112.32B382.2872.9391.7497.2227.07B386.7877.6380.4997.5322.37B390.9784.3685.1297.9915.646U-Net55.4340.5650.048959.4413U-Net88.7680.6384.597.2519.3720U-Net80.0168.7270.0693.5731.28B169.2953.8972.6990.7446.11B192.2385.891.698.0214.2B189.7982.383.2296.1717.7B263.748.6275.0686.9351.38B287.1979.8691.1395.2220.14B289.4381.5983.2596.1918.41B364.0148.9476.7487.1151.06B389.8783.2492.6196.4416.76B387.3279.1180.8295.2720.897U-Net90.1182.9393.8298.4117.0714U-Net84.6274.5283.3197.7625.4821U-Net94.8690.3290.4997.529.68B188.5380.5997.2197.9919.41B183.674.8584.4597.625.15B194.7190.0890.9397.439.92B290.3382.9996.6698.4417.01B279.9773.2180.3697.8526.79B295.3891.2692.0297.88.74B388.9681.0498.2198.1618.96B385.6876.6984.5898.0623.31B395.2491.0191.9197.738.99Figure 17
An example of GT images and segmentation results for each category of EMs by U-Net, U-Net-B1, U-Net-B2, and U-Net-B3 (from the left to the right).


### 4.2.2. Comparison with Other Methods

In this part, we conduct some comparative experiments on the segmentation of EM. During the experiments, we mainly adopt some representative segmentation methods mentioned inRelated Works, including Otsu, Canny, Watershed, MRF, and k-means. During the experiments, because the results are often insufficient, we need some postprocessing for the results. To show better segmentation results of these methods, we uniformly use the same postprocessing operations. To evaluate the overall performance of these methods, we provide the average evaluation indexes of these methods in Figure 18.Figure 18
The average evaluation indexes of Otsu, Canny, Watershed,k-means, and MRF.
(a)
Otsu(b)
Canny(c)
Watershed(d)
k-means(e)
MRFFrom Figure18, we can find none of the methods performs as well as the proposed methods. But we can find that the recall values in Figure 18 are higher than the recall values in Figure 16. This is because some of the segmentation results generated by these methods have a lot of background parts divided into the foreground. From Table 2, we can realize that as long as the foreground in the segmentation result contains the entire real foreground in GT images, the value of recall is 1 regardless of whether the oversegmentation problem is existing or not. Therefore, we should not judge the segmentation results by Recall alone.To better observe the performance of these methods, we provide the detailed indexes of the segmentation results of each category of EM under these methods in Table6. Besides, we also provide examples of the segmentation results under these methods in Figure 19.Table 6
The average indexes for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1Otsu31.1224.8881.9641.4175.128Otsu48.3738.5898.8845.0361.4215Otsu81.897291.8186.428Canny27.4219.3799.1636.5280.63Canny56.8445.7199.0459.0154.29Canny90.2482.6897.5596.3317.32Watershed32.0526.1383.5344.1273.87Watershed64.8555.685.3771.9244.4Watershed86.8777.4187.1795.1422.59k-means14.7310.8567.7440.3789.15k-means14.278.535050.6691.47k-means53.1645.8761.5677.8454.13MRF30.7924.1795.6933.575.83MRF49.840.6990.4448.8659.31MRF7868.0997.838331.912Otsu73.3263.2498.6471.4636.769Otsu70.7762.3791.1782.1737.6316Otsu58.8948.0577.6475.0251.95Canny76.2966.3399.1377.1833.67Canny59.8853.9468.7889.746.06Canny65.4654.8997.1777.6645.11Watershed67.0258.1185.3574.5441.89Watershed69.5357.4173.3591.2542.59Watershed83.9576.8689.0988.9723.14k-means52.4147.468.8870.1852.6k-means70.463.1879.4688.3236.82k-means35.7629.0652.8173.9770.94MRF65.457.3189.9763.642.69MRF62.2954.1996.4872.2445.81MRF72.8862.6586.6582.2337.353Otsu4.492.3189.4914.3197.6910Otsu40.0230.449249.1269.5617Otsu30.0423.5297.329.8476.48Canny13.910.1499.9625.3389.86Canny39.6729.8299.7851.9970.18Canny68.8160.5188.1987.9539.49Watershed12.229.8769.1144.0290.13Watershed52.7946.3381.2969.1953.67Watershed75.7170.4584.891.429.55k-means4.382.259011.9797.75k-means26.8321.0560.1158.3278.95k-means22.8816.5489.2326.3583.46MRF4.372.249012.897.76MRF39.9930.4999.7749.6669.51MRF71.5762.0693.7180.5737.944Otsu4.492.329011.3297.6811Otsu62.3347.798.3153.0852.318Otsu73.2563.8993.3168.7836.11Canny7.083.7696.7817.3196.24Canny96.3393.0898.6997.266.92Canny93.3487.7191.5795.4612.29Watershed7.114.1762.741.2895.83Watershed81.3170.8176.3888.3429.19Watershed90.5683.1690.2793.616.84k-means4.492.329011.3397.68k-means37.2326.1564.8146.3773.85k-means44.0935.6564.457.6464.35MRF4.672.419013.1797.59MRF75.8965.4991.3974.9934.51MRF71.0459.0774.3777.6540.935Otsu31.0924.6896.4529.6375.3212Otsu71.3859.378.3587.1540.719Otsu40.2833.4781.9948.4866.53Canny38.8433.1699.9150.2466.84Canny81.0170.994.0495.3429.1Canny57.1550.7997.1661.4449.21Watershed35.9428.9282.4458.5971.08Watershed58.0344.5649.5692.155.44Watershed63.6256.875.3281.5743.2k-means14.1710.1161.9241.0689.89k-means49.3542.7846.2892.7557.22k-means40.6433.9880.8349.7666.02MRF27.9522.3196.1830.0177.69MRF71.4162.595.1777.937.5MRF53.1548.0695.2952.4651.946Otsu24.7214.7590.0122.9885.2513Otsu51.6641.9293.1460.6258.0820Otsu45.8437.6266.3365.4962.38Canny36.8723.5699.2539.8176.44Canny72.9659.9498.689.640.06Canny65.6555.6695.174.5344.34Watershed30.7519.2581.7439.3480.75Watershed58.6645.7366.2686.7154.27Watershed53.9441.8464.5773.4558.16k-means24.7214.7590.0122.9985.25k-means32.1924.8866.8963.7675.12k-means21.6516.8643.6362.2783.14MRF23.9514.480.0132.3285.6MRF46.3437.6763.3470.3462.33MRF63.2554.975.3672.2945.17Otsu47.3941.7986.366.3758.2114Otsu50.5440.8896.866.859.1221Otsu83.7875.289.9686.0924.8Canny50.2940.1999.4964.5959.81Canny58.6648.9278.6291.0351.08Canny94.6189.9596.997.2910.05Watershed63.9655.6279.3979.3144.38Watershed50.6341.2462.4487.2158.76Watershed74.0969.2471.3790.1530.76k-means5.93.4626.1973.996.54k-means43.2635.9667.4881.2664.04k-means58.8452.8859.8784.8247.12MRF34.4926.4786.9550.0673.53MRF37.3928.7593.2948.9871.25MRF75.4967.6298.9371.0332.38Figure 19
An example of GT images and segmentation results for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF methods (from the left to the right).


### 4.2.3. Comparison with our Previous Work

In our previous work [3], the EMDS-4 data set we used contains only 20 categories. The 17th category (Gymnodinium), which is used in this paper, is excluded from our previous work. Besides, we only use Average Recall and Overall Accuracy to evaluate the segmentation performance in our previous work. Therefore, we provide the evaluation indexes of the segmentation results obtained by mU-Net-B3 with denseCRF without the 17th category. Furthermore, in our previous work, there are six models for segmentation: Per-pixel RF (noEdges), CRF with Potts pairwise potentials (Potts), CRF with contrast-sensitive Potts model (PottsCS), fully connected CRF with Gaussian pairwise potentials (denseCRF), fully connected CRF on segmentation results by the original DeepLab method [23] (denseCRForg), and fully convolutional network (FCN). We provide the Average Recall and Overall Accuracy values of mU-Net-B3 with denseCRF as postprocessing and our previous models in Figure 20. It can be found from Figure 20 that compared with the previous models, the Average Recall is improved by more than 7% and the increase of Overall Accuracy is by at least 1%. From that, we can realize mU-Net-B3 with denseCRF we proposed in this paper performs better than the models in our previous work.Figure 20
The average recall and overall accuracy of mU-Net-B3 with denseCRF and our previous models.
(a)
Average recall(b)
Overall accuracy


## 4.2.1. Evaluation of Different BLOCKs

In this part, we make comparisons between different mU-Net-BXs and U-Net on memory requirement, time requirement, and segmentation performance.Memory Requirement: The memory requirements of U-Net and mU-Net-BXs are provided in Table3. As we can see, the memory requirements of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 355 MB, 407 MB, 136 MB, and 103 MB, respectively. Obviously, mU-Net-B3 has the lowest memory requirement.Table 3
The memory requirements of U-Net and mU-Net-BXs.
ModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Memory requirement355 MB407 MB136 MB103 MBTime Requirement: For 840 training images and 210 testing images, the time requirements of U-Net and these improved models, which include training and average testing time, are provided in Table4. The training time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 35.756 minutes, 78.235 minutes, 28.53 minutes, and 36.521 minutes, respectively. The average testing time of U-Net, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are 0.045 seconds, 0.134 seconds, 0.091 seconds, and 0.148 seconds, respectively. We can find that all these networks have a short test time that is less than 0.15 s, showing their feasibility in the practical EM image segmentation task.Table 4
The memory requirements of U-Net and mU-Net-BXs.
Memory requirementModelU-NetmU-Net-B1mU-Net-B2mU-Net-B3Training35.76 min78.24 min28.53 min36.52 minAverage testing0.045 s0.134 s0.091 s0.148 sSegmentation Performance: As the workflow is shown in Figure1, the evaluation indexes of all improved models are provided with denseCRF as the postprocessing. The overall segmentation performance of U-Net and these improved models are shown in Figure 16. As we can see, all the improved models make better performance than U-Net. Compared with U-Net, the average Dice values of all the improved models are increased by more than 1.8%, and in particular, the improvements of mU-Net-B1 and mU-Net-B2 are more than 2%. The average Jaccard values of mU-Net-B1, mU-Net-B2, and mU-Net-B3 make 2.89%, 2.75%, and 2.32% improvements, respectively. Likewise, the improvements of the average Recall values made by these improved models are 4.98%, 4.91%, and 4.85%, respectively, and for the average Accuracy values, the improvements of these improved models are 0.65%, 0.34%, and 0.15%, respectively. The average VOE values of the improved models are reduced by 2.89%, 2.75%, and 2.32%, respectively.Figure 16
The average evaluation indexes of U-Net and mU-Net-BXs with denseCRF.
(a)
U-Net(b)
mU-Net-B1(c)
mU-Net-B2(d)
mU-Net-B3Summary: From the above, we can find that all the improved models make better segmentation performance than U-Net. Compared with mU-Net-B1 and mU-Net-B2, mU-Net-B3 has the lowest memory requirement, relatively low time requirement, and the similar performance, so it has a big potential in the EM image segmentation work.After evaluating the overall performance of these methods, we also provide the detailed indexes and segmentation result examples of each category of EM under these methods in Table5 and Figure 17, respectively.Table 5
The average indexes for each category of EM generated by U-Net, mU-Net-BXs. For short, in this table, mU-Net-B1, mU-Net-B2, and mU-Net-B3 are abbreviated as B1, B2, and B3, respectively.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1U-Net71.857.4759.1397.5342.538U-Net92.7787.4693.7497.8412.5415U-Net93.0687.2391.3697.3812.77B171.356.7659.1997.4143.24B194.0589.3994.998.1910.61B192.9687.2692.2697.4612.74B272.0457.7759.4697.5642.23B294.5189.9895.898.2910.02B293.7188.392.1297.7211.7B372.1657.8659.5297.5742.14B394.599096.9798.2310B393.1287.2591.6297.4912.752U-Net94.8791.1892.5497.58.829U-Net86.5780.9883.5197.5519.0216U-Net89.882.2384.4197.5917.77B197.4795.2498.2898.634.76B188.9982.2388.7297.1117.77B191.985.4387.1898.0614.57B295.0491.1596.5297.198.85B292.0486.1789.0298.1513.83B292.6886.6286.8398.1713.38B397.6995.6998.6798.714.31B389.3882.4990.8897.0117.51B392.486.1887.9398.2213.823U-Net94.0688.8692.1999.711.1410U-Net91.3684.395.1398.9715.717U-Net89.2883.1694.2297.9516.84B193.0287.0993.1199.6212.91B193.9788.7995.5299.0211.21B186.7679.8795.4697.8420.13B294.2589.2593.4499.7110.75B294.1789.2395.8798.9410.77B285.7779.1595.1897.4820.85B394.389.3892.8299.7110.62B394.7190.0195.5299.349.99B384.9478.9995.2296.7921.014U-Net48.8338.2444.2496.6461.7611U-Net88.4882.4783.792.2717.5318U-Net93.0887.2788.1695.0712.73B158.1644.957.0297.1255.1B196.5193.4195.3197.196.59B194.8990.3591.6996.489.65B260.7647.6657.6897.3152.34B296.192.6394.6796.917.37B294.2389.2190.3596.0710.79B359.246.2960.4496.6853.71B392.0886.3587.4993.9513.65B394.1288.9990.1895.8211.015U-Net87.4678.8391.1997.2521.1712U-Net83.3273.2176.6396.9126.7919U-Net91.5685.285.4398.3714.8B186.3877.7495.298.0922.26B181.672.1179.5796.6527.89B193.6388.3788.6698.7211.63B285.4477.8894.7297.922.12B282.8973.4582.7396.8626.55B293.3787.6887.9598.4112.32B382.2872.9391.7497.2227.07B386.7877.6380.4997.5322.37B390.9784.3685.1297.9915.646U-Net55.4340.5650.048959.4413U-Net88.7680.6384.597.2519.3720U-Net80.0168.7270.0693.5731.28B169.2953.8972.6990.7446.11B192.2385.891.698.0214.2B189.7982.383.2296.1717.7B263.748.6275.0686.9351.38B287.1979.8691.1395.2220.14B289.4381.5983.2596.1918.41B364.0148.9476.7487.1151.06B389.8783.2492.6196.4416.76B387.3279.1180.8295.2720.897U-Net90.1182.9393.8298.4117.0714U-Net84.6274.5283.3197.7625.4821U-Net94.8690.3290.4997.529.68B188.5380.5997.2197.9919.41B183.674.8584.4597.625.15B194.7190.0890.9397.439.92B290.3382.9996.6698.4417.01B279.9773.2180.3697.8526.79B295.3891.2692.0297.88.74B388.9681.0498.2198.1618.96B385.6876.6984.5898.0623.31B395.2491.0191.9197.738.99Figure 17
An example of GT images and segmentation results for each category of EMs by U-Net, U-Net-B1, U-Net-B2, and U-Net-B3 (from the left to the right).


## 4.2.2. Comparison with Other Methods

In this part, we conduct some comparative experiments on the segmentation of EM. During the experiments, we mainly adopt some representative segmentation methods mentioned inRelated Works, including Otsu, Canny, Watershed, MRF, and k-means. During the experiments, because the results are often insufficient, we need some postprocessing for the results. To show better segmentation results of these methods, we uniformly use the same postprocessing operations. To evaluate the overall performance of these methods, we provide the average evaluation indexes of these methods in Figure 18.Figure 18
The average evaluation indexes of Otsu, Canny, Watershed,k-means, and MRF.
(a)
Otsu(b)
Canny(c)
Watershed(d)
k-means(e)
MRFFrom Figure18, we can find none of the methods performs as well as the proposed methods. But we can find that the recall values in Figure 18 are higher than the recall values in Figure 16. This is because some of the segmentation results generated by these methods have a lot of background parts divided into the foreground. From Table 2, we can realize that as long as the foreground in the segmentation result contains the entire real foreground in GT images, the value of recall is 1 regardless of whether the oversegmentation problem is existing or not. Therefore, we should not judge the segmentation results by Recall alone.To better observe the performance of these methods, we provide the detailed indexes of the segmentation results of each category of EM under these methods in Table6. Besides, we also provide examples of the segmentation results under these methods in Figure 19.Table 6
The average indexes for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF.
EMMethodsEvaluation metricsEMMethodsEvaluation metricsEMMethodsEvaluation metricsDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOEDiceJaccardRecallAccuracyVOE1Otsu31.1224.8881.9641.4175.128Otsu48.3738.5898.8845.0361.4215Otsu81.897291.8186.428Canny27.4219.3799.1636.5280.63Canny56.8445.7199.0459.0154.29Canny90.2482.6897.5596.3317.32Watershed32.0526.1383.5344.1273.87Watershed64.8555.685.3771.9244.4Watershed86.8777.4187.1795.1422.59k-means14.7310.8567.7440.3789.15k-means14.278.535050.6691.47k-means53.1645.8761.5677.8454.13MRF30.7924.1795.6933.575.83MRF49.840.6990.4448.8659.31MRF7868.0997.838331.912Otsu73.3263.2498.6471.4636.769Otsu70.7762.3791.1782.1737.6316Otsu58.8948.0577.6475.0251.95Canny76.2966.3399.1377.1833.67Canny59.8853.9468.7889.746.06Canny65.4654.8997.1777.6645.11Watershed67.0258.1185.3574.5441.89Watershed69.5357.4173.3591.2542.59Watershed83.9576.8689.0988.9723.14k-means52.4147.468.8870.1852.6k-means70.463.1879.4688.3236.82k-means35.7629.0652.8173.9770.94MRF65.457.3189.9763.642.69MRF62.2954.1996.4872.2445.81MRF72.8862.6586.6582.2337.353Otsu4.492.3189.4914.3197.6910Otsu40.0230.449249.1269.5617Otsu30.0423.5297.329.8476.48Canny13.910.1499.9625.3389.86Canny39.6729.8299.7851.9970.18Canny68.8160.5188.1987.9539.49Watershed12.229.8769.1144.0290.13Watershed52.7946.3381.2969.1953.67Watershed75.7170.4584.891.429.55k-means4.382.259011.9797.75k-means26.8321.0560.1158.3278.95k-means22.8816.5489.2326.3583.46MRF4.372.249012.897.76MRF39.9930.4999.7749.6669.51MRF71.5762.0693.7180.5737.944Otsu4.492.329011.3297.6811Otsu62.3347.798.3153.0852.318Otsu73.2563.8993.3168.7836.11Canny7.083.7696.7817.3196.24Canny96.3393.0898.6997.266.92Canny93.3487.7191.5795.4612.29Watershed7.114.1762.741.2895.83Watershed81.3170.8176.3888.3429.19Watershed90.5683.1690.2793.616.84k-means4.492.329011.3397.68k-means37.2326.1564.8146.3773.85k-means44.0935.6564.457.6464.35MRF4.672.419013.1797.59MRF75.8965.4991.3974.9934.51MRF71.0459.0774.3777.6540.935Otsu31.0924.6896.4529.6375.3212Otsu71.3859.378.3587.1540.719Otsu40.2833.4781.9948.4866.53Canny38.8433.1699.9150.2466.84Canny81.0170.994.0495.3429.1Canny57.1550.7997.1661.4449.21Watershed35.9428.9282.4458.5971.08Watershed58.0344.5649.5692.155.44Watershed63.6256.875.3281.5743.2k-means14.1710.1161.9241.0689.89k-means49.3542.7846.2892.7557.22k-means40.6433.9880.8349.7666.02MRF27.9522.3196.1830.0177.69MRF71.4162.595.1777.937.5MRF53.1548.0695.2952.4651.946Otsu24.7214.7590.0122.9885.2513Otsu51.6641.9293.1460.6258.0820Otsu45.8437.6266.3365.4962.38Canny36.8723.5699.2539.8176.44Canny72.9659.9498.689.640.06Canny65.6555.6695.174.5344.34Watershed30.7519.2581.7439.3480.75Watershed58.6645.7366.2686.7154.27Watershed53.9441.8464.5773.4558.16k-means24.7214.7590.0122.9985.25k-means32.1924.8866.8963.7675.12k-means21.6516.8643.6362.2783.14MRF23.9514.480.0132.3285.6MRF46.3437.6763.3470.3462.33MRF63.2554.975.3672.2945.17Otsu47.3941.7986.366.3758.2114Otsu50.5440.8896.866.859.1221Otsu83.7875.289.9686.0924.8Canny50.2940.1999.4964.5959.81Canny58.6648.9278.6291.0351.08Canny94.6189.9596.997.2910.05Watershed63.9655.6279.3979.3144.38Watershed50.6341.2462.4487.2158.76Watershed74.0969.2471.3790.1530.76k-means5.93.4626.1973.996.54k-means43.2635.9667.4881.2664.04k-means58.8452.8859.8784.8247.12MRF34.4926.4786.9550.0673.53MRF37.3928.7593.2948.9871.25MRF75.4967.6298.9371.0332.38Figure 19
An example of GT images and segmentation results for each EM category generated by Otsu, Canny, Watershed,k-means, and MRF methods (from the left to the right).


## 4.2.3. Comparison with our Previous Work

In our previous work [3], the EMDS-4 data set we used contains only 20 categories. The 17th category (Gymnodinium), which is used in this paper, is excluded from our previous work. Besides, we only use Average Recall and Overall Accuracy to evaluate the segmentation performance in our previous work. Therefore, we provide the evaluation indexes of the segmentation results obtained by mU-Net-B3 with denseCRF without the 17th category. Furthermore, in our previous work, there are six models for segmentation: Per-pixel RF (noEdges), CRF with Potts pairwise potentials (Potts), CRF with contrast-sensitive Potts model (PottsCS), fully connected CRF with Gaussian pairwise potentials (denseCRF), fully connected CRF on segmentation results by the original DeepLab method [23] (denseCRForg), and fully convolutional network (FCN). We provide the Average Recall and Overall Accuracy values of mU-Net-B3 with denseCRF as postprocessing and our previous models in Figure 20. It can be found from Figure 20 that compared with the previous models, the Average Recall is improved by more than 7% and the increase of Overall Accuracy is by at least 1%. From that, we can realize mU-Net-B3 with denseCRF we proposed in this paper performs better than the models in our previous work.Figure 20
The average recall and overall accuracy of mU-Net-B3 with denseCRF and our previous models.
(a)
Average recall(b)
Overall accuracy


## 4.3. Evaluation of Patch-Level Segmentation

Although mU-Net-B3 with Dense CRF performs well on the segmentation task for most categories of EM, there are still some shortages. For example, as the results ofColpoda shown in Figure 21, mU-Net-B3 is not able to segment the whole object, leading to an undersegmentation result. Therefore, we use patch-level segmentation to make up this shortage.Figure 21
Examples for proving the validity of combining pixel-level segmentation with patch-level segmentation. From top to bottom, images in each EM represent GT image, pixel-level segmentation result, patch-level segmentation result, and combined result, respectively.


### 4.3.1. The Criterion for Assigning the Labels

In this part, we mainly discuss the criterion for assigning the labels to the patch in training and validation data sets and the determination of buffer size. As we mentioned above, we divide the patches into two categories: (With Object) and (Without Object). The criterion for assigning these two labels to the patch is whether the area of the object is more than half of the total area of the patch. There are two reasons for using the half area as the criterion. The first reason is that when we choose 0.25 area and 0.75 area as the criteria, the results do not make much difference. This is because when we, respectively, use these three criteria, the number of patches in the two categories varies so little. We provide detailed numbers of patches in the two categories under different criteria in Table7. It means that most patches that contain objects are divided into (With Object). The second reason is that it can show the lowest loss and the highest accuracy on the validation data set when compared with 0.25 and 0.75 areas, respectively. The loss values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 26.74%, 26.37%, and 27.38%, respectively. The accuracy values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 90.24%, 90.33%, and 90.12%, respectively. Besides, we provide some segmentation results under different criteria as examples in Figure 22.Table 7
The number of patches in different categories under different criteria.
CategoryCriterion0.250.50.75(With Object)206701857516823(Without Object)868508894590697Figure 22
Patch-level segmentation results under different criteria.


### 4.3.2. The Determination of Buffer Size

From Figure22, we can find that the patch-level segmentation results contain a lot of noises around the objects we need to segment. We only want to retain the useful parts of the patch-level segmentation results and remove the useless parts. The direct way is establishing buffers near the pixel-level segmentation results. The challenge is how to set the size of the buffer. The solution we propose is combining the patch-level segmentation results under different buffer size settings with pixel-level segmentation results and comparing the combined results with GT images to determine the size of the final buffer based on the performance of evaluation indexes. Furthermore, we make a comparison between the buffers of different sizes. It starts with a buffer size of 2 pixels and gradually increases the buffer size by 2 pixels until the buffer size is 40 pixels. After that, the patch-level segmentation results after different buffer processing are combined with the pixel-level segmentation results. Finally, the combined results are compared with GT images to obtain relevant evaluation indexes, which are shown in Figure 23. We determine the buffer area size corresponding to the intersection point of Accuracy and Recall in Figure 23 as the final buffer size setting. The buffer size corresponding to the intersection point is 26 pixels. Besides, we provide the patch-level segmentation results in the form of fluorescent green masks in Figure 24.Figure 23
The evaluation indexes of the combined results under different buffers.Figure 24
Examples of patch-level segmentation results with buffer.


## 4.3.1. The Criterion for Assigning the Labels

In this part, we mainly discuss the criterion for assigning the labels to the patch in training and validation data sets and the determination of buffer size. As we mentioned above, we divide the patches into two categories: (With Object) and (Without Object). The criterion for assigning these two labels to the patch is whether the area of the object is more than half of the total area of the patch. There are two reasons for using the half area as the criterion. The first reason is that when we choose 0.25 area and 0.75 area as the criteria, the results do not make much difference. This is because when we, respectively, use these three criteria, the number of patches in the two categories varies so little. We provide detailed numbers of patches in the two categories under different criteria in Table7. It means that most patches that contain objects are divided into (With Object). The second reason is that it can show the lowest loss and the highest accuracy on the validation data set when compared with 0.25 and 0.75 areas, respectively. The loss values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 26.74%, 26.37%, and 27.38%, respectively. The accuracy values of using 0.25 area, 0.5 area, and 0.75 area as the criterion are 90.24%, 90.33%, and 90.12%, respectively. Besides, we provide some segmentation results under different criteria as examples in Figure 22.Table 7
The number of patches in different categories under different criteria.
CategoryCriterion0.250.50.75(With Object)206701857516823(Without Object)868508894590697Figure 22
Patch-level segmentation results under different criteria.


## 4.3.2. The Determination of Buffer Size

From Figure22, we can find that the patch-level segmentation results contain a lot of noises around the objects we need to segment. We only want to retain the useful parts of the patch-level segmentation results and remove the useless parts. The direct way is establishing buffers near the pixel-level segmentation results. The challenge is how to set the size of the buffer. The solution we propose is combining the patch-level segmentation results under different buffer size settings with pixel-level segmentation results and comparing the combined results with GT images to determine the size of the final buffer based on the performance of evaluation indexes. Furthermore, we make a comparison between the buffers of different sizes. It starts with a buffer size of 2 pixels and gradually increases the buffer size by 2 pixels until the buffer size is 40 pixels. After that, the patch-level segmentation results after different buffer processing are combined with the pixel-level segmentation results. Finally, the combined results are compared with GT images to obtain relevant evaluation indexes, which are shown in Figure 23. We determine the buffer area size corresponding to the intersection point of Accuracy and Recall in Figure 23 as the final buffer size setting. The buffer size corresponding to the intersection point is 26 pixels. Besides, we provide the patch-level segmentation results in the form of fluorescent green masks in Figure 24.Figure 23
The evaluation indexes of the combined results under different buffers.Figure 24
Examples of patch-level segmentation results with buffer.


## 4.4. Evaluation of Combined Segmentation Results

To observe the advantages of combining patch-level segmentation with pixel-level segmentation better, we provide some examples and their corresponding evaluation indexes in Figures21 and 25, respectively. We can find that patch-level segmentation effectively helps to improve the shortage of pixel-level segmentation.Figure 25
The evaluation indexes for combined segmentation results.


## 4.5. Segmentation Result Fusion and Presentation

Finally, we provide the combined results of patch-level segmentation results and pixel-level segmentation results in Figure13. The yellow parts in the images are the overlapping areas of the patch-level segmentation results (fluorescent green parts) and pixel-level segmentation results (red parts). The purple outline plotted on the images is the GT images.


## 5. Conclusion and Future Work

In this paper, we propose a multilevel segmentation method for the EM segmentation task, which includes pixel-level segmentation and patch-level segmentation.In our pixel-level segmentation, we propose mU-Net-B3 with denseCRF for EM segmentation. It mainly uses the idea of Inception and the use of concatenate operations to reduce the memory requirement. Besides, it also uses denseCRF to obtain global information to further optimize the segmentation results. The proposed method not only performs better than U-Net but also reduces the memory requirement from 355 MB to 103 MB. In the evaluation of segmentation results generated by this proposed method, the values of evaluation indexes Dice, Jaccard, Recall, Accuracy, and VOE (volume overlap error) are 87.13%, 79.74%, 87.12%, 96.91%, and 20.26%, respectively. Compared with U-Net, the first four indexes are improved by 1.89%, 2.32%, 4.84%, and 0.14%, respectively, and the last index is decreased by 2.32%. Besides, compared with our previous methods in [3], the performance of segmentation results is significantly improved, and the details of indexes are shown in Figure 20.Since the method used in pixel-level segmentation cannot segment some details in the image, we use patch-level segmentation to render assistance to improve it. In the patch-level segmentation, we use transfer learning, which is using our data to fine-tune the pretrained VGG-16, to perform the patch-level segmentation task. We can find from Figure13 that the patch-level segmentation can effectively assist the pixel-level segmentation to cover more details.In our future work, we plan to increase the amount of data in the data set to improve the performance. Meanwhile, we have not optimized the time requirement in pixel-level segmentation yet, but we will adjust the relevant parameters to reduce the time requirement.



---

*Source: 4621403-2020-07-08.xml*