# Segmentation of Overlapped Cervical Cells Using Asymmetric Mixture Model and Shape Constraint Level Set Method

Authors: Yiming Huang; Hongqing Zhu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Accuracy segmentation of the nuclei and cytoplasm in Pap smear images is challenging in cervix cytological analysis. In this paper, a new fusion algorithm combining the asymmetric generalized Gaussian and Cauchy mixture model (GGCMM) with a shape constraint level set method to segment overlapping cervical smear cells is put forward. The proposed approach starts by separating nuclei and cytoplasm cluster through asymmetric GGCMM, where each component is a mixture of generalized Gaussian distribution and Cauchy distribution. The proposed asymmetric GGCMM takes into account the asymmetry of generalized Gaussian distribution and the heavier tail of Cauchy distribution. New probability distribution fits different shapes of observed data more flexibly. Then, we apply the morphological operation to remove fake nuclei which is usually much smaller than real nuclei. After that, the improved level set energy function with a distance map and a new shape prior term are applied to extract the contours of overlapping cervical cells. Due to this new level set energy function, the segmentation of every individual cell worked well, especially in overlapping areas. We evaluate the proposed method by using the ISBI 2014 Challenge Dataset. The results demonstrate that our approach outperforms existing methods in extracting overlapping cervical cells and obtains accurate cell contours.


---

## Body

## 1. Introduction

Cervical cancer is a disease caused by malignant cells formed in the tissues of the cervix. In the early stage of cervical cancer, there are no obvious symptoms of the disease. In spite of this, it can be detected with the Papanicolaou test (also known as the Pap smear test) combined with a regular screening program and an appropriate follow-up. However, locating abnormal cells in thousands of cells is still a tough and endless work for pathologists. And results of this manual recognition may be different for different observers, which may lead to misdiagnosis. Therefore, a computer-aided diagnosis system can help to estimate whether cytopathy occurs on cervical cells accurately and efficiently. The precise segmentation of cervical nuclei and cytoplasm is a crucial part of this system.Recent computer-aided diagnosis systems mainly aim to segment nuclei and cellular clusters. Automatic thresholding, morphological operations [1], watershed [2], and active contour models [3] are primary methods for nuclei or cellular mass segmentation. These approaches assume that cervical cells do not overlap or in a slight degree. They failed to isolate individual cytoplasm well. However, real cervical cells usually have a higher overlapping degree. In addition, the segmentation accuracy of cervical cells is also affected by noise, poor contrast, dust, variable staining, and similar grayscale between different cell components. These popular methods are still weak in robustness and need further improvement. Recently, although there has been progress in segmentation of overlapping cervical cells [4, 5], the isolation of nuclei and cytoplasm from overlapping cervical cell clusters is an important task.In this paper, we present an asymmetric generalized Gaussian and Cauchy mixture model (GGCMM) to extract different components of overlapped cervical cells and then apply a shape constraint level set energy function to get the individual cytoplasm inside each cellular cluster. Our method is performed in two steps: extraction of nuclei and cellular clusters and segmentation of overlapping cytoplasm. In the first step, the Cauchy distribution used in our model is a heavier tailed one and makes the distribution of the proposed model asymmetric by combining with generalized Gaussian distribution. Besides, the shape parameter of generalized Gaussian distribution controls the tail of the distribution. Therefore, new probability distribution has flexibility to fit different shapes of observed data. Our distribution provides a heavy-tailed alternative symmetric distribution for potential outliers and therefore can produce a segmentation method which is more robust to outliers. The image of cervical cells is characterized by poor cytoplasmic contrast and noise; therefore, the proposed heavy-tailed asymmetric distribution plays an important role in balance between the noise and the image details (outliers). Our asymmetric GGCMM could accurately partition the Pap smear images into nuclei, cytoplasmic clusters, and background, which would serve as the input of modified level set framework. In the second step, in our level set energy function, we added a distance map and shape constraint term to correct the evolution of the boundary. Distance map could make boundary evolve along with an accurate direction and adjust the evolution speed of the boundary. The elliptical shape prior is defined based on the geometry of the clump and the detected centroid of nucleus. The shape constraint prior could modify the final cell contour shape to make it approach to the ground truth. The experimental dataset of overlapped cervical cells comes from the ISBI 2014 Challenge Dataset [6]. The evaluation is carried out to prove that the proposed algorithm is superior to other related methods in segmentation accuracy.


## 2. Related Studies

In the last few decades, many methods have been raised to solve the overlapping cervical cell segmentation problem [7, 8]. For example, Ushizima et al. [9] proposed an unsupervised method which combines superpixel with Voronoi technique to detect cervical cells. Their method incorporated low-mean pixel clusters with adaptive histogram equalization to improve nuclei detection and applied Voronoi diagrams to implement cervical cell segmentation. Lu et al. [4] represented an algorithm that combines quick shift, Gaussian mixture model (GMM), and maximally stable extremal region detector (MSER) [10] with joint level set method to segment overlapping cervical cells. Their method was also the baseline in the ISBI 2014 overlapping cervical cytology image segmentation challenge. Standard Gaussian distribution is short tail and symmetric, which means that it is weak in the case of heavy outliers. It is hard to fit various shapes of the observed data [11]. Zhou et al. [12] divided cervical cell images into overlapping subimages and applied adaptive threshold estimator to each of them. Tareef et al. [13] turned to incorporate SLIC [14] with support vector machine (SVM) to get nucleus and cytoplasm boundaries, and they applied sparse coding (SC) theory and some morphological processing to reconstruct every cell’s boundary. Recently, multipass watershed method [15] was also used in overlapping cervical cell segmentation, which applied different thresholds to get different regions-of-interest (ROIs). In addition, Chang et al. [16] introduced a method which incorporates morphology techniques with double threshold to detect cervical cells, and it was able to segment every cell in a short time. Active contour model was also covered in [17] to extract cells’ features. Chen et al. [18] presented an improved level set method based on combinations of edge, region, and prior information. But the energy function they proposed did not consider overlapping areas.In recent years, full convolution network (FCN) has shown its high performance in image processing [19]. FCN can preserve the spatial information of input image and label all pixels separately. U-Net is a specific neural network built upon FCN, and it can extract and classify different kinds of features to obtain good performance even when the training data are few [20]. Despite the classical U-Net has achieved impressive performance [21], it does not focus on ROI which would lead to a waste of computing resources and space, such as the excessive processing of irrelevant regions. Attention mechanism can handle this problem well [22]. This mechanism has been widely used in natural language processing (NLP) [23] and image processing [24]. More recently, Oktay et al. [25] integrated the attention gate (AG) model to classical U-Net architecture to highlight salient features.


## 3. Dataset Description

We train our approach on the training set provided by the ISBI 2014 Challenge Dataset [6]. There are totally 945 synthetic cytology images in this dataset, including 855 training images and 90 testing images with a size of 512 × 512. All these images were generated by collecting the nuclei and cytoplasm from depth of field (EDF) images, recombined by various transforms and using an alpha channel with a random value (from 0.88 to 0.99) to locate them on synthetic images. Then, some transforms, such as rotation, scale, and random linear brightness transform, are used to gather the cells in EDF images and put those cells into the random location of synthetic images. The cells in the synthetic images must have at least one overlap area, and the overlap coefficient is in one of the following options: (0, 0.1], (0.1, 0.2], (0.2, 0.3], (0.3, 0.4], and (0.4, 0.5]. Each image has 2∼10 Pap stained cervical cells per clump on average. In addition, all images are annotated with the ground truth of cell nuclei and cytoplasm. Figure 1 shows some examples of extended EDF Pap smear images from the ISBI 2014 Challenge Dataset, and each image has 8∼10 Pap stained cervical cells per clump, including ground truth of cell nuclei and cytoplasm.Figure 1
Examples of Pap smear images from the ISBI 2014 Challenge Dataset. Upper row: original images; middle row: ground truth of nuclei; last row: ground truth of cytoplasm.


## 4. The Proposed Model

The proposed approach implements three steps to segment the overlapping cervical cells. Firstly, the nuclei and the contours of cervical cell clumps are extracted through the asymmetric GGCMM model. Based on the hypothesis each nucleus represents a cell, we could acquire the corresponding cytoplasm according to each nucleus. Therefore, segmentation of nucleus is the important step toward analysis of cervical smear images. Secondly, depending upon the count of nuclei, obtain the initial contour of individual cell because the cytoplasmic boundary is usually located at an uneven radial distance from the centroid of its associated nucleus. Therefore, the shape of every detected nucleus can be applied to define an ellipse that represents the region of the corresponding cytoplasm. The third step focuses on cytoplasm segmentation for each cell within each clump. Each cell clump is processed individually to obtain single cell boundary in terms of the modified level set technique based on distance map and shape prior. Lastly, update the contour of individual cell to bring it to a more realistic level. We start by introducing the cell component labeling.


### 4.1. Cell Component Labeling Based on GGCMM

Letx = {x1, …, xn, …, xN} denote a set of N pixels in an image, and xn stands for an observation at the n-th pixel of an image. {Ω1, …, Ωk, …, ΩK} represents different labels. The finite mixture model assumes that the density function fxnΘ at each pixel xn can be described by(1)fxnΘ=∑k=1KπnkpxnΩk,where Θ represents the model parameters, and the prior probability πnk satisfies the following constraints:(2)0≤πnk≤1,∑k=1Kπnk=1.In this study,pxnΩk is the mixture of generalized Gaussian and Cauchy distributions, which are written in the following form:(3)pxnΩk=ηk1GNxnθk1+ηk2Cxnθk2,where ηk1 and ηk2 are called weighting factors and satisfy the following constraints:(4)0≤ηk1,ηk2≤1,ηk1+ηk2=1.G
N
x
n
θ
k
1 conforms to a generalized Gaussian distribution with parameter set θk1 = {μk1, σk1, λk1}, defined by(5)GNxnμk1,σk1,λk1=Aλk1×exp−Bλk1⋅xn−μk1σk1λk1,where μk1 is the mean value, σk1 is the standard deviation, and λk1 is the shape parameter. A(λk1) and B(λk1) are, respectively, calculated by formulae(6)Aλk1=λk1Γ3/λk12σk1⋅Γ1/λk1Γ1/λk1,Bλk1=Γ3/λk1Γ1/λk1λk1/2.Gamma function Γ(z) is defined as(7)Γz=∫0∞xz−1exdx.Figure2 shows the probability density function of generalized Gaussian distribution with different parameters. It can be seen that the generalized Gaussian distribution of some parameters has too sharp peaks. It is difficult to flexibly fit observation data of different shapes with this generalized Gaussian distribution. In order to overcome this problem, we should introduce Cauchy distribution into the generalized Gaussian distribution.Figure 2
Probability density functions: (a, b) generalized Gaussian distribution; (c, d) Cauchy distribution.
(a)
(b)
(c)
(d)The function of each componentCxnθk2 submits to a Cauchy distribution, written in the form(8)Cxnμk2,γk2=1π⋅γk2⋅1+xn−μk2/γk22,where θk2 = (μk2, γk2) and μk2 and γk2 represent location and scale parameters, respectively. Figure 2 gives the plot of 1D Cauchy distribution with different parameters. We show the estimated distributions obtained by employing the symmetric Gaussian distribution and proposed asymmetric distribution pxnΩk, shown in Figure 3. It can be visualized that our asymmetric distribution makes the flexibility to fit the observed data.Figure 3
Comparative study of component distributions: (a) histogram of the observed data; (b) standard Gaussian distribution; (c) proposed asymmetric distribution.
(a)
(b)
(c)The log-likelihood function of the density function can be written as(9)LΘ=∑n=1Nlog∑k=1Kπnkηk1GNxnθk1+ηk2Cxnθk2.To estimate the parameters Θ = {μk1, σk1, λk1, μk2, γk2} of the mixture model, we maximize the log-likelihood function in (9). Considering the complexity of (9), the expectation-maximization (EM) algorithm [26] cannot be directly applied for maximizing it. Therefore, Jensen’s inequality is introduced to overcome this problem [27]. Thus, the log-likelihood function (9) can be rewritten as follows:(10)LΘ≥EΘ=∑n=1N∑k=1Kznkynk1logπnkηk1GNxnθk1+ynk2logπnkηk2Cxnθk2,where the posterior probability znk is calculated using(11)znkt=πnktpxnΩkt∑m=1KπnmtpxnΩmt.Thus, maximizing the density functionfxnΘ is equivalent to maximizing the objective function E(Θ) in (10), where hidden variables ynk1 and ynk2 are defined by(12)ynk1t=ηk1tGNxnθk1tηk1tGNxnθk1t+ηk2tCxnθk2t,(13)ynk2t=ηk2tCxnθk2tηk1tGNxnθk1t+ηk2tCxnθk2t.The derivation of parameter estimation can be found in AppendixA. The main steps involved in the proposed method are summarized as follows:Step 1: initialize the parameters Θ = {μk1, σk1, λk1, μk2, γk2}.Step 2 (E step): calculate the posterior znk with (11).Step 3 (M step): update mean μk1, standard deviation σk1, and shape parameter λk1 using (A.3), (A.4), and (A.5), respectively.Update location parameterμk2 in (A.1) and evaluate scale parameter γk2 in (A.2)Compute prior distributionπnk by using (A.6)Update the hidden variablesynk1 and ynk2 using (12) and (13), respectivelyStep 4: evaluate the log-likelihood function in (9) and check its convergence. If convergence condition is satisfied, the iteration is terminated.After parameter estimation finished, each pixel will be labeled with the largest posterior probabilityznk using(14)xn∈Ωk:IFznk≥znε;k,ε=1,2,…,K.Figure4 shows the results of nuclei detection and cervical cell clump segmentation. As shown, the proposed GGCMM is flexible enough to fit the different shapes of the cervical cell clumps.Figure 4
The results of the proposed GGCMM segmentation method. First row: original images; second row: the nuclei and cell clump segmentation.


### 4.2. Initial Curve

In the level set technique, the initial curve is crucial for the final results. Generally, many pixels in the cell boundary have a similar distance to its nucleus, and this prior helps us to locate the initial curve. In this subsection, we should infer the initial curve for each cell in terms of the geometric structure between the nucleus and nearby cytoplasm. The approach for getting the initial curve has the following three processes:(1)
Calculate the centroid for each nucleusIn practice, there are a few fake nuclei which are much smaller than normal nuclei. Therefore, we first get rid of them by morphological operation. In this study, we discuss the case that the nuclei are enclosed in the cytoplasm. Depending upon the count of nuclei, we calculate the centroid for each nucleus shown in Figure5(a). These centroids are important in distance map calculation because they also stand for cell counts.(2)
Determine junction pointsA set of points is applied to represent the cell boundary. Calculating the distance between every boundary point to each centroid, each point is affiliated to the nearest centroid. As seen in Figure5(b), the centroid is surrounded by these boundary points. These points, which keep a same distance to two centroids, marked in yellow, are regarded as junction points.(3)
Estimate initial curve in overlapping areaWe assume that the initial contour of each cell is circular. Regard the centroid as the center of the circle, and take the lines between centroid and junction point as the radius. In this way, we get two straight linesa and b of different lengths. Let a represent a long side, then the initial contour is drawn in terms of the following expression:(15)r=a+θb−aθi,θ∈0,θi,i=1,2,3,…,N,where side a is shorter than b. θ is an angle between two sides a and b with the centroid as the vertex. The length of radius will change with θ in the process of rotation, displayed in Figure 5(c). This approach makes possible to get access to an accurate contour for level set evolution. Figure 5(d) represents the initial curve from each overlapping cervical cell.Figure 5
The intermediate results for obtaining initial contour: (a) centroid of the nucleus; (b) junction points (yellow), see the place indicated by black arrowhead line; (c) initial contour of overlapped region; (d) initial contour of single cell.
(a)
(b)
(c)
(d)


### 4.3. Single Cell Contour Update Based on Elliptical Shape Prior

This subsection shows an improved level set energy function based on the DRLSE [28] model. To eliminate excessive contour evolution, this paper would apply a distance map in terms of centroid and initial contour, described as(16)Dmxk=1,ifxkoutside clump,1−ℏLxk2,ifxkinside the cellm,0,otherwise,where m is the label of cell in the clump, xk is the point inside the m-th cell, and ℏ is a constant. L(xk) is the relative distance from the centroid to the boundary points, and the range of L(xk) is [0, 1]. According to (16), one could get the distance map of single cell displayed by Figure 6(b). The grayscale value of point xk in distance map indicates the respective distance to the centroid. This distance map can be used as a constraint to prevent the cell boundary from being evolved into the part that do not belong to this cell. As we all have observed, although the shape of cervical cells is varied, the contours of most cervical cells are nearly elliptical. So, besides the initial boundary and distance map we get in previous stage, we also incorporate the elliptical shape prior constraint into our energy function. Compared to classical level set energy function without shape prior item, for example, CV model [29], the proposed energy function can stabilize the evolution process by keeping an elliptical shape of our level set function and avoiding reinitialization.Figure 6
(a) Initial contour; (b) distance map; (c) elliptical shape prior; (d) final cell contour for single cell.
(a)
(b)
(c)
(d)Considerϕix,y:Ω⟶ℜ2 representing the level set function (LSF) on an image domain Ω, and there are N¯ cervical cells being detected. So ϕii=1N¯ denotes all LSF of N¯ cells in Ω. Then, this paper defines a new energy function E(ϕi) as follows:(17)Eϕi=γRpϕi+αLgϕi+β∑j≠iAgϕi,ϕj+Sϕi,ϕj,ψ,where γ = 0.04, β = 3, and α = 8 in this paper. Rp(ϕi) stands for the distance regularization term, which could keep the signed distance property ∇ϕi=1. Rp(ϕi) is defined as(18)Rpϕi=∫Ωp∇ϕidx,where p is the potential function p:0,∞⟶ℜ, for example, ps=0.5s−12. Lgϕiis the length term in our energy function, which is defined by(19)Lgϕi=∫Ωgδϕi∇ϕidx,where g=1/1+∇Gσ∗I is the edge stop function with the Gaussian kernel Gσ, which is used to smooth the image and eliminate the effect of noise. δ(⋅) is the Dirac delta function. Agϕi,ϕj is a modified weighted region term, which contains the distance map Di(x) calculated using (16), and Agϕi,ϕj is written as(20)Agϕi,ϕj=∫ΩDix⋅gH−ϕiH−ϕjdx,where H(⋅) is the Heaviside function. Agϕi,ϕj represents the overlapping area. For the distance map Di(x) reasons, the effect of Agϕi,ϕj would eliminate if the distance between nucleus and overlapping area become farther. If ϕi and ϕj do not intersect during the evolving process, then Agϕi,ϕj=0; it is because no overlapping place is found.In order to prevent the overlapping area from disappearing due to excessive evolution, we add the followingS(ϕi, ϕj, ψ), which is regarded as a shape prior term, in our energy function:(21)Sϕi,ϕj,ψ=∫ΩH−ϕjH−ϕi−H−ψ2dx,where ψ is the elliptical shape prior, obtained from the ellipse fitting of ϕi, shown in Figure 6(c). It is because, as we have mentioned, although the shape of cervical cells varies, the vast majority of cervical cells’ contour are similar to elliptical shape. Therefore, we choose the ellipse fitting method for ϕi to get the elliptical shape prior. S(ϕi, ϕj, ψ) attains minimum when the overlapping area of LSF approaches the elliptical shape prior ψ. In this way, the cell boundary of the overlapping area is becoming smooth and realistic, and other nonoverlapping areas would not be affected by this term. Minimizing (17) depends on finding the steady state solution of the gradient flow equation for ϕi. See Appendix B for this process.


## 4.1. Cell Component Labeling Based on GGCMM

Letx = {x1, …, xn, …, xN} denote a set of N pixels in an image, and xn stands for an observation at the n-th pixel of an image. {Ω1, …, Ωk, …, ΩK} represents different labels. The finite mixture model assumes that the density function fxnΘ at each pixel xn can be described by(1)fxnΘ=∑k=1KπnkpxnΩk,where Θ represents the model parameters, and the prior probability πnk satisfies the following constraints:(2)0≤πnk≤1,∑k=1Kπnk=1.In this study,pxnΩk is the mixture of generalized Gaussian and Cauchy distributions, which are written in the following form:(3)pxnΩk=ηk1GNxnθk1+ηk2Cxnθk2,where ηk1 and ηk2 are called weighting factors and satisfy the following constraints:(4)0≤ηk1,ηk2≤1,ηk1+ηk2=1.G
N
x
n
θ
k
1 conforms to a generalized Gaussian distribution with parameter set θk1 = {μk1, σk1, λk1}, defined by(5)GNxnμk1,σk1,λk1=Aλk1×exp−Bλk1⋅xn−μk1σk1λk1,where μk1 is the mean value, σk1 is the standard deviation, and λk1 is the shape parameter. A(λk1) and B(λk1) are, respectively, calculated by formulae(6)Aλk1=λk1Γ3/λk12σk1⋅Γ1/λk1Γ1/λk1,Bλk1=Γ3/λk1Γ1/λk1λk1/2.Gamma function Γ(z) is defined as(7)Γz=∫0∞xz−1exdx.Figure2 shows the probability density function of generalized Gaussian distribution with different parameters. It can be seen that the generalized Gaussian distribution of some parameters has too sharp peaks. It is difficult to flexibly fit observation data of different shapes with this generalized Gaussian distribution. In order to overcome this problem, we should introduce Cauchy distribution into the generalized Gaussian distribution.Figure 2
Probability density functions: (a, b) generalized Gaussian distribution; (c, d) Cauchy distribution.
(a)
(b)
(c)
(d)The function of each componentCxnθk2 submits to a Cauchy distribution, written in the form(8)Cxnμk2,γk2=1π⋅γk2⋅1+xn−μk2/γk22,where θk2 = (μk2, γk2) and μk2 and γk2 represent location and scale parameters, respectively. Figure 2 gives the plot of 1D Cauchy distribution with different parameters. We show the estimated distributions obtained by employing the symmetric Gaussian distribution and proposed asymmetric distribution pxnΩk, shown in Figure 3. It can be visualized that our asymmetric distribution makes the flexibility to fit the observed data.Figure 3
Comparative study of component distributions: (a) histogram of the observed data; (b) standard Gaussian distribution; (c) proposed asymmetric distribution.
(a)
(b)
(c)The log-likelihood function of the density function can be written as(9)LΘ=∑n=1Nlog∑k=1Kπnkηk1GNxnθk1+ηk2Cxnθk2.To estimate the parameters Θ = {μk1, σk1, λk1, μk2, γk2} of the mixture model, we maximize the log-likelihood function in (9). Considering the complexity of (9), the expectation-maximization (EM) algorithm [26] cannot be directly applied for maximizing it. Therefore, Jensen’s inequality is introduced to overcome this problem [27]. Thus, the log-likelihood function (9) can be rewritten as follows:(10)LΘ≥EΘ=∑n=1N∑k=1Kznkynk1logπnkηk1GNxnθk1+ynk2logπnkηk2Cxnθk2,where the posterior probability znk is calculated using(11)znkt=πnktpxnΩkt∑m=1KπnmtpxnΩmt.Thus, maximizing the density functionfxnΘ is equivalent to maximizing the objective function E(Θ) in (10), where hidden variables ynk1 and ynk2 are defined by(12)ynk1t=ηk1tGNxnθk1tηk1tGNxnθk1t+ηk2tCxnθk2t,(13)ynk2t=ηk2tCxnθk2tηk1tGNxnθk1t+ηk2tCxnθk2t.The derivation of parameter estimation can be found in AppendixA. The main steps involved in the proposed method are summarized as follows:Step 1: initialize the parameters Θ = {μk1, σk1, λk1, μk2, γk2}.Step 2 (E step): calculate the posterior znk with (11).Step 3 (M step): update mean μk1, standard deviation σk1, and shape parameter λk1 using (A.3), (A.4), and (A.5), respectively.Update location parameterμk2 in (A.1) and evaluate scale parameter γk2 in (A.2)Compute prior distributionπnk by using (A.6)Update the hidden variablesynk1 and ynk2 using (12) and (13), respectivelyStep 4: evaluate the log-likelihood function in (9) and check its convergence. If convergence condition is satisfied, the iteration is terminated.After parameter estimation finished, each pixel will be labeled with the largest posterior probabilityznk using(14)xn∈Ωk:IFznk≥znε;k,ε=1,2,…,K.Figure4 shows the results of nuclei detection and cervical cell clump segmentation. As shown, the proposed GGCMM is flexible enough to fit the different shapes of the cervical cell clumps.Figure 4
The results of the proposed GGCMM segmentation method. First row: original images; second row: the nuclei and cell clump segmentation.


## 4.2. Initial Curve

In the level set technique, the initial curve is crucial for the final results. Generally, many pixels in the cell boundary have a similar distance to its nucleus, and this prior helps us to locate the initial curve. In this subsection, we should infer the initial curve for each cell in terms of the geometric structure between the nucleus and nearby cytoplasm. The approach for getting the initial curve has the following three processes:(1)
Calculate the centroid for each nucleusIn practice, there are a few fake nuclei which are much smaller than normal nuclei. Therefore, we first get rid of them by morphological operation. In this study, we discuss the case that the nuclei are enclosed in the cytoplasm. Depending upon the count of nuclei, we calculate the centroid for each nucleus shown in Figure5(a). These centroids are important in distance map calculation because they also stand for cell counts.(2)
Determine junction pointsA set of points is applied to represent the cell boundary. Calculating the distance between every boundary point to each centroid, each point is affiliated to the nearest centroid. As seen in Figure5(b), the centroid is surrounded by these boundary points. These points, which keep a same distance to two centroids, marked in yellow, are regarded as junction points.(3)
Estimate initial curve in overlapping areaWe assume that the initial contour of each cell is circular. Regard the centroid as the center of the circle, and take the lines between centroid and junction point as the radius. In this way, we get two straight linesa and b of different lengths. Let a represent a long side, then the initial contour is drawn in terms of the following expression:(15)r=a+θb−aθi,θ∈0,θi,i=1,2,3,…,N,where side a is shorter than b. θ is an angle between two sides a and b with the centroid as the vertex. The length of radius will change with θ in the process of rotation, displayed in Figure 5(c). This approach makes possible to get access to an accurate contour for level set evolution. Figure 5(d) represents the initial curve from each overlapping cervical cell.Figure 5
The intermediate results for obtaining initial contour: (a) centroid of the nucleus; (b) junction points (yellow), see the place indicated by black arrowhead line; (c) initial contour of overlapped region; (d) initial contour of single cell.
(a)
(b)
(c)
(d)


## 4.3. Single Cell Contour Update Based on Elliptical Shape Prior

This subsection shows an improved level set energy function based on the DRLSE [28] model. To eliminate excessive contour evolution, this paper would apply a distance map in terms of centroid and initial contour, described as(16)Dmxk=1,ifxkoutside clump,1−ℏLxk2,ifxkinside the cellm,0,otherwise,where m is the label of cell in the clump, xk is the point inside the m-th cell, and ℏ is a constant. L(xk) is the relative distance from the centroid to the boundary points, and the range of L(xk) is [0, 1]. According to (16), one could get the distance map of single cell displayed by Figure 6(b). The grayscale value of point xk in distance map indicates the respective distance to the centroid. This distance map can be used as a constraint to prevent the cell boundary from being evolved into the part that do not belong to this cell. As we all have observed, although the shape of cervical cells is varied, the contours of most cervical cells are nearly elliptical. So, besides the initial boundary and distance map we get in previous stage, we also incorporate the elliptical shape prior constraint into our energy function. Compared to classical level set energy function without shape prior item, for example, CV model [29], the proposed energy function can stabilize the evolution process by keeping an elliptical shape of our level set function and avoiding reinitialization.Figure 6
(a) Initial contour; (b) distance map; (c) elliptical shape prior; (d) final cell contour for single cell.
(a)
(b)
(c)
(d)Considerϕix,y:Ω⟶ℜ2 representing the level set function (LSF) on an image domain Ω, and there are N¯ cervical cells being detected. So ϕii=1N¯ denotes all LSF of N¯ cells in Ω. Then, this paper defines a new energy function E(ϕi) as follows:(17)Eϕi=γRpϕi+αLgϕi+β∑j≠iAgϕi,ϕj+Sϕi,ϕj,ψ,where γ = 0.04, β = 3, and α = 8 in this paper. Rp(ϕi) stands for the distance regularization term, which could keep the signed distance property ∇ϕi=1. Rp(ϕi) is defined as(18)Rpϕi=∫Ωp∇ϕidx,where p is the potential function p:0,∞⟶ℜ, for example, ps=0.5s−12. Lgϕiis the length term in our energy function, which is defined by(19)Lgϕi=∫Ωgδϕi∇ϕidx,where g=1/1+∇Gσ∗I is the edge stop function with the Gaussian kernel Gσ, which is used to smooth the image and eliminate the effect of noise. δ(⋅) is the Dirac delta function. Agϕi,ϕj is a modified weighted region term, which contains the distance map Di(x) calculated using (16), and Agϕi,ϕj is written as(20)Agϕi,ϕj=∫ΩDix⋅gH−ϕiH−ϕjdx,where H(⋅) is the Heaviside function. Agϕi,ϕj represents the overlapping area. For the distance map Di(x) reasons, the effect of Agϕi,ϕj would eliminate if the distance between nucleus and overlapping area become farther. If ϕi and ϕj do not intersect during the evolving process, then Agϕi,ϕj=0; it is because no overlapping place is found.In order to prevent the overlapping area from disappearing due to excessive evolution, we add the followingS(ϕi, ϕj, ψ), which is regarded as a shape prior term, in our energy function:(21)Sϕi,ϕj,ψ=∫ΩH−ϕjH−ϕi−H−ψ2dx,where ψ is the elliptical shape prior, obtained from the ellipse fitting of ϕi, shown in Figure 6(c). It is because, as we have mentioned, although the shape of cervical cells varies, the vast majority of cervical cells’ contour are similar to elliptical shape. Therefore, we choose the ellipse fitting method for ϕi to get the elliptical shape prior. S(ϕi, ϕj, ψ) attains minimum when the overlapping area of LSF approaches the elliptical shape prior ψ. In this way, the cell boundary of the overlapping area is becoming smooth and realistic, and other nonoverlapping areas would not be affected by this term. Minimizing (17) depends on finding the steady state solution of the gradient flow equation for ϕi. See Appendix B for this process.


## 5. Evaluation Metrics

In order to assess the segmentation results of nuclei and cytoplasm, the experiment applies the following pixel-level and object-level evaluation metrics developed by Gençtav et al. [2]. If Od and its corresponding ground truth OGT satisfy the following condition, the nucleus area Od results better in segmentation:(22)Od∩OGTOd>th,Od∩OGTOGT>th.In general, the threshold th is usually set to 0.6 [2]. We get pixel-level true positive (TPP), object-level false negative (FNO), and pixel-level false positive (FPP) from segmentation results, and then, we could compute the object-level precision (PO) and recall (RO) using(23)PO=TPOTPO+FPO,RO=TPOTPO+FNO.In the same way, the pixel-level precision (PP) and recall (RP) could be obtained where the subscript is P instead of O.Another important metric is the Dice coefficient (DC) [30], and DC represents the overlapping rate between segmentation result and ground truth. The segmentation is deemed as good when DC is higher than a certain threshold. It is computed by(24)DC=2Od∩OGTOd+OGT,where Od is the segmentation result and OGT is the ground truth of the input image. And |⋅| denotes the number of pixels in the object.The experiment applies MATLAB 2017a for nuclei and cell clump segmentation as well as cell contour initialization and evolution.


## 6. Experimental Results and Discussion

### 6.1. Parameter Analysis

Our method has three primary parametersα, β, and γ in (17). These parameters help the algorithm implement more effectively. This subsection conducts experiments to discuss the setting of these parameters. In fact, parameter setting is a complicated procedure of fine tuning. Here, we propose a strategy of fixed parameter β. By discussing the relationship between parameter α (or γ) and parameter β, parameter selection could be simplified and became more effective. For the 90 testing images of the ISBI 2014 Challenge Dataset, we fixed the coefficient β = 3 in all runs, and Table 1 reports the performance of two different settings for parameters α and γ. From the table, the average fluctuations of DC, TPP, and FPP are very limited. Therefore, the following suggestions are put forward for the parameter selection of our proposed method: (i) parameter β is fixed at 3; (ii) the range of parameter α can be set to 2∼3 times that of parameter β; and (iii) parameter γ is set within the range [0.01, 1].Table 1
Performance of the proposed method with different parameters.
α
Β
γ
TPP
FPP
FNO
DC
7
3
0.10
0.93 (0.10)
0.005 (0.004)
0.09 (0.13)
0.88 (0.09)
8
3
0.03
0.93 (0.10)
0.004 (0.004)
0.09 (0.13)
0.88 (0.09)
6
3
0.05
0.92 (0.11)
0.003 (0.004)
0.08 (0.11)
0.89 (0.08)
9
3
0.08
0.93 (0.11)
0.005 (0.004)
0.13 (0.16)
0.89 (0.08)
8
3
0.04
0.94 (0.09)
0.004 (0.003)
0.10 (0.13)
0.89 (0.07)
12
3
0.15
0.92 (0.11)
0.003 (0.004)
0.08 (0.11)
0.88 (0.07)
10
3
0.12
0.32 (0.10)
0.004 (0.003)
0.09 (0.12)
0.88 (0.08)


### 6.2. Shape Prior Analysis

In the next experiment, the effect of the elliptical shape prior on cytoplasmic segmentation would be discussed. Two performances are engaged: one with elliptical shape prior and another without prior. Figure7 shows segmentation results to four cervical images in the test set. It could be seen that, by incorporating elliptical shape prior into level set evolution, the cytoplasmic boundary extraction is governed by the representative shape feature of cells, and evolution of cytoplasmic boundary is guided toward ground truth. We also qualitatively evaluated the cytoplasm segmentation results of these images. As reported in Figure 8, the proposed framework with elliptical shape prior gives better segmentation for overlapping areas. Especially for highly overlapping cervical cells, our method effectively depicts cytoplasmic boundaries.Figure 7
Some examples of cytoplasm segmentation using our method. First row: without prior; second row: elliptical shape prior; last row: ground truth.Figure 8
Quantitative evaluation of cytoplasm segmentation: (a) Dice coefficient; (b) pixel-level false positive.
(a)
(b)


### 6.3. Performance Evaluation

Some of the visual results of the proposed methodology are illustrated in Figure9. The nuclei and cytoplasm are properly labeled by GGCMM despite the poor contrast of cell clump and background with noise. It is obvious that individual cell boundary in the overlapping area is correctly detected by the proposed level set method which involves distance map and shape prior. Specifically, the nuclei and cytoplasm segmentation results of our method are very close to ground truth (Figure 9). There is only a few difference to the ground truth. This is mainly due to the fact that the overlapping rate of cervical cells is too high and the contrast of images is poor, so it is difficult for the naked eye to accurately identify the boundary of each cell.Figure 9
Some examples of nuclei and cytoplasm segmentation. First row: original images; second row: nuclei and cytoplasm are labeled using GGCMM; third row: segmentation results of our method; fourth row: ground truth.Furthermore, we investigate the conduct of our method through comparing with several state-of-the-art segmentation algorithms, such as Lu et al. [4], Ushizima et al. [9], Nosrati and Hamarneh [31], and Tareef et al. [13]. A visual comparison of these results is reported in Figure 10. Ushizima et al.’s method has less ability to separate the overlapping regions. It is obvious that the model introduced by Nosrati et al. has advantages in the overlapping regions; however, their model is moderate in extracting the cervical cells from the background. This is because the inaccurate random decision forest probability map was utilized. It is found that the troubles Lu et al.’s method have are the same as those Nosrati et al. do. And this problem can be solved by elliptical shape prior and distance map in our method. It is worth mentioning that our segmentation process has demonstrated a competitive performance. It is because of the applying asymmetric GGCMM in current study, which helps to classify the nuclei and cytoplasm more precisely. In addition, the distance map and shape prior can produce stable and smooth results of segmentation. The following experiment conducts quantitative evaluations of our approach for nuclei and cytoplasm segmentation performance.Figure 10
Comparing segmentation results of overlapping cervical cells using different approaches. From left to right: Ushizima et al. [9], Nosrati and Hamarneh [31], Lu et al. [4], Tareef et al. [13], our segmentation results, and ground truth.


### 6.4. Evaluation of Nuclei Segmentation

The segmented nucleus is judged to be a true positive if this nucleus and its ground truth satisfies condition (22) and those are not regarded as false negative. There are a total of 540 nuclei in all images of ISBI 2014 Challenge Dataset. In our nuclei segmentation task, we did not miss any true-positive nuclei and detect all 540 nuclei. Therefore, in our method, TPO of nuclei is 540 and FNO is 0. Besides, we detected 4 false-positive nuclei because of the similarity of intensity between nuclei and the surrounding cytoplasm, which would lead to the error of the GGCMM output. Figure 9 presents the performance of our method on nuclei segmentation. The quantitative evaluation of nuclei segmentation is shown in Table 2, where μ and σ stands for the mean and standard deviation, respectively. As displayed in Table 2, GGCMM is done well, and the recall of true-positive rate in our method is 1.00 and 6% increase by Tareef et al. at 0.94, which indicates that the experiment detects all true-positive nuclei and do not leave out any one.Table 2
Quantitative evaluation of nuclei segmentation.
Approaches
P
obj
R
obj
P
pix (μ(σ))
R
pix (μ(σ))
DC (μ(σ))
Lu’s approach [4]
0.98
0.88
0.94 (0.08)
0.91 (0.08)
0.92 (0.05)
Tareef’s approach [13]
0.99
0.94
0.95 (0.06)
0.93 (0.07)
0.93 (0.04)
Our approach
0.99
1.00
0.98 (0.02)
0.94 (0.02)
0.94 (0.01)
P
obj and Robj represent the precision and recall of object-level nuclei segmentation. Ppix and Rpix indicate the precision and recall of pixel-level nuclei segmentation.


### 6.5. Evaluation of Cytoplasm Segmentation

We also qualitatively analyze the cytoplasm segmentation results on the ISBI 2014 Challenge Dataset. With our method, as shown in Table3, the pixel-based true-positive rate TPP is 0.94 (0.09), much higher than any other methods. This means that a better performance for cytoplasm segmentation can be obtained. The DC of 0.90 (0.07) is the highest DC in all above models, which is 3% higher than Ushizima et al.’s and Nostrati et al.’s methods. The pixel-based false-negative rate FNO of 0.08 (0.11) is the lowest in all above methods, which show that the results of our method is very close to the ground truth. The mean of pixel-based false negative FPP of 0.004 surpassed Nostrati et al.’s method by about 0.001.Table 3
Quantitative evaluation of cytoplasm segmentation (μ (σ)).
Approaches
TPP
FPP
FNO
DC
Ushizima et al. [9]
0.83 (0.13)
0.002 (0.002)
0.17 (0.21)
0.87 (0.08)
Nosrati and Hamarneh [31]
0.90 (0.08)
0.005 (0.004)
0.14 (0.16)
0.87 (0.07)
Lu’s approach [4]
0.90 (0.10)
0.003 (0.005)
0.31 (0.29)
0.89 (0.08)
Tareef’s approach [13]
0.91 (0.09)
0.004 (0.005)
0.27 (0.28)
0.89 (0.07)
Our approach
0.94 (0.09)
0.004 (0.004)
0.08 (0.11)
0.90 (0.07)Generally, with the pixel-based true-positive rate, with increasing TPP, the false-positive rate FPP would also increase, and vice versa. As shown in Table 3, Ushizima et al. approach had the lowest FPP, but the cost is that the TPP of their method is also the lowest one. Our method had the highest TPP among all methods, whereas our FPP still remains in a low level. Similarly, TPP and DC are negatively correlated with false-negative rate FNO. The DC values of Lu et al. and Tareef et al.’s technology are similar to ours, but their FNO values are about three times higher than us. Our segmentation method has better results in both cytoplasm and nuclei. This is because the shape constraint term in energy function prevented level set contour from overevolution, which can keep the boundary smooth and stable in overlapping areas. In addition, the distance map serves as another attractor to make the evolution of overlapping regions approach the real shapes of cell.


### 6.6. Evaluation of Segmentation for Different Cell Structures

In the last experiment, the proposed method is analyzed by evaluating all the performance measures with varying overlap ratios and number of cells. Table4 summarizes the evaluation results with an overlap ratio in one of the following ranges: [0, 0.1], [0.1, 0.2], [0.2, 0.3], [0.3, 0.4], and [0.4, 0.5], where the number of cells in each Pap image is varied between 5 and 10 cells. The threshold DC is set as 0.7. In general, the results illustrated in this table indicate that our approach has high accuracy in low overlapping ratio 0.1∼0.3 with average DC in the range [0.89, 0.96], regardless of the number of cells in a single Pap smear image. In the high overlapping ratio [0.4, 0.5], the DC still maintains at an acceptable level. The pixel-based false-positive rate FPP remained very low throughout the experiment with the highest value of 0.01, and its value is positively related to the overlapping rate. According to the results in Table 4, the pixel-based true-positive rate TPP maintained a high level in the experiment. Specifically, in the case of low overlapping rate such as [0, 0.3], this value still remains high. The false-negative rate FNO is only higher if the overlapping rate is above 0.4. In most cases, FNO is very low and may even be 0. These results prove that the proposed algorithm is promising in overlapping cervical cell segmentation.Table 4
Quantitative evaluation of nuclei and cytoplasm segmentation (μ(σ)) with different cell numbers and overlapping rate.
Cell numbers
Metrics
[0, 0.1]
[0.1, 0.2]
[0.2, 0.3]
[0.3, 0.4]
[0.4, 0.5]
5 cells
TPP
0.99 (0.02)
0.91 (0.09)
0.97 (0.06)
0.93 (0.09)
0.90 (0.08)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
0.00 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.30 (0.14)
DC
0.98 (0.02)
0.90 (0.04)
0.90 (0.05)
0.86 (0.08)
0.90 (0.03)
6 cells
TPP
0.98 (0.05)
0.96 (0.05)
0.95 (0.06)
0.93 (0.10)
0.94 (0.09)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
0.01 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.08 (0.12)
0.17 (0.23)
DC
0.96 (0.04)
0.92 (0.05)
0.89 (0.05)
0.85 (0.08)
0.89 (0.08)
7 cells
TPP
0.99 (0.01)
0.96 (0.07)
0.96 (0.07)
0.87 (0.13)
0.93 (0.09)
FPP
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.00 (0.00)
0.07 (0.10)
0.28 (0.20)
0.21 (0.10)
DC
0.97 (0.03)
0.91 (0.06)
0.91 (0.06)
0.85 (0.06)
0.89 (0.06)
8 cells
TPP
0.99 (0.01)
0.89 (0.13)
0.94 (0.07)
0.92 (0.10)
0.91 (0.06)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.06 (0.08)
0.06 (0.09)
0.06 (0.09)
0.25 (0.17)
DC
0.97 (0.02)
0.88 (0.07)
0.88 (0.08)
0.87 (0.06)
0.83 (0.06)
9 cells
TPP
0.99 (0.01)
0.90 (0.12)
0.92 (0.09)
0.92 (0.13)
0.92 (0.12)
FPP
0.00 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.05 (0.08)
0.17 (0.08)
0.11 (0.00)
0.22 (0.16)
DC
0.96 (0.03)
0.85 (0.07)
0.85 (0.06)
0.84 (0.08)
0.81 (0.08)
10 cells
TPP
0.98 (0.04)
0.94 (0.09)
0.94 (0.06)
0.88 (0.11)
0.91 (0.12)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.01)
0.01 (0.00)
0.01 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.15 (0.07)
0.35 (0.07)
DC
0.96 (0.03)
0.89 (0.06)
0.85 (0.07)
0.83 (0.07)
0.82 (0.05)
Average
TPP
0.99 (0.04)
0.94 (0.08)
0.94 (0.07)
0.92 (0.11)
0.91 (0.11)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.04)
0.01 (0.01)
0.01 (0.01)
FNO
0.00 (0.00)
0.07 (0.08)
0.04 (0.06)
0.11 (0.14)
0.30 (0.15)
DC
0.96 (0.03)
0.90 (0.06)
0.89 (0.06)
0.86 (0.08)
0.86 (0.07)


## 6.1. Parameter Analysis

Our method has three primary parametersα, β, and γ in (17). These parameters help the algorithm implement more effectively. This subsection conducts experiments to discuss the setting of these parameters. In fact, parameter setting is a complicated procedure of fine tuning. Here, we propose a strategy of fixed parameter β. By discussing the relationship between parameter α (or γ) and parameter β, parameter selection could be simplified and became more effective. For the 90 testing images of the ISBI 2014 Challenge Dataset, we fixed the coefficient β = 3 in all runs, and Table 1 reports the performance of two different settings for parameters α and γ. From the table, the average fluctuations of DC, TPP, and FPP are very limited. Therefore, the following suggestions are put forward for the parameter selection of our proposed method: (i) parameter β is fixed at 3; (ii) the range of parameter α can be set to 2∼3 times that of parameter β; and (iii) parameter γ is set within the range [0.01, 1].Table 1
Performance of the proposed method with different parameters.
α
Β
γ
TPP
FPP
FNO
DC
7
3
0.10
0.93 (0.10)
0.005 (0.004)
0.09 (0.13)
0.88 (0.09)
8
3
0.03
0.93 (0.10)
0.004 (0.004)
0.09 (0.13)
0.88 (0.09)
6
3
0.05
0.92 (0.11)
0.003 (0.004)
0.08 (0.11)
0.89 (0.08)
9
3
0.08
0.93 (0.11)
0.005 (0.004)
0.13 (0.16)
0.89 (0.08)
8
3
0.04
0.94 (0.09)
0.004 (0.003)
0.10 (0.13)
0.89 (0.07)
12
3
0.15
0.92 (0.11)
0.003 (0.004)
0.08 (0.11)
0.88 (0.07)
10
3
0.12
0.32 (0.10)
0.004 (0.003)
0.09 (0.12)
0.88 (0.08)


## 6.2. Shape Prior Analysis

In the next experiment, the effect of the elliptical shape prior on cytoplasmic segmentation would be discussed. Two performances are engaged: one with elliptical shape prior and another without prior. Figure7 shows segmentation results to four cervical images in the test set. It could be seen that, by incorporating elliptical shape prior into level set evolution, the cytoplasmic boundary extraction is governed by the representative shape feature of cells, and evolution of cytoplasmic boundary is guided toward ground truth. We also qualitatively evaluated the cytoplasm segmentation results of these images. As reported in Figure 8, the proposed framework with elliptical shape prior gives better segmentation for overlapping areas. Especially for highly overlapping cervical cells, our method effectively depicts cytoplasmic boundaries.Figure 7
Some examples of cytoplasm segmentation using our method. First row: without prior; second row: elliptical shape prior; last row: ground truth.Figure 8
Quantitative evaluation of cytoplasm segmentation: (a) Dice coefficient; (b) pixel-level false positive.
(a)
(b)


## 6.3. Performance Evaluation

Some of the visual results of the proposed methodology are illustrated in Figure9. The nuclei and cytoplasm are properly labeled by GGCMM despite the poor contrast of cell clump and background with noise. It is obvious that individual cell boundary in the overlapping area is correctly detected by the proposed level set method which involves distance map and shape prior. Specifically, the nuclei and cytoplasm segmentation results of our method are very close to ground truth (Figure 9). There is only a few difference to the ground truth. This is mainly due to the fact that the overlapping rate of cervical cells is too high and the contrast of images is poor, so it is difficult for the naked eye to accurately identify the boundary of each cell.Figure 9
Some examples of nuclei and cytoplasm segmentation. First row: original images; second row: nuclei and cytoplasm are labeled using GGCMM; third row: segmentation results of our method; fourth row: ground truth.Furthermore, we investigate the conduct of our method through comparing with several state-of-the-art segmentation algorithms, such as Lu et al. [4], Ushizima et al. [9], Nosrati and Hamarneh [31], and Tareef et al. [13]. A visual comparison of these results is reported in Figure 10. Ushizima et al.’s method has less ability to separate the overlapping regions. It is obvious that the model introduced by Nosrati et al. has advantages in the overlapping regions; however, their model is moderate in extracting the cervical cells from the background. This is because the inaccurate random decision forest probability map was utilized. It is found that the troubles Lu et al.’s method have are the same as those Nosrati et al. do. And this problem can be solved by elliptical shape prior and distance map in our method. It is worth mentioning that our segmentation process has demonstrated a competitive performance. It is because of the applying asymmetric GGCMM in current study, which helps to classify the nuclei and cytoplasm more precisely. In addition, the distance map and shape prior can produce stable and smooth results of segmentation. The following experiment conducts quantitative evaluations of our approach for nuclei and cytoplasm segmentation performance.Figure 10
Comparing segmentation results of overlapping cervical cells using different approaches. From left to right: Ushizima et al. [9], Nosrati and Hamarneh [31], Lu et al. [4], Tareef et al. [13], our segmentation results, and ground truth.


## 6.4. Evaluation of Nuclei Segmentation

The segmented nucleus is judged to be a true positive if this nucleus and its ground truth satisfies condition (22) and those are not regarded as false negative. There are a total of 540 nuclei in all images of ISBI 2014 Challenge Dataset. In our nuclei segmentation task, we did not miss any true-positive nuclei and detect all 540 nuclei. Therefore, in our method, TPO of nuclei is 540 and FNO is 0. Besides, we detected 4 false-positive nuclei because of the similarity of intensity between nuclei and the surrounding cytoplasm, which would lead to the error of the GGCMM output. Figure 9 presents the performance of our method on nuclei segmentation. The quantitative evaluation of nuclei segmentation is shown in Table 2, where μ and σ stands for the mean and standard deviation, respectively. As displayed in Table 2, GGCMM is done well, and the recall of true-positive rate in our method is 1.00 and 6% increase by Tareef et al. at 0.94, which indicates that the experiment detects all true-positive nuclei and do not leave out any one.Table 2
Quantitative evaluation of nuclei segmentation.
Approaches
P
obj
R
obj
P
pix (μ(σ))
R
pix (μ(σ))
DC (μ(σ))
Lu’s approach [4]
0.98
0.88
0.94 (0.08)
0.91 (0.08)
0.92 (0.05)
Tareef’s approach [13]
0.99
0.94
0.95 (0.06)
0.93 (0.07)
0.93 (0.04)
Our approach
0.99
1.00
0.98 (0.02)
0.94 (0.02)
0.94 (0.01)
P
obj and Robj represent the precision and recall of object-level nuclei segmentation. Ppix and Rpix indicate the precision and recall of pixel-level nuclei segmentation.


## 6.5. Evaluation of Cytoplasm Segmentation

We also qualitatively analyze the cytoplasm segmentation results on the ISBI 2014 Challenge Dataset. With our method, as shown in Table3, the pixel-based true-positive rate TPP is 0.94 (0.09), much higher than any other methods. This means that a better performance for cytoplasm segmentation can be obtained. The DC of 0.90 (0.07) is the highest DC in all above models, which is 3% higher than Ushizima et al.’s and Nostrati et al.’s methods. The pixel-based false-negative rate FNO of 0.08 (0.11) is the lowest in all above methods, which show that the results of our method is very close to the ground truth. The mean of pixel-based false negative FPP of 0.004 surpassed Nostrati et al.’s method by about 0.001.Table 3
Quantitative evaluation of cytoplasm segmentation (μ (σ)).
Approaches
TPP
FPP
FNO
DC
Ushizima et al. [9]
0.83 (0.13)
0.002 (0.002)
0.17 (0.21)
0.87 (0.08)
Nosrati and Hamarneh [31]
0.90 (0.08)
0.005 (0.004)
0.14 (0.16)
0.87 (0.07)
Lu’s approach [4]
0.90 (0.10)
0.003 (0.005)
0.31 (0.29)
0.89 (0.08)
Tareef’s approach [13]
0.91 (0.09)
0.004 (0.005)
0.27 (0.28)
0.89 (0.07)
Our approach
0.94 (0.09)
0.004 (0.004)
0.08 (0.11)
0.90 (0.07)Generally, with the pixel-based true-positive rate, with increasing TPP, the false-positive rate FPP would also increase, and vice versa. As shown in Table 3, Ushizima et al. approach had the lowest FPP, but the cost is that the TPP of their method is also the lowest one. Our method had the highest TPP among all methods, whereas our FPP still remains in a low level. Similarly, TPP and DC are negatively correlated with false-negative rate FNO. The DC values of Lu et al. and Tareef et al.’s technology are similar to ours, but their FNO values are about three times higher than us. Our segmentation method has better results in both cytoplasm and nuclei. This is because the shape constraint term in energy function prevented level set contour from overevolution, which can keep the boundary smooth and stable in overlapping areas. In addition, the distance map serves as another attractor to make the evolution of overlapping regions approach the real shapes of cell.


## 6.6. Evaluation of Segmentation for Different Cell Structures

In the last experiment, the proposed method is analyzed by evaluating all the performance measures with varying overlap ratios and number of cells. Table4 summarizes the evaluation results with an overlap ratio in one of the following ranges: [0, 0.1], [0.1, 0.2], [0.2, 0.3], [0.3, 0.4], and [0.4, 0.5], where the number of cells in each Pap image is varied between 5 and 10 cells. The threshold DC is set as 0.7. In general, the results illustrated in this table indicate that our approach has high accuracy in low overlapping ratio 0.1∼0.3 with average DC in the range [0.89, 0.96], regardless of the number of cells in a single Pap smear image. In the high overlapping ratio [0.4, 0.5], the DC still maintains at an acceptable level. The pixel-based false-positive rate FPP remained very low throughout the experiment with the highest value of 0.01, and its value is positively related to the overlapping rate. According to the results in Table 4, the pixel-based true-positive rate TPP maintained a high level in the experiment. Specifically, in the case of low overlapping rate such as [0, 0.3], this value still remains high. The false-negative rate FNO is only higher if the overlapping rate is above 0.4. In most cases, FNO is very low and may even be 0. These results prove that the proposed algorithm is promising in overlapping cervical cell segmentation.Table 4
Quantitative evaluation of nuclei and cytoplasm segmentation (μ(σ)) with different cell numbers and overlapping rate.
Cell numbers
Metrics
[0, 0.1]
[0.1, 0.2]
[0.2, 0.3]
[0.3, 0.4]
[0.4, 0.5]
5 cells
TPP
0.99 (0.02)
0.91 (0.09)
0.97 (0.06)
0.93 (0.09)
0.90 (0.08)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
0.00 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.30 (0.14)
DC
0.98 (0.02)
0.90 (0.04)
0.90 (0.05)
0.86 (0.08)
0.90 (0.03)
6 cells
TPP
0.98 (0.05)
0.96 (0.05)
0.95 (0.06)
0.93 (0.10)
0.94 (0.09)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
0.01 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.08 (0.12)
0.17 (0.23)
DC
0.96 (0.04)
0.92 (0.05)
0.89 (0.05)
0.85 (0.08)
0.89 (0.08)
7 cells
TPP
0.99 (0.01)
0.96 (0.07)
0.96 (0.07)
0.87 (0.13)
0.93 (0.09)
FPP
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.00 (0.00)
0.07 (0.10)
0.28 (0.20)
0.21 (0.10)
DC
0.97 (0.03)
0.91 (0.06)
0.91 (0.06)
0.85 (0.06)
0.89 (0.06)
8 cells
TPP
0.99 (0.01)
0.89 (0.13)
0.94 (0.07)
0.92 (0.10)
0.91 (0.06)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.06 (0.08)
0.06 (0.09)
0.06 (0.09)
0.25 (0.17)
DC
0.97 (0.02)
0.88 (0.07)
0.88 (0.08)
0.87 (0.06)
0.83 (0.06)
9 cells
TPP
0.99 (0.01)
0.90 (0.12)
0.92 (0.09)
0.92 (0.13)
0.92 (0.12)
FPP
0.00 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.00)
0.01 (0.01)
FNO
0.00 (0.00)
0.05 (0.08)
0.17 (0.08)
0.11 (0.00)
0.22 (0.16)
DC
0.96 (0.03)
0.85 (0.07)
0.85 (0.06)
0.84 (0.08)
0.81 (0.08)
10 cells
TPP
0.98 (0.04)
0.94 (0.09)
0.94 (0.06)
0.88 (0.11)
0.91 (0.12)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.01)
0.01 (0.00)
0.01 (0.00)
FNO
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
0.15 (0.07)
0.35 (0.07)
DC
0.96 (0.03)
0.89 (0.06)
0.85 (0.07)
0.83 (0.07)
0.82 (0.05)
Average
TPP
0.99 (0.04)
0.94 (0.08)
0.94 (0.07)
0.92 (0.11)
0.91 (0.11)
FPP
0.00 (0.00)
0.00 (0.00)
0.01 (0.04)
0.01 (0.01)
0.01 (0.01)
FNO
0.00 (0.00)
0.07 (0.08)
0.04 (0.06)
0.11 (0.14)
0.30 (0.15)
DC
0.96 (0.03)
0.90 (0.06)
0.89 (0.06)
0.86 (0.08)
0.86 (0.07)


## 7. Conclusions

In this paper, a fusion algorithm, involving asymmetric GGCMM and shape constraint level set method, was proposed to extract overlapping cervical cells in Pap smear images. Applying asymmetric GGCMM could help us to extract the cytoplasm clumps and nuclei more precisely than other previous methods. More specifically, we proposed a novel asymmetric probability distribution through combining generalized Gaussian distribution and Cauchy distribution so that each component of the proposed asymmetric GGCMM can model the observed data of different shapes. Furthermore, the shape constraint level set energy function which considered the distance map about the initial single cell boundary and elliptical shape prior makes it possible for the proposed approach to get a smooth cell contour of Pap images. Here, both distance map and shape constraint prior can prevent the excessive evolution of level set. The evaluation of contour extracted results proved that the proposed fusion scheme was effective and reliable in overlapping cervical cell segmentation tasks.The main conclusions of the study may be summarized as follows: (i) nucleus is the best biomarker and plays an important role in disease diagnosis. Each cell contains a nucleus, and segmenting nuclei is generally easier than segmenting individual cytoplasm. Actually, the nuclei detection and segmentation method GGCMM proposed in this work is a general segmentation algorithm, which could be applied to various natural images. (ii) In addition to the cervical smear images, the proposed approach is also applicable to other kinds of cell images. (iii) One question raised from this work is how to select the proper weight factorsα, β, and γ in (17). (iv) Another limitation is that Cauchy distribution is applied to the proposed cell clump extraction, which increases the complexity of the algorithm. One possible solution to overcome this problem is to apply deep learning techniques. However, the deep learning methods require a large number of cervical cell images and high time consumption. In our future work, we will explore these possibilities. In addition, we train the method on the training set provided by the ISBI 2014 Challenge Dataset. This dataset does not contain images of serious cervical tumors. In reality, however, with the growth of the disease in the cell, the nucleus almost occupied the cytoplasmic area. Sometimes, the entire cytoplasm is not visible in the image. In the future, we will evaluate the effectiveness of our method on the cervical cell with malignant tumors.



---

*Source: 3728572-2020-04-07.xml*