# An Automatic Indirect Immunofluorescence Cell Segmentation System

Authors: Yung-Kuan Chan; Der-Chen Huang; Kuo-Ching Liu; Rong-Tai Chen; Xiaoyi Jiang
Journal: Mathematical Problems in Engineering
 (2014)

---

## Abstract

Indirect immunofluorescence (IIF) with HEp-2 cells has been used for the detection of antinuclear autoantibodies (ANA) in systemic autoimmune diseases. The ANA testing allows us to scan a broad range of autoantibody entities and to describe them by distinct fluorescence patterns. Automatic inspection for fluorescence patterns in an IIF image can assist physicians, without relevant experience, in making correct diagnosis. How to segment the cells from an IIF image is essential in developing an automatic inspection system for ANA testing. This paper focuses on the cell detection and segmentation; an efficient method is proposed for automatically detecting the cells with fluorescence pattern in an IIF image. Cell culture is a process in which cells grow under control. Cell counting technology plays an important role in measuring the cell density in a culture tank. Moreover, assessing medium suitability, determining population doubling times, and monitoring cell growth in cultures all require a means of quantifying cell population. The proposed method also can be used to count the cells from an image taken under a fluorescence microscope.


---

## Body

## 1. Introduction

IIF with HEp-2 cells has been frequently employed to detect ANA in systemic autoimmune diseases [1]. ANA testing can be used to scan a broad range of autoantibody entities and to describe them by distinct fluorescence patterns. The fluorescence patterns are usually identified by physician manually inspecting the slides with the help of a microscope. However, due to lacking in satisfying automation of inspection and a low level of standardization, this procedure still needs highly specialized and experienced technician or physician to obtain diagnostic result. For this purpose, automatic inspection for fluorescence patterns in an IIF image may assist physicians, without relevant experience, in making correct diagnosis. As ANA testing becomes more popularly used, a functional automatic inspection system is essential and its clinical application becomes more urgent [2].Roughly, IIF with HEp-2 cells can be classified into six different main patterns: diffuse, peripheral, coarse speckled, fine speckled, discrete speckled, and nucleolar patterns. Figure1 shows the six distinct autoantibody fluorescence patterns in IIF images. From an image processing point of view, the fluorescence cell belonging to diffuse, peripheral, coarse speckled, or fine speckled pattern normally includes only one connected region. On the contrary, the discrete speckled and nucleolar patterns contain mass and some bright spots [3].Figure 1
Six main patterns of IIF images.Ideally, a fluorescent image [4] shows only the structure of interest that is labelled with fluorescent dye, while the responses of unstained cells remain unobserved. Specificity of dyes with respect to cell types is sufficient for identifying supporting cells and receptor cells. The application of an array of fluorochromes has made it possible to identify cells and submicroscopic cellular components with a high degree of specificity amid nonfluorescing material. Many different fluorescent dyes can be used to stain different structures or chemical compounds. The fluorescence microscope is capable of revealing the presence of a single molecule. Through multiple fluorescent labelling, different probes can simultaneously identify several target molecules.Segmenting cells from an IIF image plays an important role in developing an automatic ANA inspection system. Many cell segmentation methods [1, 5–7] are available to separate cells from an image but cannot be employed to cut cells off well from a fluorescence image. Most of them adopt a global threshold to convert a gray-level image into a binary image, but the intensities of cells in a fluorescence image may lie in an extremely wide range. It is very difficult to detect all cells using a global threshold only. Hence, the methods mentioned above cannot often give a good segmentation result for fluorescence images.An automatic ANA inspection system can be divided into HEp-2 cell segmentation, fluorescence pattern classification, and computer aided diagnosis phases. This study will focus on the first phase, developing an effective method for automatically cutting off the cells with fluorescence pattern from IIF images.In this study, a color selector is presented to transform a color IIF image into a gray-level image, a run length enhancer is provided to suppress the hazy fluorescent halo, an adaptive filter is proposed to smooth the surfaces as well as filling in the holes on the cells, and a gradient computing method is given to effectively compute the gradients of an image. These proposed techniques can be also applied to segment objects from other kinds of images. In addition, watershed and distance transform techniques are used to split overlapping objects. Since the characteristics of the six cell patterns in IIF images are extremely different, a rough classifier is proposed to decide which set of the parameter values should be employed in the proposed segmentation method for segmenting cells from the IIF images with different patterns.Cell culture [8] is the process by which prokaryotic, eukaryotic, or plant cells are cultivated under controlled conditions [9]. This process allows individual cells to act as an independent unit. In cell culture, the size and quantity of cells will increase during cell division, and the growth of cells will be constrained by culture variables such as nutrient depletion. Culture techniques have become an important and extensive part of biotechnology in applied research areas such as genetics, cytology, and pathology. Cell culture also brings huge economic benefits and prospects to agriculture, crop genetics, crop breeding, and food additives.Since the growing cells must be separated into two or more dishes to avoid excessive growth in cell culture, biologists need to know the precise number or density of cells. Quantity measurement is an important task in cell culture. In addition, assessing medium suitability, determining population doubling times, and monitoring cell growth in culture all require ways to quantifying cell population. Cell quantification also allows standardization for manipulations such as transfection or cell fusion.Generally, biologists count the cells under microscope one by one or utilize the chambers. The most widely used chamber is the hemocytometer, a device designed for estimating the number of cells in a given volume under a microscope. This device separates the whole-cell samples into several large, equal squares, and each large square contains many small squares. Counting the cells in several small squares or larger squares provides an estimate of the total number of cells. However, manually counting the cells on an image that contains an extremely large number of cells is a time-consuming, arduous, and inaccurate task. An automatic software system to count the cells on an image taken under a microscope would save time and frustration and improve accuracy.The cells in an IIF image generally lie in a wide range of gray level intensities; even the intensities of some cells are too weak for human eyes to find them. The purpose of this paper is to develop a system for automatically segmenting the cells in an indirect immunofluorescence (IIF) image. We call it automatic indirect immunofluorescence cell segmentation (AIICS) system which will enhance the cell contours and cut off the cells from an IIF image. The AIICS system also can be used to count the cells on the IIF image.In the AIICS system, a color selector is proposed to transform a color IIF image into a gray-level image, a run length enhancer and an adaptive filter to get rid of noise and highlight the contours of objects on the IIF image, a new gradient computing method to compute the gradient of object contours, and an adaptive thresholding method to decide the most proper thresholds for detecting the cell contours on an IIF image. Moreover, a rough classifier is provided to classify IIF images and GAPD to decide the most suitable parameters used in the AIICS system.


## 2. Related Works

In this section, we will briefly review some cell segmentation methods [1, 5–7], the performances of which will be compared with the performance of the AIICS system in cell segmentation and counting. In addition, some measurements of object segmentation and counting errors [10] will be briefly reviewed as well.


### 2.1. Reviews

Eddins [5] presented an image segmentation strategy for isolating objects from an image. The objects could be anything: blood cells, stars, toner spots on a printed page, DNA microarray elements, or even quantum semiconductor dots. First, the strategy converts the image into a gray-scale one and uses a morphological top-hat operator with a disk-shaped structuring element to smooth out the uneven illumination. Second, Otsu’s thresholding method [11] is employed to determine a good threshold for converting the image to a binary one. Finally, this strategy computes the distance transform of the complement of the binary image, modifies it to force the background to be its own catchment basin, and then uses the watershed transform to extract objects.Althoff et al. [1] provided a cell contour segmentation method to cut off the neural stem cells from a time-lapse image sequence. The method is an iterative process for every image in the sequence and can be described as follows.(i)
Use multiscale Laplacian of Gaussian filters to separate image background from cell regions.(ii)
Select the centroids of the blobs that are most likely to be cells.(iii)
Use a dynamic programming to segment the cells.Tang and Ewert [6] also offered a method for severing neural stem cells from a sequence of images. This method first applies a Gaussian filter to remove noise from the images. It then performs fuzzy thresholding as follows: all pixels with intensity below a lower threshold tl are set to 0, and all pixels with intensity above a higher threshold th are set to 1. The gray level intensities between tl and th are linearly rescaled to the range [0,1]. The chosen lower threshold tl=μ+0.3σ, where μ is the mean value and σ is the standard deviation of the background intensity. Similarly, the chosen higher threshold th=μ+4σ is high enough to guarantee that the pixels brighter than th are really well inside the cells. Through the use of a fuzzy thresholding approach, the method becomes less sensitive to the exact values of these threshold levels than would have been the case if a standard crisp threshold had been used. Next, fuzzy gray weighted distance transform and watershed transform are used to separate cells from images.In 2008, Yan et al. [7] proposed a method for segmenting the nuclei of cells from a genome-wide RNAi screening image. The method converts an image into a gray-scale one and applies Otsu’s thresholding method [11] to decide a suitable threshold for converting the gray-scale image to a binary image. It then computes the distance transform of the complement of the binary image, employs an enhanced watershed algorithm to detect the nucleus contours of cells, and removes the objects with small areas.


### 2.2. Classification and Segmentation Measures

Positive predictive value and sensitivity are two most frequently used criteria to evaluate the effectiveness of a classifier. Positive predictive value can be seen as a measure of exactness or fidelity, whereas sensitivity is a measure of completeness. Table1 lists all possible classification conditions in a binary classification. The positive predictive value PPV and sensitivity SEN are defined as follows [10]:
(1)PPV=TPTP+FP,SEN=TPTP+FN.Table 1
Classification condition.
Ground truth
Cluster 1
Cluster 2
Test outcome
Cluster 1
True positive (TP)
False positive(FP)
Cluster 2
False negative (FN)
True negative(TN)There may be a trade-off problem between positive predictive value and sensitivity; greater positive predictive value decreases sensitivity and greater sensitivity leads to decreased positive predictive value.F-measure F is the harmonic mean of PPV and SEN and takes account of both measures [12]:
(2)F=21/PPV+1/SEN=2TP2TP+FN+FP.Appropriate validation of segmentation is important for clinical acceptance of a segmentation method. The positive predictive value, sensitivity, and detection accuracy (DACC) are also often employed to measure the segmentation accuracy of a segmentation method. DACC can be defined as follows:(3)DACC=TP′TP′+FP′+FN′.Here,TP′ is the number of objects which are detected by a segmentation method and appear in ground truth; FP′ is the number of objects that are detected by the segmentation method but do not appear in ground truth; FN′ is the number of objects that are not detected by the segmentation method but appear in the ground truth. In this paper, we will take PPV, SEN, and DACC to measure the performance of the AIICS system.


## 2.1. Reviews

Eddins [5] presented an image segmentation strategy for isolating objects from an image. The objects could be anything: blood cells, stars, toner spots on a printed page, DNA microarray elements, or even quantum semiconductor dots. First, the strategy converts the image into a gray-scale one and uses a morphological top-hat operator with a disk-shaped structuring element to smooth out the uneven illumination. Second, Otsu’s thresholding method [11] is employed to determine a good threshold for converting the image to a binary one. Finally, this strategy computes the distance transform of the complement of the binary image, modifies it to force the background to be its own catchment basin, and then uses the watershed transform to extract objects.Althoff et al. [1] provided a cell contour segmentation method to cut off the neural stem cells from a time-lapse image sequence. The method is an iterative process for every image in the sequence and can be described as follows.(i)
Use multiscale Laplacian of Gaussian filters to separate image background from cell regions.(ii)
Select the centroids of the blobs that are most likely to be cells.(iii)
Use a dynamic programming to segment the cells.Tang and Ewert [6] also offered a method for severing neural stem cells from a sequence of images. This method first applies a Gaussian filter to remove noise from the images. It then performs fuzzy thresholding as follows: all pixels with intensity below a lower threshold tl are set to 0, and all pixels with intensity above a higher threshold th are set to 1. The gray level intensities between tl and th are linearly rescaled to the range [0,1]. The chosen lower threshold tl=μ+0.3σ, where μ is the mean value and σ is the standard deviation of the background intensity. Similarly, the chosen higher threshold th=μ+4σ is high enough to guarantee that the pixels brighter than th are really well inside the cells. Through the use of a fuzzy thresholding approach, the method becomes less sensitive to the exact values of these threshold levels than would have been the case if a standard crisp threshold had been used. Next, fuzzy gray weighted distance transform and watershed transform are used to separate cells from images.In 2008, Yan et al. [7] proposed a method for segmenting the nuclei of cells from a genome-wide RNAi screening image. The method converts an image into a gray-scale one and applies Otsu’s thresholding method [11] to decide a suitable threshold for converting the gray-scale image to a binary image. It then computes the distance transform of the complement of the binary image, employs an enhanced watershed algorithm to detect the nucleus contours of cells, and removes the objects with small areas.


## 2.2. Classification and Segmentation Measures

Positive predictive value and sensitivity are two most frequently used criteria to evaluate the effectiveness of a classifier. Positive predictive value can be seen as a measure of exactness or fidelity, whereas sensitivity is a measure of completeness. Table1 lists all possible classification conditions in a binary classification. The positive predictive value PPV and sensitivity SEN are defined as follows [10]:
(1)PPV=TPTP+FP,SEN=TPTP+FN.Table 1
Classification condition.
Ground truth
Cluster 1
Cluster 2
Test outcome
Cluster 1
True positive (TP)
False positive(FP)
Cluster 2
False negative (FN)
True negative(TN)There may be a trade-off problem between positive predictive value and sensitivity; greater positive predictive value decreases sensitivity and greater sensitivity leads to decreased positive predictive value.F-measure F is the harmonic mean of PPV and SEN and takes account of both measures [12]:
(2)F=21/PPV+1/SEN=2TP2TP+FN+FP.Appropriate validation of segmentation is important for clinical acceptance of a segmentation method. The positive predictive value, sensitivity, and detection accuracy (DACC) are also often employed to measure the segmentation accuracy of a segmentation method. DACC can be defined as follows:(3)DACC=TP′TP′+FP′+FN′.Here,TP′ is the number of objects which are detected by a segmentation method and appear in ground truth; FP′ is the number of objects that are detected by the segmentation method but do not appear in ground truth; FN′ is the number of objects that are not detected by the segmentation method but appear in the ground truth. In this paper, we will take PPV, SEN, and DACC to measure the performance of the AIICS system.


## 3. AIICS System

The AIICS system contains two stages: preprocessing and cell segmenting. The preprocessing stage is to enhance cell contours and suppress noise contours for facilitating further cell segmenting. The cell segmenting stage is to cut off cells from an IIF image. The AIICS system will take different parameters for segmenting cells in different type IIF images. Figure2 shows the framework of the AIICS system.Figure 2
The framework of the AIICS system.


### 3.1. Preprocessing

In the preprocessing stage, the AIICS system first transforms a color IIF imagefRGB into a gray-level image f and then employs a run length enhancer and an adaptive filter to eliminate noise and highlight the contours of objects on f.


#### 3.1.1. Color Selector

Since antibodies bind stably and specifically to their corresponding antigen, they are invaluable as probes to identify a particular molecule in cells, tissues, or biological fluids. Antibody molecules can be used by a variety of different labelling techniques to locate their target molecules accurately in single cells or in tissue sections. When the antibody itself or the anti-immunoglobulin antibody is detected, it is labelled with a fluorescent dye; the technique is known as immunofluorescence microscopy. Under the indirect immunofluorescence microscopy, the chosen dye-antibody complex binds only to specific proteins in the cell. Cells may be stained by different dyes to different colors. The list of most available fluorescent labels includes red, green, blue, cyan, or yellow fluorescent proteins.To remove the effect of dye color on cell segmentation, the AIICS system transforms each color IIF imagefRGB into a gray-level image f. The AIICS system computes the color histogram of fRGB with 6 bins. Let black (background color), red, green, blue, cyan, and yellow be the representative colors of the six bins, respectively, and let (CRi, CGi, CBi) be the RGB color components of the representative color of the ith bin. In fRGB, for each pixel p with three color components (CR, CG, CB), p will be thrown into bin j where
(4)j=Arg(Min⁡i=05(CRi-CR)2+(CGi-CG)2+(CBi-CB)2).Letni be the number of pixels which are thrown into bin i, and k is defined as k=Arg(Max⁡i=15ni). The AIICS system then projects the hth pixel with three color components (CR, CG, CB) in fRGB on a line passing through (0,0,0) and (CRk, CGk, CBk), and assigns the value, obtained by projecting (CR, CG, CB) on the line, to the gray-level intensity of the hth pixel of f0. Let max and min be the maximal and minimal gray-level intensities of all the pixels in f0. The AIICS system then changes the gray-level intensity x of each pixel in f0 into ((x-min⁡)/(max⁡-min⁡))×255. This operation is to stretch the contrasts of f0 from 0 to 255.


#### 3.1.2. Run Length Enhancer

In IIF images, cells are bright objects protruding out from a uniform dark background. However, some cells are often surrounded by a hazy fluorescent halo, such as the imagein Figure3(a). The hazy fluorescent halo will make cell segmentation more difficult. Therefore, this paper proposed a run length enhancer to suppress the hazy fluorescent halo. For each pixel f0(i,j) located at the coordinates (i,j) on f0, the run length enhancer draws eight line segments l0,l1,…,l7, all of which pass through f0(i,j), and the included angle between lh and X-axis is 22.5×h°. Let Wr, consisting of mr×mr pixels, be a window on f0, where f0(i,j) is the central pixel of Wr. We call Wr the related window of f0(i,j). Assume that lh passes through pixels Ph1 and Ph2 which are located on the boundary of Wr. Figure 4 illustrates the related window Wr of f0(i,j) marked by red color. If the gray-level intensity of f0(i,j) is greater than the average gray-level intensity of all the pixels in Wr, then the run length enhancer assigns Max⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j); otherwise it gives Min⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j), where chk is the gray-level intensity of each pixel on line segment lh from Ph1 to Ph2.The result of preprocessing stage.
(a)
f
0
(b)
f
r
(c)
f
aFigure 4
The eight line segments in run length enhancer.The run length enhancer then stretches the gray-level contrast offr from 0 to 255. Figure 3(b) is the fr obtained by executing the run length enhancer on image f0 in Figure 3(a). Figure 3(b) explains that the run length enhancer can suppress the hazy fluorescent halo.


#### 3.1.3. Adaptive Filter

Figure3(b) shows that some cells are uneven and there may exist some holes on the cells in fr, indicated by a red arrow; therefore, the AIICS system adopts an adaptive filter to smooth the surfaces and to fill in the holes on the cells. Let Aveg be the average gray-level intensity of all the pixels in fr, let Wa, consisting of ma×ma pixels, be a related window of fr(i,j), and let Avel as well as Min⁡a be the average and minimal gray-level intensities of all the pixels in Wa. Since cell pixels in fr are generally brighter than background pixels, the adaptive filter gives
(5)Avel,ifAvel≥Aveg,Min⁡a,otherwise,
to fa(i,j). After that, the adaptive filter stretches the pixel contrast of fa from 0 to 255. Figure 3(c) is the fa obtained by executing the adaptive filter on the image in Figure 3(b). Figure 3(c) shows that the adaptive filter can smooth the surfaces and fill in the holes on the cells in fr.


### 3.2. Cell Segmenting

This stage is to cut off cells from an IIF image. It consists of gradient computing, contour extracting, and overlapping cells splitting steps.


#### 3.2.1. Gradient Computing

The gradients of pixels in an image can provide an abundance of object contour information. Let∇fa denote the gradient vector field of pixel fa(i,j) where ∇ is the vector differential operator. The components of ∇fa are the partial derivatives of fa(i,j). That is,
(6)∇fa=[gx,gy]=[∂fa∂x,∂fa∂y].
The magnitude of the vector is given to
(7)Mag(∇fa)=(gx2+gy2)1/2=((∂fa∂x)2+(∂fa∂y)2)1/2.Letgxx, gxy, gyx, and gyy be
(8)gxx=∂2fa∂x2,gxy=∂2fa∂x∂y,gyx=∂2fa∂y∂x,gyy=∂2fa∂y2,
and let v=(vx,vy) be a unit vector in XY plane. We define the local contrast of fa(i,j) at the XY plane in the direction of v as
(9)gxxvx2+2gxyvxvy+gyyvy2,A=(gxxgyxgxygyy).It is well known that a quadratic form equation (9) has a maximal and a minimal value for varying v. These extreme values coincide with the eigenvalues of matrix A, and they are attained when v is the corresponding eigenvector [13]. The extreme values can be obtained by (10)
(10)λ±=gxx+gyy2±(gxx-gyy)24+gxygyx.For each pixelfa(i,j), a pair (λ+,λ-) is given, so that λ+ is the greatest gradient magnitude of fa(i,j) in a certain gradient direction θ+; λ- is the gradient of fa(i,j) in other gradient direction θ- and the included angle between θ+ and θ- is 90°.After computing theλ+ and λ- of all the pixels in fa, the λ+ and λ- of fa(i,j) are, respectively, transformed into
(11)f+(i,j)=255×λ+-min+max+-min+,f-(i,j)=255×λ--min-max--min-,
where max+ and min+ are the maximum and the minimum λ+ of all the pixels in fa and max- and min- are the maximum and the minimum λ- of all the pixels in fa. Figures 5(b) and 5(c) demonstrate the f+ and f- obtained by the gradient computing from the image fa shown in Figure 5(a).The example results obtained by the gradient computing.
(a)
f
a
(b)
f
+
(c)
f
-
(d)
f
±From Figure5, one can obviously observe that f+ and f- can effectively describe the gradient of fa but the object contours in f+ (resp. f-) are expanded (resp. shrunk) compared to the real object contours in fa. Hence, f+ and f- are integrated into f± by the following formula:
(12)f±(i,j)=f+(i,j)(255-f-(i,j))+1,
where the 1 in denominator is to ensure the value in denominator not to be zero.Next, the contrasts of all the pixels inf± are stretched to range from 0 to 255. Figure 5(d) is the f± obtained by executing the gradient computing approach on the image in Figure 5(a). Figure 5(d) shows that the gradient computing approach can provide the impressive gradient of an image.


#### 3.2.2. Contour Extracting

Since the gradients of some object contours are indistinct, such as the gradient indicated by a red arrow in Figure5(d), the AIICS system takes an adaptive thresholding approach to transform f± into a binary image fb in which a 1-bit (resp. 0-bit) represents one white pixel (resp. black pixel). The adaptive thresholding approach first uses Otsu’s thresholding method [11] to partition all the pixels in f± into two groups according to their gray-level intensities, then respectively computes the average gray-level intensity of all the pixels in each group, and considers the smaller one μb of the two average gray-level intensities to be the gray-level intensity of the background pixels in f±.LetWb, consisting of mb×mb pixels, be the related window of f±(x,y), let μl be the average gray-level intensity of all the pixels in Wb, and let σg be the standard deviation of the gray-level intensities of all the pixels in f±. Two thresholds T1 and T2 are set to
(13)T1=μb+σg,T2=μb+σg2.
The fb(x,y) is given to be
(14)fb(x,y)={1,(fg(x,y)>T1)or(T2<fg(x,y)≤T1andμl>T1),0,otherwise.Figure6(a) illustrates the fb generated by the adaptive thresholding approach from the image in Figure 5(d).The example results obtained by the thinning and spur pruning operations.
(a)
f
b
(b)
f
t
(c)
f
s
(d)
f
cSince the contrast of the pixel at the vicinity of the cell contour or noise is generally high, it may cause a false contour with the thickness of over one pixel. The contour of an object should be one pixel in thickness. The AIICS system adopts a hit-and-miss transform-based skeletonization (HMTS) algorithm [14] to thin the contour of an object so that the contour of an object is the thickness of one pixel. We refer to the eliminated candidate contour pixels as the redundant-contour pixels and the remaining contour pixels as true-contour pixels. We call this algorithm the thinning operation.LetWt(x,y), consisting of 3×3 pixels, be the related window of fb(x,y). The thinning operation compares Wt(x,y) with each of the eight structuring elements shown in Figure 7, where the gray pixels stand for the do-not-care pixels (A do-not-care pixel may be a 1-bit pixel or a 0-bit pixel). The thinning operation considers that Wt(x,y) is matched if the positions and values of 1- and 0-bits on one structuring element are completely the same as those on Wt(x,y), regardless of the do-not-care pixels. If Wt(x,y) is matched, ft(x,y)=0; otherwise, ft(x,y)=fb(x,y). The algorithm repeats this procedure until no more thinning can be performed. The thinning algorithm is to cut off the redundant-contour pixels, so that the contours have a thickness of only one pixel. Figure 6(b) shows the result after running the thinning operation on the image in Figure 6(a).Figure 7
Eight structuring elements for thinning.However, the uneven object contours tend to cause small spurs on the obtained skeleton by the thinning operation. These spurs are not expected. Therefore, a spur pruning operation is required to remove them. The procedure of the spur pruning operation [14] is entirely the same as the thinning operation except for the eight structuring elements in Figure 7 which are replaced by the eight structuring elements in Figure 8. Let fs be the result obtained by the spur pruning operation from ft. Figure 6(c) displays the result image fs after running the spur pruning operation on the image ft shown in Figure 6(b).Figure 8
Eight structuring elements for Spurs.Infs, some false contours should be removed. Let Pi and Pj be two pixels on the contour of the kth object Ok in fs, and let dij be the distance between Pi and Pj. Assume dhk is the maximal one among all the dij’s on the contour of Ok. We call dhk the major length of Ok. If the major length of Ok is less than the one-third of the average major length of all the object contours in fs, then Ok is considered to be noise and removed from fs. After that, fs in Figure 6(c) is changed into fc in Figure 6(d).


#### 3.2.3. Overlapping Cells Splitting

Some juxtaposed objects may share overlapping regions, which will make cell distinguishing and counting imprecise. We call the area containing some touching or overlapping cells an overlapping area. The watershed algorithm is commonly used to split the overlapping object [15–18]. Watershed based segmentation can give good results for gray level images with different minima and catchment basins. For a binary image, however, there are only two gray levels 0 and 1, respectively, standing for background and cell. If two cells are connected together in the binary image fc, only one minimum and catchment basin will be formed in the topographic surface. In employing the watershed algorithm to segment connected cells, distance transform [19] is often adopted to preprocess the image to make it more suitable for segmentation by watershed algorithm, for assigning the catchment basins in the overlapping area.Distance transform [19, 20] labels each pixel in an overlapping area with the distance of the pixel to the nearest pixel outside the overlapping area. Let R be an overlapping area in fc. For each pixel P(x,y) in R located at the coordinates (x,y) on fc, the distance transform can be determined by the following three steps.(1)
Attach a labeld(x,y)=∞ to each pixel P(x,y) in the overlapping area on fc and give a label d(x,y)=0 to each pixel outside the overlapping area.(2)
Then travel throughfc pixel by pixel. Replace the label of each pixel P(x,y) in fc with Min⁡{d(x,y), d(x-1,y-1)+1, d(x-1,y)+1, d(x-1,y+1)+1, d(x,y-1)+1, d(x,y)+1, d(x,y+1)+1, d(x+1,y-1)+1, d(x+1,y)+1, d(x+1,y+1)+1}.(3)
Repeat step (2), until all labels have been converted to finite values.The most intuitive way to explain the watershed based segmentation is to imagine that a hole is drilled in each minimum of the surface and we flood water into different catchment basins from the holes. If the water of different catchment basins is likely to merge due to further immersion, a dam is built to prevent the merging. This flooding process will eventually reach a stage when only the top of dam (the watershed lines) is visible above the water line.LetPiPj¯ be the line segment passing through Pi and Pj which are two pixels on the contour of the kth object Ok in fc, let dij be the distance between Pi and Pj, let DM=dlm be the major length of Ok, where dlm is the longest distance between any two pixels on the contour of Ok, and let (x1,y1) as well as (x2,y2) be the coordinates of Pl and Pm on fc. Assume Dm=dl′m′ is the distance of pixels Pl′ and Pm′ on the contour of Ok, Pl′Pm′¯ passes through ((x1+x2)/2,(y1+y2)/2), and Pl′Pm′¯ is perpendicular to PlPm¯. We call D⊥ the minor length of Ok.LetAk be the area of Ok and let Aave be the average area of all the objects on fc. The AIICS system considers the object Ok to be an overlapping area and takes the watershed algorithm [17] to split Ok into more objects, only if
(15)DMD⊥>1.5,Ak>2×Aave.After that,fc is converted into other binary image fw. Figure 9(b) demonstrates the fw obtained by running this watershed algorithm on the fc of the image in Figure 9(a), where only the overlapping areas surrounded by white closed curves in Figure 9(b) are further split by the watershed algorithm.The results of overlapping cells splitting.
(a)
Original image
(b)
f
w
(c)
f
mThere may be still some cells split into more than one region. The AIICS system tries to combine them. LetμA and σA be the average and standard deviation of the areas of all the object regions on Iw. If the area of object region Ok is less than μA-0.5×σA, the AIICS system merges Ok into one object region which is adjacent to Ok and has the longest common edge with Ok. After region combination, fw is converted into fm. Figure 9(c) displays the fm obtained from the fw in Figure 9(b) by the region combination.


## 3.1. Preprocessing

In the preprocessing stage, the AIICS system first transforms a color IIF imagefRGB into a gray-level image f and then employs a run length enhancer and an adaptive filter to eliminate noise and highlight the contours of objects on f.


### 3.1.1. Color Selector

Since antibodies bind stably and specifically to their corresponding antigen, they are invaluable as probes to identify a particular molecule in cells, tissues, or biological fluids. Antibody molecules can be used by a variety of different labelling techniques to locate their target molecules accurately in single cells or in tissue sections. When the antibody itself or the anti-immunoglobulin antibody is detected, it is labelled with a fluorescent dye; the technique is known as immunofluorescence microscopy. Under the indirect immunofluorescence microscopy, the chosen dye-antibody complex binds only to specific proteins in the cell. Cells may be stained by different dyes to different colors. The list of most available fluorescent labels includes red, green, blue, cyan, or yellow fluorescent proteins.To remove the effect of dye color on cell segmentation, the AIICS system transforms each color IIF imagefRGB into a gray-level image f. The AIICS system computes the color histogram of fRGB with 6 bins. Let black (background color), red, green, blue, cyan, and yellow be the representative colors of the six bins, respectively, and let (CRi, CGi, CBi) be the RGB color components of the representative color of the ith bin. In fRGB, for each pixel p with three color components (CR, CG, CB), p will be thrown into bin j where
(4)j=Arg(Min⁡i=05(CRi-CR)2+(CGi-CG)2+(CBi-CB)2).Letni be the number of pixels which are thrown into bin i, and k is defined as k=Arg(Max⁡i=15ni). The AIICS system then projects the hth pixel with three color components (CR, CG, CB) in fRGB on a line passing through (0,0,0) and (CRk, CGk, CBk), and assigns the value, obtained by projecting (CR, CG, CB) on the line, to the gray-level intensity of the hth pixel of f0. Let max and min be the maximal and minimal gray-level intensities of all the pixels in f0. The AIICS system then changes the gray-level intensity x of each pixel in f0 into ((x-min⁡)/(max⁡-min⁡))×255. This operation is to stretch the contrasts of f0 from 0 to 255.


### 3.1.2. Run Length Enhancer

In IIF images, cells are bright objects protruding out from a uniform dark background. However, some cells are often surrounded by a hazy fluorescent halo, such as the imagein Figure3(a). The hazy fluorescent halo will make cell segmentation more difficult. Therefore, this paper proposed a run length enhancer to suppress the hazy fluorescent halo. For each pixel f0(i,j) located at the coordinates (i,j) on f0, the run length enhancer draws eight line segments l0,l1,…,l7, all of which pass through f0(i,j), and the included angle between lh and X-axis is 22.5×h°. Let Wr, consisting of mr×mr pixels, be a window on f0, where f0(i,j) is the central pixel of Wr. We call Wr the related window of f0(i,j). Assume that lh passes through pixels Ph1 and Ph2 which are located on the boundary of Wr. Figure 4 illustrates the related window Wr of f0(i,j) marked by red color. If the gray-level intensity of f0(i,j) is greater than the average gray-level intensity of all the pixels in Wr, then the run length enhancer assigns Max⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j); otherwise it gives Min⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j), where chk is the gray-level intensity of each pixel on line segment lh from Ph1 to Ph2.The result of preprocessing stage.
(a)
f
0
(b)
f
r
(c)
f
aFigure 4
The eight line segments in run length enhancer.The run length enhancer then stretches the gray-level contrast offr from 0 to 255. Figure 3(b) is the fr obtained by executing the run length enhancer on image f0 in Figure 3(a). Figure 3(b) explains that the run length enhancer can suppress the hazy fluorescent halo.


### 3.1.3. Adaptive Filter

Figure3(b) shows that some cells are uneven and there may exist some holes on the cells in fr, indicated by a red arrow; therefore, the AIICS system adopts an adaptive filter to smooth the surfaces and to fill in the holes on the cells. Let Aveg be the average gray-level intensity of all the pixels in fr, let Wa, consisting of ma×ma pixels, be a related window of fr(i,j), and let Avel as well as Min⁡a be the average and minimal gray-level intensities of all the pixels in Wa. Since cell pixels in fr are generally brighter than background pixels, the adaptive filter gives
(5)Avel,ifAvel≥Aveg,Min⁡a,otherwise,
to fa(i,j). After that, the adaptive filter stretches the pixel contrast of fa from 0 to 255. Figure 3(c) is the fa obtained by executing the adaptive filter on the image in Figure 3(b). Figure 3(c) shows that the adaptive filter can smooth the surfaces and fill in the holes on the cells in fr.


## 3.1.1. Color Selector

Since antibodies bind stably and specifically to their corresponding antigen, they are invaluable as probes to identify a particular molecule in cells, tissues, or biological fluids. Antibody molecules can be used by a variety of different labelling techniques to locate their target molecules accurately in single cells or in tissue sections. When the antibody itself or the anti-immunoglobulin antibody is detected, it is labelled with a fluorescent dye; the technique is known as immunofluorescence microscopy. Under the indirect immunofluorescence microscopy, the chosen dye-antibody complex binds only to specific proteins in the cell. Cells may be stained by different dyes to different colors. The list of most available fluorescent labels includes red, green, blue, cyan, or yellow fluorescent proteins.To remove the effect of dye color on cell segmentation, the AIICS system transforms each color IIF imagefRGB into a gray-level image f. The AIICS system computes the color histogram of fRGB with 6 bins. Let black (background color), red, green, blue, cyan, and yellow be the representative colors of the six bins, respectively, and let (CRi, CGi, CBi) be the RGB color components of the representative color of the ith bin. In fRGB, for each pixel p with three color components (CR, CG, CB), p will be thrown into bin j where
(4)j=Arg(Min⁡i=05(CRi-CR)2+(CGi-CG)2+(CBi-CB)2).Letni be the number of pixels which are thrown into bin i, and k is defined as k=Arg(Max⁡i=15ni). The AIICS system then projects the hth pixel with three color components (CR, CG, CB) in fRGB on a line passing through (0,0,0) and (CRk, CGk, CBk), and assigns the value, obtained by projecting (CR, CG, CB) on the line, to the gray-level intensity of the hth pixel of f0. Let max and min be the maximal and minimal gray-level intensities of all the pixels in f0. The AIICS system then changes the gray-level intensity x of each pixel in f0 into ((x-min⁡)/(max⁡-min⁡))×255. This operation is to stretch the contrasts of f0 from 0 to 255.


## 3.1.2. Run Length Enhancer

In IIF images, cells are bright objects protruding out from a uniform dark background. However, some cells are often surrounded by a hazy fluorescent halo, such as the imagein Figure3(a). The hazy fluorescent halo will make cell segmentation more difficult. Therefore, this paper proposed a run length enhancer to suppress the hazy fluorescent halo. For each pixel f0(i,j) located at the coordinates (i,j) on f0, the run length enhancer draws eight line segments l0,l1,…,l7, all of which pass through f0(i,j), and the included angle between lh and X-axis is 22.5×h°. Let Wr, consisting of mr×mr pixels, be a window on f0, where f0(i,j) is the central pixel of Wr. We call Wr the related window of f0(i,j). Assume that lh passes through pixels Ph1 and Ph2 which are located on the boundary of Wr. Figure 4 illustrates the related window Wr of f0(i,j) marked by red color. If the gray-level intensity of f0(i,j) is greater than the average gray-level intensity of all the pixels in Wr, then the run length enhancer assigns Max⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j); otherwise it gives Min⁡h=07((1/mr)∑kchk) to the gray-level intensity of fr(i,j), where chk is the gray-level intensity of each pixel on line segment lh from Ph1 to Ph2.The result of preprocessing stage.
(a)
f
0
(b)
f
r
(c)
f
aFigure 4
The eight line segments in run length enhancer.The run length enhancer then stretches the gray-level contrast offr from 0 to 255. Figure 3(b) is the fr obtained by executing the run length enhancer on image f0 in Figure 3(a). Figure 3(b) explains that the run length enhancer can suppress the hazy fluorescent halo.


## 3.1.3. Adaptive Filter

Figure3(b) shows that some cells are uneven and there may exist some holes on the cells in fr, indicated by a red arrow; therefore, the AIICS system adopts an adaptive filter to smooth the surfaces and to fill in the holes on the cells. Let Aveg be the average gray-level intensity of all the pixels in fr, let Wa, consisting of ma×ma pixels, be a related window of fr(i,j), and let Avel as well as Min⁡a be the average and minimal gray-level intensities of all the pixels in Wa. Since cell pixels in fr are generally brighter than background pixels, the adaptive filter gives
(5)Avel,ifAvel≥Aveg,Min⁡a,otherwise,
to fa(i,j). After that, the adaptive filter stretches the pixel contrast of fa from 0 to 255. Figure 3(c) is the fa obtained by executing the adaptive filter on the image in Figure 3(b). Figure 3(c) shows that the adaptive filter can smooth the surfaces and fill in the holes on the cells in fr.


## 3.2. Cell Segmenting

This stage is to cut off cells from an IIF image. It consists of gradient computing, contour extracting, and overlapping cells splitting steps.


### 3.2.1. Gradient Computing

The gradients of pixels in an image can provide an abundance of object contour information. Let∇fa denote the gradient vector field of pixel fa(i,j) where ∇ is the vector differential operator. The components of ∇fa are the partial derivatives of fa(i,j). That is,
(6)∇fa=[gx,gy]=[∂fa∂x,∂fa∂y].
The magnitude of the vector is given to
(7)Mag(∇fa)=(gx2+gy2)1/2=((∂fa∂x)2+(∂fa∂y)2)1/2.Letgxx, gxy, gyx, and gyy be
(8)gxx=∂2fa∂x2,gxy=∂2fa∂x∂y,gyx=∂2fa∂y∂x,gyy=∂2fa∂y2,
and let v=(vx,vy) be a unit vector in XY plane. We define the local contrast of fa(i,j) at the XY plane in the direction of v as
(9)gxxvx2+2gxyvxvy+gyyvy2,A=(gxxgyxgxygyy).It is well known that a quadratic form equation (9) has a maximal and a minimal value for varying v. These extreme values coincide with the eigenvalues of matrix A, and they are attained when v is the corresponding eigenvector [13]. The extreme values can be obtained by (10)
(10)λ±=gxx+gyy2±(gxx-gyy)24+gxygyx.For each pixelfa(i,j), a pair (λ+,λ-) is given, so that λ+ is the greatest gradient magnitude of fa(i,j) in a certain gradient direction θ+; λ- is the gradient of fa(i,j) in other gradient direction θ- and the included angle between θ+ and θ- is 90°.After computing theλ+ and λ- of all the pixels in fa, the λ+ and λ- of fa(i,j) are, respectively, transformed into
(11)f+(i,j)=255×λ+-min+max+-min+,f-(i,j)=255×λ--min-max--min-,
where max+ and min+ are the maximum and the minimum λ+ of all the pixels in fa and max- and min- are the maximum and the minimum λ- of all the pixels in fa. Figures 5(b) and 5(c) demonstrate the f+ and f- obtained by the gradient computing from the image fa shown in Figure 5(a).The example results obtained by the gradient computing.
(a)
f
a
(b)
f
+
(c)
f
-
(d)
f
±From Figure5, one can obviously observe that f+ and f- can effectively describe the gradient of fa but the object contours in f+ (resp. f-) are expanded (resp. shrunk) compared to the real object contours in fa. Hence, f+ and f- are integrated into f± by the following formula:
(12)f±(i,j)=f+(i,j)(255-f-(i,j))+1,
where the 1 in denominator is to ensure the value in denominator not to be zero.Next, the contrasts of all the pixels inf± are stretched to range from 0 to 255. Figure 5(d) is the f± obtained by executing the gradient computing approach on the image in Figure 5(a). Figure 5(d) shows that the gradient computing approach can provide the impressive gradient of an image.


### 3.2.2. Contour Extracting

Since the gradients of some object contours are indistinct, such as the gradient indicated by a red arrow in Figure5(d), the AIICS system takes an adaptive thresholding approach to transform f± into a binary image fb in which a 1-bit (resp. 0-bit) represents one white pixel (resp. black pixel). The adaptive thresholding approach first uses Otsu’s thresholding method [11] to partition all the pixels in f± into two groups according to their gray-level intensities, then respectively computes the average gray-level intensity of all the pixels in each group, and considers the smaller one μb of the two average gray-level intensities to be the gray-level intensity of the background pixels in f±.LetWb, consisting of mb×mb pixels, be the related window of f±(x,y), let μl be the average gray-level intensity of all the pixels in Wb, and let σg be the standard deviation of the gray-level intensities of all the pixels in f±. Two thresholds T1 and T2 are set to
(13)T1=μb+σg,T2=μb+σg2.
The fb(x,y) is given to be
(14)fb(x,y)={1,(fg(x,y)>T1)or(T2<fg(x,y)≤T1andμl>T1),0,otherwise.Figure6(a) illustrates the fb generated by the adaptive thresholding approach from the image in Figure 5(d).The example results obtained by the thinning and spur pruning operations.
(a)
f
b
(b)
f
t
(c)
f
s
(d)
f
cSince the contrast of the pixel at the vicinity of the cell contour or noise is generally high, it may cause a false contour with the thickness of over one pixel. The contour of an object should be one pixel in thickness. The AIICS system adopts a hit-and-miss transform-based skeletonization (HMTS) algorithm [14] to thin the contour of an object so that the contour of an object is the thickness of one pixel. We refer to the eliminated candidate contour pixels as the redundant-contour pixels and the remaining contour pixels as true-contour pixels. We call this algorithm the thinning operation.LetWt(x,y), consisting of 3×3 pixels, be the related window of fb(x,y). The thinning operation compares Wt(x,y) with each of the eight structuring elements shown in Figure 7, where the gray pixels stand for the do-not-care pixels (A do-not-care pixel may be a 1-bit pixel or a 0-bit pixel). The thinning operation considers that Wt(x,y) is matched if the positions and values of 1- and 0-bits on one structuring element are completely the same as those on Wt(x,y), regardless of the do-not-care pixels. If Wt(x,y) is matched, ft(x,y)=0; otherwise, ft(x,y)=fb(x,y). The algorithm repeats this procedure until no more thinning can be performed. The thinning algorithm is to cut off the redundant-contour pixels, so that the contours have a thickness of only one pixel. Figure 6(b) shows the result after running the thinning operation on the image in Figure 6(a).Figure 7
Eight structuring elements for thinning.However, the uneven object contours tend to cause small spurs on the obtained skeleton by the thinning operation. These spurs are not expected. Therefore, a spur pruning operation is required to remove them. The procedure of the spur pruning operation [14] is entirely the same as the thinning operation except for the eight structuring elements in Figure 7 which are replaced by the eight structuring elements in Figure 8. Let fs be the result obtained by the spur pruning operation from ft. Figure 6(c) displays the result image fs after running the spur pruning operation on the image ft shown in Figure 6(b).Figure 8
Eight structuring elements for Spurs.Infs, some false contours should be removed. Let Pi and Pj be two pixels on the contour of the kth object Ok in fs, and let dij be the distance between Pi and Pj. Assume dhk is the maximal one among all the dij’s on the contour of Ok. We call dhk the major length of Ok. If the major length of Ok is less than the one-third of the average major length of all the object contours in fs, then Ok is considered to be noise and removed from fs. After that, fs in Figure 6(c) is changed into fc in Figure 6(d).


### 3.2.3. Overlapping Cells Splitting

Some juxtaposed objects may share overlapping regions, which will make cell distinguishing and counting imprecise. We call the area containing some touching or overlapping cells an overlapping area. The watershed algorithm is commonly used to split the overlapping object [15–18]. Watershed based segmentation can give good results for gray level images with different minima and catchment basins. For a binary image, however, there are only two gray levels 0 and 1, respectively, standing for background and cell. If two cells are connected together in the binary image fc, only one minimum and catchment basin will be formed in the topographic surface. In employing the watershed algorithm to segment connected cells, distance transform [19] is often adopted to preprocess the image to make it more suitable for segmentation by watershed algorithm, for assigning the catchment basins in the overlapping area.Distance transform [19, 20] labels each pixel in an overlapping area with the distance of the pixel to the nearest pixel outside the overlapping area. Let R be an overlapping area in fc. For each pixel P(x,y) in R located at the coordinates (x,y) on fc, the distance transform can be determined by the following three steps.(1)
Attach a labeld(x,y)=∞ to each pixel P(x,y) in the overlapping area on fc and give a label d(x,y)=0 to each pixel outside the overlapping area.(2)
Then travel throughfc pixel by pixel. Replace the label of each pixel P(x,y) in fc with Min⁡{d(x,y), d(x-1,y-1)+1, d(x-1,y)+1, d(x-1,y+1)+1, d(x,y-1)+1, d(x,y)+1, d(x,y+1)+1, d(x+1,y-1)+1, d(x+1,y)+1, d(x+1,y+1)+1}.(3)
Repeat step (2), until all labels have been converted to finite values.The most intuitive way to explain the watershed based segmentation is to imagine that a hole is drilled in each minimum of the surface and we flood water into different catchment basins from the holes. If the water of different catchment basins is likely to merge due to further immersion, a dam is built to prevent the merging. This flooding process will eventually reach a stage when only the top of dam (the watershed lines) is visible above the water line.LetPiPj¯ be the line segment passing through Pi and Pj which are two pixels on the contour of the kth object Ok in fc, let dij be the distance between Pi and Pj, let DM=dlm be the major length of Ok, where dlm is the longest distance between any two pixels on the contour of Ok, and let (x1,y1) as well as (x2,y2) be the coordinates of Pl and Pm on fc. Assume Dm=dl′m′ is the distance of pixels Pl′ and Pm′ on the contour of Ok, Pl′Pm′¯ passes through ((x1+x2)/2,(y1+y2)/2), and Pl′Pm′¯ is perpendicular to PlPm¯. We call D⊥ the minor length of Ok.LetAk be the area of Ok and let Aave be the average area of all the objects on fc. The AIICS system considers the object Ok to be an overlapping area and takes the watershed algorithm [17] to split Ok into more objects, only if
(15)DMD⊥>1.5,Ak>2×Aave.After that,fc is converted into other binary image fw. Figure 9(b) demonstrates the fw obtained by running this watershed algorithm on the fc of the image in Figure 9(a), where only the overlapping areas surrounded by white closed curves in Figure 9(b) are further split by the watershed algorithm.The results of overlapping cells splitting.
(a)
Original image
(b)
f
w
(c)
f
mThere may be still some cells split into more than one region. The AIICS system tries to combine them. LetμA and σA be the average and standard deviation of the areas of all the object regions on Iw. If the area of object region Ok is less than μA-0.5×σA, the AIICS system merges Ok into one object region which is adjacent to Ok and has the longest common edge with Ok. After region combination, fw is converted into fm. Figure 9(c) displays the fm obtained from the fw in Figure 9(b) by the region combination.


## 3.2.1. Gradient Computing

The gradients of pixels in an image can provide an abundance of object contour information. Let∇fa denote the gradient vector field of pixel fa(i,j) where ∇ is the vector differential operator. The components of ∇fa are the partial derivatives of fa(i,j). That is,
(6)∇fa=[gx,gy]=[∂fa∂x,∂fa∂y].
The magnitude of the vector is given to
(7)Mag(∇fa)=(gx2+gy2)1/2=((∂fa∂x)2+(∂fa∂y)2)1/2.Letgxx, gxy, gyx, and gyy be
(8)gxx=∂2fa∂x2,gxy=∂2fa∂x∂y,gyx=∂2fa∂y∂x,gyy=∂2fa∂y2,
and let v=(vx,vy) be a unit vector in XY plane. We define the local contrast of fa(i,j) at the XY plane in the direction of v as
(9)gxxvx2+2gxyvxvy+gyyvy2,A=(gxxgyxgxygyy).It is well known that a quadratic form equation (9) has a maximal and a minimal value for varying v. These extreme values coincide with the eigenvalues of matrix A, and they are attained when v is the corresponding eigenvector [13]. The extreme values can be obtained by (10)
(10)λ±=gxx+gyy2±(gxx-gyy)24+gxygyx.For each pixelfa(i,j), a pair (λ+,λ-) is given, so that λ+ is the greatest gradient magnitude of fa(i,j) in a certain gradient direction θ+; λ- is the gradient of fa(i,j) in other gradient direction θ- and the included angle between θ+ and θ- is 90°.After computing theλ+ and λ- of all the pixels in fa, the λ+ and λ- of fa(i,j) are, respectively, transformed into
(11)f+(i,j)=255×λ+-min+max+-min+,f-(i,j)=255×λ--min-max--min-,
where max+ and min+ are the maximum and the minimum λ+ of all the pixels in fa and max- and min- are the maximum and the minimum λ- of all the pixels in fa. Figures 5(b) and 5(c) demonstrate the f+ and f- obtained by the gradient computing from the image fa shown in Figure 5(a).The example results obtained by the gradient computing.
(a)
f
a
(b)
f
+
(c)
f
-
(d)
f
±From Figure5, one can obviously observe that f+ and f- can effectively describe the gradient of fa but the object contours in f+ (resp. f-) are expanded (resp. shrunk) compared to the real object contours in fa. Hence, f+ and f- are integrated into f± by the following formula:
(12)f±(i,j)=f+(i,j)(255-f-(i,j))+1,
where the 1 in denominator is to ensure the value in denominator not to be zero.Next, the contrasts of all the pixels inf± are stretched to range from 0 to 255. Figure 5(d) is the f± obtained by executing the gradient computing approach on the image in Figure 5(a). Figure 5(d) shows that the gradient computing approach can provide the impressive gradient of an image.


## 3.2.2. Contour Extracting

Since the gradients of some object contours are indistinct, such as the gradient indicated by a red arrow in Figure5(d), the AIICS system takes an adaptive thresholding approach to transform f± into a binary image fb in which a 1-bit (resp. 0-bit) represents one white pixel (resp. black pixel). The adaptive thresholding approach first uses Otsu’s thresholding method [11] to partition all the pixels in f± into two groups according to their gray-level intensities, then respectively computes the average gray-level intensity of all the pixels in each group, and considers the smaller one μb of the two average gray-level intensities to be the gray-level intensity of the background pixels in f±.LetWb, consisting of mb×mb pixels, be the related window of f±(x,y), let μl be the average gray-level intensity of all the pixels in Wb, and let σg be the standard deviation of the gray-level intensities of all the pixels in f±. Two thresholds T1 and T2 are set to
(13)T1=μb+σg,T2=μb+σg2.
The fb(x,y) is given to be
(14)fb(x,y)={1,(fg(x,y)>T1)or(T2<fg(x,y)≤T1andμl>T1),0,otherwise.Figure6(a) illustrates the fb generated by the adaptive thresholding approach from the image in Figure 5(d).The example results obtained by the thinning and spur pruning operations.
(a)
f
b
(b)
f
t
(c)
f
s
(d)
f
cSince the contrast of the pixel at the vicinity of the cell contour or noise is generally high, it may cause a false contour with the thickness of over one pixel. The contour of an object should be one pixel in thickness. The AIICS system adopts a hit-and-miss transform-based skeletonization (HMTS) algorithm [14] to thin the contour of an object so that the contour of an object is the thickness of one pixel. We refer to the eliminated candidate contour pixels as the redundant-contour pixels and the remaining contour pixels as true-contour pixels. We call this algorithm the thinning operation.LetWt(x,y), consisting of 3×3 pixels, be the related window of fb(x,y). The thinning operation compares Wt(x,y) with each of the eight structuring elements shown in Figure 7, where the gray pixels stand for the do-not-care pixels (A do-not-care pixel may be a 1-bit pixel or a 0-bit pixel). The thinning operation considers that Wt(x,y) is matched if the positions and values of 1- and 0-bits on one structuring element are completely the same as those on Wt(x,y), regardless of the do-not-care pixels. If Wt(x,y) is matched, ft(x,y)=0; otherwise, ft(x,y)=fb(x,y). The algorithm repeats this procedure until no more thinning can be performed. The thinning algorithm is to cut off the redundant-contour pixels, so that the contours have a thickness of only one pixel. Figure 6(b) shows the result after running the thinning operation on the image in Figure 6(a).Figure 7
Eight structuring elements for thinning.However, the uneven object contours tend to cause small spurs on the obtained skeleton by the thinning operation. These spurs are not expected. Therefore, a spur pruning operation is required to remove them. The procedure of the spur pruning operation [14] is entirely the same as the thinning operation except for the eight structuring elements in Figure 7 which are replaced by the eight structuring elements in Figure 8. Let fs be the result obtained by the spur pruning operation from ft. Figure 6(c) displays the result image fs after running the spur pruning operation on the image ft shown in Figure 6(b).Figure 8
Eight structuring elements for Spurs.Infs, some false contours should be removed. Let Pi and Pj be two pixels on the contour of the kth object Ok in fs, and let dij be the distance between Pi and Pj. Assume dhk is the maximal one among all the dij’s on the contour of Ok. We call dhk the major length of Ok. If the major length of Ok is less than the one-third of the average major length of all the object contours in fs, then Ok is considered to be noise and removed from fs. After that, fs in Figure 6(c) is changed into fc in Figure 6(d).


## 3.2.3. Overlapping Cells Splitting

Some juxtaposed objects may share overlapping regions, which will make cell distinguishing and counting imprecise. We call the area containing some touching or overlapping cells an overlapping area. The watershed algorithm is commonly used to split the overlapping object [15–18]. Watershed based segmentation can give good results for gray level images with different minima and catchment basins. For a binary image, however, there are only two gray levels 0 and 1, respectively, standing for background and cell. If two cells are connected together in the binary image fc, only one minimum and catchment basin will be formed in the topographic surface. In employing the watershed algorithm to segment connected cells, distance transform [19] is often adopted to preprocess the image to make it more suitable for segmentation by watershed algorithm, for assigning the catchment basins in the overlapping area.Distance transform [19, 20] labels each pixel in an overlapping area with the distance of the pixel to the nearest pixel outside the overlapping area. Let R be an overlapping area in fc. For each pixel P(x,y) in R located at the coordinates (x,y) on fc, the distance transform can be determined by the following three steps.(1)
Attach a labeld(x,y)=∞ to each pixel P(x,y) in the overlapping area on fc and give a label d(x,y)=0 to each pixel outside the overlapping area.(2)
Then travel throughfc pixel by pixel. Replace the label of each pixel P(x,y) in fc with Min⁡{d(x,y), d(x-1,y-1)+1, d(x-1,y)+1, d(x-1,y+1)+1, d(x,y-1)+1, d(x,y)+1, d(x,y+1)+1, d(x+1,y-1)+1, d(x+1,y)+1, d(x+1,y+1)+1}.(3)
Repeat step (2), until all labels have been converted to finite values.The most intuitive way to explain the watershed based segmentation is to imagine that a hole is drilled in each minimum of the surface and we flood water into different catchment basins from the holes. If the water of different catchment basins is likely to merge due to further immersion, a dam is built to prevent the merging. This flooding process will eventually reach a stage when only the top of dam (the watershed lines) is visible above the water line.LetPiPj¯ be the line segment passing through Pi and Pj which are two pixels on the contour of the kth object Ok in fc, let dij be the distance between Pi and Pj, let DM=dlm be the major length of Ok, where dlm is the longest distance between any two pixels on the contour of Ok, and let (x1,y1) as well as (x2,y2) be the coordinates of Pl and Pm on fc. Assume Dm=dl′m′ is the distance of pixels Pl′ and Pm′ on the contour of Ok, Pl′Pm′¯ passes through ((x1+x2)/2,(y1+y2)/2), and Pl′Pm′¯ is perpendicular to PlPm¯. We call D⊥ the minor length of Ok.LetAk be the area of Ok and let Aave be the average area of all the objects on fc. The AIICS system considers the object Ok to be an overlapping area and takes the watershed algorithm [17] to split Ok into more objects, only if
(15)DMD⊥>1.5,Ak>2×Aave.After that,fc is converted into other binary image fw. Figure 9(b) demonstrates the fw obtained by running this watershed algorithm on the fc of the image in Figure 9(a), where only the overlapping areas surrounded by white closed curves in Figure 9(b) are further split by the watershed algorithm.The results of overlapping cells splitting.
(a)
Original image
(b)
f
w
(c)
f
mThere may be still some cells split into more than one region. The AIICS system tries to combine them. LetμA and σA be the average and standard deviation of the areas of all the object regions on Iw. If the area of object region Ok is less than μA-0.5×σA, the AIICS system merges Ok into one object region which is adjacent to Ok and has the longest common edge with Ok. After region combination, fw is converted into fm. Figure 9(c) displays the fm obtained from the fw in Figure 9(b) by the region combination.


## 4. Rough Classifier

There are six distinct autoantibody fluorescence pattern cells which have quite different fluorescence characteristics. To effectively sever the cells from the IIF images, the authors roughly classify IIF images into two categories, uniform pattern category and fleck pattern category. Most of the cells on an image in uniform pattern category are with coarse speckled pattern, diffused pattern, fine speckled pattern, or peripheral pattern, while most cells on an image in fleck pattern category are with discrete speckled pattern or nucleolar pattern. The nuclei of the cells on the images in fleck pattern category have bright spots. Since both image categories have significantly different properties, the AIICS system will take a rough classifier to classify IIF images and use distinct parameter values to segment the cells from the images.Given an IIF image, the rough classifier first applies Otsu’s thresholding method [11] to determine a threshold Tb for converting the IIF image into a binary image. A pixel is considered to an object pixel if its gray-level intensity is greater than Tb; else it is considered to be one background pixel. If the average area of all the cell regions is less than a given threshold Tarea, then the image is classified to fleck pattern category otherwise to uniform pattern category. Then, the AIICS system will take different mr and ma to segment cells on the images from different categories. In this paper, we will use a genetic algorithm to compute the most suitable Tarea, mr, and ma.


## 5. Genetic Algorithm Based Parameter Detector (GAPD)

According to the experimental results, the AIICS system can give better results when given differentmr and ma for segmenting cells from fleck pattern category images and from uniform pattern category images. Assume that mr and ma are given to be mr=mr′ as well as ma=ma′ for segmenting cells from fleck pattern category images and mr=mr′′ as well as ma=ma′′ for segmenting cells from uniform pattern category images.The performance of the AIICS system is significantly affected by the values ofTarea, mr′, ma′, mr′′, ma′′, and mb in segmenting the cells on an IIF image. In this paper, a genetic algorithm based parameter detector (GAPD) is provided to determine the most suitable values of Tarea, mr′, ma′, mr′′, ma′′, and mb. GAPD makes use of a binary string, consisting of six substrings sarea, sr′, sa′, sr′′, sa′′, and sb which, respectively, comprise narea, nr′, na′, nr′′, na′′, and nb binary bits, to represent a chromosome Ch. sarea, sr′, sa′, sr′′, sa′′, and sb can be used to describe Tarea, mr′, ma′, mr′′, ma′′, and mb in segmenting cells from IIF images. For each chromosome Ch, Tarea, mr′, ma′, mr′′, ma′′, and mb can be encoded as 100+5×narea′, 2nr′′+3, 2na′′+3, 2nr′′′+3, 2na′′′+3, and 2nb′+3, where narea′, nr′′, na′′, nr′′′, na′′′, and nb′ are the numbers of 1-bits in sarea, sr′, sa′, sr′′, sa′′, and sb, individually.GAPD applies the accumulated historical IIF images to train the most appropriate values ofTarea, mr′, ma′, mr′′, ma′′, and mb by using a genetic algorithm. The manually drawn cell regions are considered as a collection of ground truths. The relative distance error (RDE) [21] is often adopted to measure the segmentation errors of a segmentation method. GAPD will use RDE as the measure of fitness of Ch based on the values of Tarea, mr′, ma′, mr′′, ma′′, and mb encoded by Ch.Initially, GAPD randomly generatesN chromosomes, each with narea+nr′+na′+nr′′+na′′+nb binary bits. To evolve the best solution, the genetic algorithm repeatedly executes mutation, crossover, and selection operations until the relative fitness values of the reserved chromosomes are very similar to one another or the number of iterations is equal to MAXRUN.In mutation operation, for each of theN reserved chromosomes, GAPD uses a random number generator to specify one bit b from each of the substrings sarea, sr′, sa′, sr′′, sa′′, and sb. Then, all the b’s are replaced by ¬b’s to generate a new chromosome, where ¬ signifies the operator “NOT.”In crossover operation, GAPD uses a random number generator to designateN′ pairs of chromosomes from the N reserved chromosomes. Let Ch[i⋯j] be the substring of the ith to jth bits in Ch. For each chromosome pair (Ch1,Ch2), GAPD concatenates
(16)Ch1[1⋯⌊narea2⌋],Ch2[⌊narea2⌋+1⋯narea],Ch1[narea+1⋯(narea+⌊nr′2⌋)],Ch2[(narea+⌊nr′2⌋+1)⋯narea+nr′],Ch1[narea+nr′+1⋯(narea+nr′+⌊na′2⌋)],Ch2[(narea+nr′+⌊na′2⌋+1)⋯narea+nr′+na′],Ch1[narea+nr′+na′+1⋯(narea+nr′+na′+⌊nr′′2⌋)],Ch2[(narea+nr′+na′+⌊nr′′2⌋+1)⋯narea⌊nr′′2⌋+nr′+na′+nr′′],Ch1[⌊na′′2⌋narea+nr′+na′+nr′′+1⋯(narea+nr′+na′+nr′′+⌊na′′2⌋)],Ch2[(narea+nr′+na′+nr′′+⌊na′′2⌋+1)⋯narea⌊na′′2⌋+nr′+na′+nr′′+na′′],Ch1[narea+nr′+na′+nr′′+na′′⌊nb2⌋+1⋯(narea+nr′+na′+nr′′+na′′+⌊nb2⌋)],Ch2[(narea+nr′+na′+nr′′+na′′+⌊nb2⌋+1)⋯narea⌊nb2⌋+nr′+na′+nr′′+na′′+nb],
into a new chromosome, and
(17)Ch2[1⋯⌊narea2⌋],Ch1[⌊narea2⌋+1⋯narea],Ch1[(narea+⌊nr′2⌋+1)⋯narea+nr′],Ch2[narea+nr′+1⋯(narea+nr′+⌊na′2⌋)],Ch1[(narea+nr′+⌊na′2⌋+1)⋯narea+nr′+na′],Ch2[narea+nr′+na′+1⋯(narea+nr′+na′+⌊nr′′2⌋)],Ch1[(narea+nr′+na′+⌊nr′′2⌋+1)⋯narea⌊nr′′2⌋+nr′+na′+nr′′],Ch2[narea+nr′+na′+nr′′⌊na′′2⌋+1⋯(narea+nr′+na′+nr′′+⌊na′′2⌋)],Ch1[(narea+nr′+na′+nr′′+⌊na′′2⌋+1)⋯narea⌊na′′2⌋+nr′+na′+nr′′+na′′],Ch2[⌊nb2⌋narea+nr′+na′+nr′′+na′′+1⋯(narea+nr′+na′+nr′′+na′′+⌊nb2⌋)],Ch1[(narea+nr′+na′+nr′′+na′′+⌊nb2⌋+1)⋯narea⌊nb2⌋+nr′+na′+nr′′+na′′+nb],
into another new chromosome.In selection operation,N optimal chromosomes are selected from the N chromosomes reserved in the previous iteration and N as well as 2×N′ chromosomes created in the mutation and crossover operations, respectively, according to their fitness. GAPD continuously performs the three operations—mutation, crossover, and selection, until the related fitness of the reserved N chromosomes are very close. Finally, GAPD employs the chromosome Ch with the best fitness in the N reserved chromosomes to describe Tarea, mr′, ma′, mr′′, ma′′, and mb in segmenting cells from IIF images. Figure 10(a) is a chromosome Ch indicating Tarea=110, mr′=9, ma′=mr′′=ma′′=7, and mb=5, where narea=nr′=na′=nr′′=na′′=nb=4. Figure 10(b) displays a new chromosome Ch′ derived from Ch by the mutation operation, where the underlined bits are the randomly selected bits b. Figure 10(c) displays two new chromosomes, Ch1′ and Ch2′, generated from chromosomes Ch1 and Ch2 by the crossover operation.Example for GAPD.
(a)
A chromosome Ch
(b)
A new chromosome created from Ch by mutation operator
(c)
ChromosomesCh1′ and Ch2′ derived from Ch1 and Ch2 by crossover operator


## 6.  Experiments

This section is to explore the performance of the AIICS system in segmenting and counting the cells on an IIF image by experiments. In these experiments, 195 IIF images, 160 of which are in uniform pattern category and 35 of which are in fleck pattern category, are used as test images. Figure1 shows a part of the test images. One can obviously observe that there are quite different characteristics with these images. The AIICS system hence employs rough classifier to classify an IIF image into fleck category or uniform pattern category and gives different values of mr and ma in segmenting the cells on the IIF image.The first experiment is to compute the most suitable parametersTarea, mr′, ma′, mr′′, ma′′, and mb. In this experiment, 17 of the 160 uniform pattern images and 3 of the 35 fleck pattern images are randomly selected to train Tarea, mr′, ma′, mr′′, ma′′, and mb via GAPD, where N=10, N′=10, nr′=na′=nr′′=na′′=nb=10, and MAXRUN=200. The experimental results show that the AIICS system will give better segmentation results when Tarea=210, mr′=3, ma′=11, mr′′=17, ma′′=3, and mb=3 in segmenting cells from IIF image. In the following experiments, Tarea, mr′, ma′, mr′′, ma′′, and mb are set to the values obtained in this experiment.The purpose of the second experiment is to probe the performance of the gradient computing method proposed in this paper. In this experiment, the AIICS system is applied to segment cells out from 20 IIF images, except that the gradient computing method is replaced by Sobel [22], Prewitt [22], Roberts [22], and Laplacian [22]. The 20 IIF images, including 1242 cells, are randomly selected from the 195 test images. Misclassification error (ME) [23], relative area error (RAE) [23], modified Hausdorff distance (MHD) [23], and relative distance error (RDE) [21] are often adopted to measure the segmentation errors. In this experiment, they are used to evaluate the performances of Sobel, Prewitt, Roberts, and Laplacian and the gradient computing method proposed in this paper. Table 2 shows the results of this experiment. The experimental results indicate that our method is much better than Sobel, Prewitt, Roberts, and Laplacian methods in computing the gradient of an image.Table 2
The segmentation errors obtained in the second experiment.
Sobel
Prewitt
Roberts
Laplacian
AIICS
ME
0.12
0.13
0.11
0.10
0.05
RAE
0.64
0.70
0.51
0.48
0.11
MHD
8960.99
6355.46
4989.59
2103.70
89.00
RDE
7384.81
5493.53
4333.16
2327.26
241.23The third experiment is to explore the performances of the AIICS system, Eddins’, Althoff et al.’s, Tang et al.’s, and Yan et al.’s methods in segmenting cells on an IIF image. In this experiment, the 20 test images employed in the second experiment are also used as test images; ME, RAE, MHD, and RDE are employed to evaluate the segmentation errors. Table 3 demonstrates the results obtained in this experiment.Table 3
The segmentation errors obtained in the third experiment.
Eddins
Althoff
Tang
Yan
AIICS
ME
0.34
0.44
0.73
0.54
0.05
RAE
0.53
0.62
0.81
0.71
0.11
MHD
1322.71
1331.15
1414.72
1311.13
89.00
RDE
1523.62
1615.57
1598.56
1515.31
241.23The fourth experiment is to scrutinize the performance of the AIICS system and to compare this performance with those of Eddins’ [5], Althoff et al.’s [1], Tang and Ewert [6], and Yan et al.’s [7] methods in counting cells on an IIF image after segmenting the cells out. Tables 4, 5, and 6 show the experimental results, respectively, obtained by using all 195 IIF images, only 160 uniform pattern images, and only 35 fleck pattern images as the test data. The experimental results indicate that the AIICS system is much superior to Eddins’ [5], Althoff et al.’s [1], Tang and Ewert [6], and Yan et al.’s [7] methods in counting cells on an IIF image.Table 4
The segmentation errors obtained in the third experiment.
Eddins
Althoff
Tang
Yan
AIICS
PPV
0.47
0.69
0.86
0.85
0.98
SEN
0.70
0.37
0.72
0.76
0.90
DACC
0.39
0.32
0.64
0.67
0.89Table 5
The results of experiment 4 by using 160 uniform pattern images as test data.
Eddins
Althoff
Tang
Yan
AIICS
PPV
0.45
0.73
0.89
0.89
0.99
SEN
0.79
0.43
0.73
0.80
0.92
DACC
0.41
0.37
0.67
0.73
0.91Table 6
The results of experiment 4 by using 35 fleck pattern images as test data.
Eddins
Althoff
Tang
Yan
AIICS
PPV
0.81
0.39
0.74
0.67
0.94
SEN
0.32
0.11
0.66
0.58
0.82
DACC
0.30
0.09
0.54
0.45
0.78The fifth experiment is to investigate the effect of rough classifier. In this experiment, 160 uniform pattern images and 35 fleck pattern images are used as test data. Table7 shows the experimental results in classifying IIF images to the uniform pattern category or to the fleck pattern category by rough classifier. The classification measures obtained in this experiment are PPV=0.988, SEN=0.943, and F=0.965.Table 7
The classification results obtained by the rough classifier.
Ground Truth
Uniform
Uniform
Test outcome
Uniform
160
4
Fleck
0
31Table7 displays that four images in fleck pattern category are erroneously classified to uniform pattern category. After that, in this experiment, the AIICS system is used to separate the cells from the four images with (mr=17, ma=3) and (mr=3, ma=11), respectively. Table 8 demonstrates the experimental results which tell that it is more appropriate to categorize the four images to the uniform pattern category in cell segmentation by the AIICS system.Table 8
The segmentation results of the erroneously classified images obtained by the AIICSC system with(mr=17,ma=3) and (mr=3,ma=11).
(
m
r
=
17
,
m
a
=
3)
(
m
r
=
3
,
m
a
=
11
)
PPV
0.98
1.00
SEN
0.94
0.87
DACC
0.92
0.87


## 7. Conclusions

In this paper, the AIICS system is proposed to automatically segment and count cells on an IIF image. The experimental results show that the AIICS system can effectively segment and count the cells on an IIF image, even though the characteristics of the IIF images are extremely different. In this paper, color selector, run length enhancer, and adaptive filter are provided to remove noise and to enhance the contours of cells on an image. To cut off the cells from an image effectively, a new gradient computing method is proposed to compute the gradient of edge and an adaptive threshold method is given to decide the most proper thresholds for detecting the cells on an image. In addition, rough classifier is presented to classify IIF images. GAPD is presented to decide the most suitable parameters used in the AIICS system. These techniques can be also applied in segmenting the objects from other kind of images.



---

*Source: 501206-2014-05-22.xml*