# Visual Sensor Based Image Segmentation by Fuzzy Classification and Subregion Merge

Authors: Huidong He; Xiaoqian Mao; Wei Li; Linwei Niu; Genshe Chen
Journal: Journal of Electrical and Computer Engineering
 (2017)

---

## Abstract

The extraction and tracking of targets in an image shot by visual sensors have been studied extensively. The technology of image segmentation plays an important role in such tracking systems. This paper presents a new approach to color image segmentation based on fuzzy color extractor (FCE). Different from many existing methods, the proposed approach provides a new classification of pixels in a source color image which usually classifies an individual pixel into several subimages by fuzzy sets. This approach shows two unique features: the spatial proximity and color similarity, and it mainly consists of two algorithms: CreateSubImage and MergeSubImage. We apply the FCE to segment colors of the test images from the database at UC Berkeley in the RGB, HSV, and YUV, the three different color spaces. The comparative studies show that the FCE applied in the RGB space is superior to the HSV and YUV spaces. Finally, we compare the segmentation effect with Canny edge detection and Log edge detection algorithms. The results show that the FCE-based approach performs best in the color image segmentation.


---

## Body

## 1. Introduction

Image segmentation is a fundamental issue in image processing and computer vision as it is a preliminary process in many applications that need to segment an image into meaningful regions. Applications of segmenting gray-level images are very limited so studies on color image segmentation become interesting because color images provide much more information than gray-level ones. A number of image segmentation techniques have been proposed in past decades [1–3]. A physical model in [4] is proposed for color image understanding. A widely used technique for color image segmentation is based on the idea of region growing. A color segmentation algorithm combining region growing and region merging processes is discussed in [5]. The fuzzy growing and expanding approach presented in [6] uses histogram tables for fine segmentation. A method for segmentation of color texture region using the defined class maps is presented in [7]. An approach to contrast analysis using fuzzy region growing is documented in [8]. Color image segmentation using seeded region growing is presented in [9–11]. Most pixel clustering algorithms work in the RGB color space in which the pixels are represented by red, green, and blue components. Clustering methods [12, 13], for example, c-means fuzzy classification [14], are often considered as an unsupervised classification of pixels as a prior knowledge of the image. Both supervised and unsupervised segmentation of textured color images are presented in [15, 16]. The level setting approaches to process a color image for segmentation are discussed in [17, 18]. All of the proposed techniques above provide a crisp segmentation of images, where each pixel is classified into a unique subset. This classification may not reflect an understanding of images by humans very well, as the work [1] stated “the image segmentation problem is basically one of psychophysical perception, and therefore not susceptible to a purely analytical solution.” Probably, one of the difficulties in color image segmentation is how to define color conception, since an understanding of colors in an image varies by different observers.Considering uncertainty and ambiguity in many real applications, this paper presents an approach to color image segmentation based on a fuzzy color extractor (FCE). Its preliminary version appears in [19]. The FCE is directly extended from the fuzzy gray-level extractor, which was applied to recognize landmarks on roads for autonomous vehicle navigation in [20–22]; that is, the fuzzy rules for extracting a road from a gray-level image are as follows:If the gray level of a pixel is close to the gray-level of the road,Then one can make it black, orElse one can make it to white.If these are modified to the rules for extracting an object from a color image, thenif the color components of a pixel closely matches the color components of an object,then one can extract this pixel as aMatched pixel of the object, orelse one can extract this pixel as anUnmatched pixel.The FCE viewed as a mapping operator of measuring colors is able to extract colors that are similar to a color pattern (seed). If a pixel is extracted it is defined as aMatched pixel; otherwise it is defined as anUnmatched pixel. In doing this, we develop procedures IniSeedList and UpdateSeedList for automatic selection of seeds. Our approach mainly consists of two algorithms: CreateSubImage and MergeSubImage. The CreateSubImage algorithm always extracts the seed as the firstMatched pixel into a subimage and continually uses the FCE to check neighbors of eachMatched pixel to grow the subimage. The subimage is created when neighbors’ colors of allMatched pixels are “not close to” the seed’s color. The CreateSubImage algorithm iteratively invokes UpdateSeedList to select new seeds to create next subimages until all the pixels in the image are processed. Different from all of the color segmentation techniques discussed above, the CreateSubImage algorithm most likely classifies an individual pixel into several subimages by fuzzy sets. As a result, two created subimages may share a common area, which can be considered as a color similarity metric for merging the subimages. The MergeSubImage algorithm first removes insignificant subimages from the set of subimages and merges significant subimages in order to extract meaningful segmented images. The insignificant subimages usually have a “too small” or a “too large” ratio of their area sizes over the source image. A “too small” insignificant subimage is mainly caused by a tiny fragment or a noise pixel in the source image, while a “too large” insignificant image is created by a “bad” seed. Then, the algorithm merges two subimages together according to their color similarity described by the size of their common pixels. We tested the proposed approach using exemplar color images from the database at UC Berkeley [23] in three different color spaces and discussed its effectiveness and robustness by comparing segmentation effect with two edge detection algorithms, namely, Canny edge detection and Log edge detection.


## 2. Three Color Spaces

In many color-based computer vision applications, it is very important to segment colors in an image, but it is very difficult due to the fact that colors are ambiguous in uncertain environments. In this study, we apply the FCE to segment colors in the image represented in the RGB, HSV, and YUV spaces.


### 2.1. RGB Color Space

RGB is the most traditional color space and is widely used in image segmentation because of its simplicity and fast processing [24, 25]. Colors consist of red (R), green (G), and blue (B) components, as shown in Figure 1(a).Figure 1
Color distribution in the RGB and HSV color spaces.
(a)
RGB color space
(b)
HSV color space


### 2.2. HSV Color Space

HSV is another color space widely used for image processing, where a color is decomposed into hue (H), saturation (S), and value (V). The distribution of color in the HSV color space is shown in Figure1(b). The color components in the RGB space can be transformed into the HSV space by (1)max=maxR,G,B,min=minR,G,B,V=maxS=max-minmax×255,H=G-Bmax-min×60R=max120+B-Rmax-min×60G=max240+R-Gmax-min×60B=maxH=H+360,ifH<0,H=H×255÷360.Usually, the RGB values are between 0 and 255, so the HSV values are also between 0 and 255 through the transformation by (1).


### 2.3. YUV Color Space

YUV is the color space which is used in an analog color TV system, which translates color image components obtained in the RGB space into the luminance Y and the colors B -Y (U) and R - Y (V) by(2)Y=0.299R+0.587G+0.114B,U=-0.1687R-0.3313G+0.500B+128,V=0.500R-0.4187G-0.0813B+128.The YUV values are between 0 and 255 through the transformation by (2). The advantage of using the YUV color space for color segmentation is that the luminance Y is separated from the colors U and V.


## 2.1. RGB Color Space

RGB is the most traditional color space and is widely used in image segmentation because of its simplicity and fast processing [24, 25]. Colors consist of red (R), green (G), and blue (B) components, as shown in Figure 1(a).Figure 1
Color distribution in the RGB and HSV color spaces.
(a)
RGB color space
(b)
HSV color space


## 2.2. HSV Color Space

HSV is another color space widely used for image processing, where a color is decomposed into hue (H), saturation (S), and value (V). The distribution of color in the HSV color space is shown in Figure1(b). The color components in the RGB space can be transformed into the HSV space by (1)max=maxR,G,B,min=minR,G,B,V=maxS=max-minmax×255,H=G-Bmax-min×60R=max120+B-Rmax-min×60G=max240+R-Gmax-min×60B=maxH=H+360,ifH<0,H=H×255÷360.Usually, the RGB values are between 0 and 255, so the HSV values are also between 0 and 255 through the transformation by (1).


## 2.3. YUV Color Space

YUV is the color space which is used in an analog color TV system, which translates color image components obtained in the RGB space into the luminance Y and the colors B -Y (U) and R - Y (V) by(2)Y=0.299R+0.587G+0.114B,U=-0.1687R-0.3313G+0.500B+128,V=0.500R-0.4187G-0.0813B+128.The YUV values are between 0 and 255 through the transformation by (2). The advantage of using the YUV color space for color segmentation is that the luminance Y is separated from the colors U and V.


## 3. Fuzzy Color Extractor

The fuzzy color extractor proposed here enables us to deal with such uncertainty. We will only discuss how to use the FCE-based algorithm defined in the RGB color space to segment color images, because the algorithm can directly process the color images in the HSV and YUV spaces, which are converted from the RGB space. Figure1(a) shows the RGB space, where colors are represented by their red, green, and blue components in an orthogonal Cartesian space. The upper-left corner in the image is the origin. The color of each pixel p(n,m) denoted by pn,mRGB is processed to separate its red, green, and blue components (pn,mR, pn,mG, pn,mB). The FCE extracts a cluster of colors based on a defined color pattern (seed) denoted by seedRGB, seedRGB and is either directly defined by its RGB components (seedR, seedG, seedB) or determined by a pixel in the image. The color components’ differences between a pixel pn,mRGB and seedRGB are calculated as follows: (3)difn,mR=pn,mR-seedR,difn,mG=pn,mG-seedG,difn,mB=pn,mB-seedB,0≤m<M,0≤n<N.M and N represent the size of an array which holds an image. The corresponding fuzzy rules can be obtained as follows:Ifdifn,mR and dif(n,mG and difn,mB are Zero,Thenp(n,m) is Matched.Ifdifn,mR or difn,mG or difn,mB is Negative or Positive,Thenp(n,m) is Unmatched.Both the rules indicate thatp(n,m) belongs to the object to be extracted, if the Euclidean distances between pn,mRGB and seedRGB along the three axes in RGB coordinate system are small enough; otherwise, p(n,m) does not belong to the object.Figure2(a) shows the membership functions (μN(x), μZ(x), μP(x)) for the input fuzzy variables(Negative,Zero,Positive) defined by (4)μNx=1-255≤x<-α2x+α1α1-α2-α2≤x<-α10-α1≤x≤255,μZx=0-255≤x<-α2x+α2α2-α1-α2≤x<-α11-α1≤x≤α1x-α1α2-α1α1≤x<α20α2≤x≤255,μPx=0-255≤x<-α1x+α2α2-α1α1≤x<α21α2≤x≤255,and Figure 2(b) shows the membership functions (μM(x), μU(x)) for the output fuzzy variables(Matched,Unmatched) defined by (5)μMx=ρM-xρM0≤x<ρM0ρM≤x≤255,μUx=00≤x<ρUx-ρU255-ρUρU≤x≤255,where ρM+ρU=255. Based on difn,mR, difn,mG, and difn,mB, the fuzzy rules produce the weight ωm forMatched and the weight ωu forUnmatched by (6)ωm=minμZR,μZG,μZB,ωu=maxμNR,μNG,μNB,μPR,μPG,μPB.Figure 2(b) shows the produced areas in the output domain, while ωm and ωu cut μM(x) and μU(x). A crisp output value, ΔρF, is calculated by the centroid defuzzification method:(7)ΔρF=∫μoutxxdx∫μoutxdx,where μoutx represents the envelope function of the areas cut by ωm and ωu in fuzzy output domain. If ΔρF<σ,  p(n,m) is extracted, p(n,m) is not extracted, where σ is a threshold. The FCE can be understood as a mapping operator between Euclidean distances {difn,mR, difn,mG, difn,mB} in the RGB space and a difference ΔρF in the intensity space under a fuzzy metric.Figure 2
Definition of membership functions for the FCE.
(a)
Membership functions for color differences
(b)
Membership functions for defuzzificationThe example in Figure3 shows how to create some subimages with interesting colors from the source image IFLOWER by the FCE. First, we define three seeds: seedRED = (255, 0, 0), seedGREEN = (0, 255, 0), and seedBLUE = (0, 0, 255) to create three subimages {IRED, IGREEN, IBLUE} that hold the words “red,” “green,” and “blue” from the source image IFLOWER, as shown in Figure 3(b). Also, we select a seed p0,0RGB = (6, 6, 8) at the origin to remove the background from the source image IFLOWER as shown in Figure 3(c). Figure 3(d) shows IPETAL=IFLOWER-(IRED∪IGREEN∪IBLUE∪IBACKGROUND) that holds the color components of the flower petal.Figure 3
Extraction of colors inIFLOWER image by FCE.
(a)
I
F
L
O
W
E
R
(b)
{
I
R
E
D
,
I
G
R
E
E
N
,
I
B
L
U
E
}
(c)
I
B
A
C
K
G
R
O
U
N
D
(d)
I
P
E
T
A
L


## 4. Creating Subimages Using FCE

This section discusses the CreateSubImage algorithm for creating a set of subimages from a color image using the FCE in the RGB space. A critical step in this algorithm is to select seeds for the FCE. For this purpose, we evenly divide the RGB space into the eight subspacesν_black, ν_red, ν_green, ν_blue, ν_yellow, ν_pink, ν_cyan, ν_white, which are defined by the well-defined colors: black, red, green, blue, yellow, pink, cyan, white, at their corresponding vertices of the RGB space, as shown in Figure 1(a). Table 1 lists their definitions; for example, ν_black is defined by a cubic [0,127]×[0,127]×[0,127] on the red-axis, green-axis, and blue-axis.Table 1
Definitions of the eight subspaces shown in Figure1(a).
Vertexi
Subspace
Red-axis
Green-axis
Blue-axis
v
1
R
G
B (0, 0, 0)
ν_black
[
0
,
127
]
[
0
,
127
]
[
0
,
127
]
v
2
R
G
B (0, 0, 255)
ν_blue
[
0
,
127
]
[
0
,
127
]
[
128
,
255
]
v
3
R
G
B (0, 255, 0)
ν_green
[
0
,
127
]
[
128
,
255
]
[
0
,
127
]
v
4
R
G
B (0, 255, 255)
ν_cyan
[
0
,
127
]
[
128
,
255
]
[
128
,
255
]
v
5
R
G
B (255, 0, 0)
ν_red
[
128
,
255
]
[
0
,
127
]
[
0
,
127
]
v
6
R
G
B (255, 0, 255)
ν_pink
[
128
,
255
]
[
0
,
127
]
[
128
,
255
]
v
7
R
G
B (255, 255, 0)
ν_yellow
[
128
,
255
]
[
128
,
255
]
[
0
,
127
]
v
8
R
G
B (255, 255, 255)
ν_white
[
128
,
255
]
[
128
,
255
]
[
128
,
255
]We calculate the distances of each pixel’s RGB components to the eight well-defined colors by(8)din,m=∑pn,mRGB-viRGB2i=1,2,…,8,where n and m are the index of a pixel in the image, pn,mRGB is the color components of the pixel located at (n,m), viRGB is the color components of vertex i in the RGB space, and di(n,m) is the distance between the pixel’s color and vertex i.We developed two procedures, IniSeedList and UpdateSeedList, to initialize a seed list and to update it. For an input image, IniSeedList classifies each pixel into one of the eight subspaces according to its minimum distance to vertexi that is a well-defined color in the RGB space. In each subspace, IniSeedList selects such a pixel as a seed candidate, seed_candi, that has the shortest distance to its vertex i. Then, IniSeedList inserts seed_candi into the seed list in order of the number of pixels classified in the subspaces and returns it. UpdateSeedList checks if seed_candi is popped out or is extracted by the FCE. If so, UpdateSeedList updates seed_candi in its corresponding subspace. The pseudocode of the IniSeedList procedures is shown in Pseudocode 1 and the pseudocode of the UpdateSeedList is shown in Pseudocode 2.Pseudocode 1
IniSeedList (IIMG)
for i← 1 to 8 do
pixel_numi← 0;
for n← 1 to N do
for m← 1 to M do
Calculate di(n,m) in (8);
d
k
(
n
,
m
)
← min{di(n,m)};
pixel_numk← pixel_numk+1;
Find seed_cand(i) in each subspace;
// with the shortest distance to its vertex i.
Push seed_cand(i) into SeedList;
Sort seed_cand(i);
// according to pixel number classified in the subspaces.
return SeedList;Pseudocode 2
UpdateSeedList (IIMG, SeedList)
subspace_index (i) ← 0;
for i← 1 to 8 do
if (seed_cand(i)∈IIMG)
subspace_index (i) ← 1;
else
Remove seed_cand(i) from SeedList;
for i← 1 to 8 do
if subspace_index (i) = 0
Find seed_cand(i) in subspace i;
// with the shortest distance to its vertex i.
Push seed_cand(i) into SeedList;
Sort seed_cand(i);
// according to pixel number classified in the subspaces.
return SeedList;Table2 shows an example of a seed list generated by IniSeedList from the color image, IBIRD. Columns 1 and 2 in Table 2 indicate the defined subspaces and the number of the classified pixels in the subspaces, column 3 shows the order of seed_candi in the list, and columns 4 and 5 list the RGB color components and locations of seed_candi in IIMG, respectively. The top seed in Table 2 is seed_cand1 = (244, 251, 243) with location (269, 271) from the subspace ν_white holding 100,369 pixels. After the algorithm pops out this seed for creating IM(i), UpdateSeedList produces a new seed candidate in the subspace ν_white from IU(1) and inserts it on the top of the list. The new seed_cand1 = (240, 251, 247) is located at (283, 267) in IU(1) and is also from the subspace ν_white with 105,045 pixels. Please note that the input image to IniSeedList is IIMG=IBIRD-ICENTER, instead of IBIRD itself. ICENTER is an image extracted by seedCENTER=(127,127,127) that is the center of the RGB space. It is difficult to determine in which subspace the center color is classified, so any seed close to the center may not be able to differentiate colors very well, as shown in Figure 4(a). In order to guarantee to get a “good” seed, our approach does not select any seed falling in ICENTER. It is also possible that there is no pixel in a subspace; that is, the seed candidates are less than eight. For example, the seed candidates for IBIRD are not available in the subspaces ν_blue, ν_yellow, ν_green, and ν_pink.Table 2
The seed list generated from the color image,IBIRD.
Subspace
Number of pixels
seed_cand(i) order
seed_cand(i) colors
seed_cand(i) locations (n,m)
ν_white
100369
1
(244, 251, 243)
(269, 271)
ν_black
19393
2
(0, 6, 0)
(193, 233)
ν_cyan
70
3
(151, 100, 57)
(216, 190)
ν_red
26
4
(206, 166, 107)
(225, 204)
ν_blue
0
N/A
N/A
N/A
ν_yellow
0
N/A
N/A
N/A
ν_green
0
N/A
N/A
N/A
ν_pink
0
N/A
N/A
N/AFigure 4
(a) The imageICENTER extracted by the seed, seedCENTER = (127, 127, 127). (b) An insignificant image, ITEMP, extracted by a “bad” seed. (c) The image IUNCLASSIED.
(a)
I
C
E
N
T
E
R
(b)
I
T
E
M
P
(c)
I
U
N
C
L
A
S
S
I
E
DThe pseudocode of the CreateSubImage procedures is listed as shown in Pseudocode3.Pseudocode 3
CreateSubImage (ISRC, center)
i
← 0; IM(i+1)← ø; IU(i)←ISRC;
If (center = true)
I
C
E
N
T
E
R
← FuzExtractor (ISRC, seedCENTER);
else ICENTER← ø;
I
I
M
G
←
I
U
(
i
)
-
I
C
E
N
T
E
R;
SeedList ← IniSeedList (IIMG);
while (SeedList ≠ ø) do
s
e
e
d
R
G
B
← The first element in SeedList;
I
T
E
M
P
← FuzExtractor (ISRC, seedRGB);
if ITEMP/ISRC>ɛ
if (ITEMP/ISRC<γ1 &var (ITEMP) <γ2)
I
M
(
i
+
1
)
←
I
T
E
M
P;
I
U
(
i
+
1
)
←
I
S
R
C – (IM(i+1)∪IM(i-1)⋯IM(1));
I
I
M
G
←
I
U
(
i
+
1
)
-
I
C
E
N
T
E
R;
SeedList ← IniSeedList (IIMG);
i
←
i
+
1;
end
returnIMi+1;CreateSubImage creates a set of subimages from a color source image. CreateSubImage invokes mainly three procedures: IniSeedList, UpdateSeedList, and FuzExtractor. The pseudocode of the FuzExtractor is shown in Pseudocode4.Pseudocode 4
FuzExtractor (ISRC, seedRGB)
Move seedRGB into IMATCH;
Push four neighbor pixels of seedRGB into queue;
while (queue≠ø) do
p
i
x
e
l
F
I
R
S
T
← the top element in queue;
Calculate ΔρF using (3);
if (ΔρF<σ)
Move pixelFIRST into IMATCH;
Push unduplicated neighbors of pixelFIRST into queue;
end
returnIMATCH;CreateSubImage has two arguments:ISRC and center, where ISRC is a source image and center is a Boolean variable. If center = true, ICENTER is created by seedCENTER, otherwise ICENTER is assigned by the empty image ø. FuzExtractor is mainly implemented based on (3) and extracts a subimage ITEMP. The FuzExtractor procedure starts with seedRGB to generate IMATCH with region growing. Because seedRGB belongs to its subimage IMATCH, FuzExtractor pushes its four neighbor pixels into a queue and checks the first-in element, pixelFIRST, in the queue to see if it belongs to the subimage IMATCH. If pixelFIRST belongs to IMATCH, its new four neighbors are pushed into the queue. The FuzExtractor procedure repeats this process until the queue is empty, which indicates that there are no more pixels in the source image ISRC matching the seedRGB color. After FuzExtractor returns IMATCH, CreateSubImage assigns IMATCH to ITEMP as described.CreateSubImage uses two conditions to check ifITEMP is significant or not. The first condition is to check ITEMP/ISRC that it is a ratio of ITEMP area over ISRC one. If this ratio is too small, CreateSubImage ignores ITEMP as ITEMP holds too few pixels extracted from the source image ISRC. ITEMP is most likely affected by noise and is not significant for any object in the source image. CreateSubImage also uses the condition, (ITEMP/ISRC) &var (ITEMP), to check ITEMP, where var (ITEMP) is a variance defined by(9)ϖ=EITEMP2,varITEMP=EITEMP-ϖ2.A big ITEMP/ISRC indicates a large area size of ITEMP and a big var (ITEMP) implies that ITEMP contains the color components with a wide range distribution, as shown in Figure 4(b). Such a subimage is also considered as insignificant. If ITEMP is significant, CreateSubImage assigns ITEMP to a Matched subimage IM(i+1) and creates an UnmatchedIU(i+1), where IM(i+1) holds the color components matching the color of seedRGB and IU(i+1) holds unextracted pixels in the source image. CreateSubImage iteratively processes IU(i) until IU(i)= ø; that is, each pixel in the image is processed.There may exist tiny holes in the subimageIM(i). Because these tiny fragments may be insignificant for object pattern features, the algorithm removes them by filling up such tiny holes with their original pixels. Usually, these tiny fragments are caused by a small RGB distance of two pixels that have the same RGB values and are located in the same row or in the same column, so the algorithm checks such a distance if it is less than a threshold, δ = 7. If so, the algorithm fills the tiny holes when creating the subimage, IM(i). The algorithm generates a seed from the Unmatched subimage IU(i) but always creates a subimage IM(i) from the source image ISRC so it is very probable that the algorithm extracts an individual pixel into multiple subimages. We use a list to maintain IMi according to a sending order of their sizes. Table 3 shows an example of the proposed algorithm that creates 20 subimages from the source image IBIRD in Figure 5(a). Column 1 in Table 3 contains IM(i) ID, columns 2 and 3 list the locations and the color components of the seeds for extracting the subimages IM(i) using the FCE, column 4 shows the number of pixels extracted into the subimages IM(i), and column 5 records the variance of IM(i).Table 3
Subimages created fromIBIRD.
ID
Locations
RGB colors
Pixels
Variance
(
1
)
(269, 271)
(244, 251, 243)
14308
148
(
2
)
(283, 267)
(240, 251, 247)
9674
114
(
3
)
(267, 243)
(230, 252, 239)
64924
150
(
4
)
(193, 233)
(0, 6, 0)
15643
917
(
5
)
(358, 174)
(228, 246, 232)
10492
157
(
6
)
(276, 280)
(229, 232, 221)
15307
240
(
7
)
(292, 302)
(209, 222, 213)
10894
372
(
8
)
(58, 141)
(197, 216, 212)
13815
590
(
9
)
(292, 296)
(1, 12, 4)
3003
1419
(
10
)
(290, 302)
(11, 14, 5)
3529
1602
(
11
)
(464, 285)
(7, 22, 25)
3941
1591
(
12
)
(275, 252)
(17, 30, 13)
17645
1311
(
13
)
(320, 261)
(20, 38, 26)
4831
1961
(
14
)
(68, 138)
(54, 39, 16)
26009
2416
(
15
)
(216, 190)
(151, 100, 57)
2916
2394
(
16
)
(357, 158)
(50, 53, 46)
26769
2703
(
17
)
(293, 294)
(185, 198, 189)
11018
534
(
18
)
(352, 171)
(206, 191, 162)
67427
375
(
19
)
(49, 238)
(89, 72, 42)
28510
3311
(
20
)
(1, 320)
(58, 86, 108)
26845
4777Figure 5
Source image,IBIRD; segmented images in the RGB, HSV, and YUV space, ISEG, ISEG′, and ISEG′′; manually made benchmark, IBECHMARK; and partitioned objects in the RGB space, IOBJ(i).
(a)
I
B
I
R
D
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
C
H
M
A
R
K
(f)
I
O
B
J
(
1
)
(g)
I
O
B
J
(
2
)
(h)
I
O
B
J
(
3
)
(i)
I
O
B
J
(
4
)
(j)
I
O
B
J
(
5
)
(k)
I
O
B
J
(
6
)


## 5. Merging Subimages

The proposed algorithm merges significant subimages into a set of meaningful segmented objects by the procedure MergeSubImage. The pseudocode of the MergeSubImage procedure is shown in Pseudocode5.Pseudocode 5
MergeSubImage ({IM(i)})
K
←
I;
for i←1 to I do
for l←i-1 to I do
if (IMi∩IM(l))/IM(i)>τ;
I
M
(
l
)
←
I
M
(
i
)
∪
I
M
(
l
);
Remove IM(i) from the list;
K
←
K
-
1;
break;
for k←1 to K do
I
O
B
J
(
k
)
←
I
M
(
k
);
I
S
E
G
←
I
O
B
J
(
k
)
∪
I
O
B
J
(
k
-
1
)
⋯
I
O
B
J
(
1
);
returnISEG;The inputs to the procedure MergeSubImage are the subimagesIMi, and the output from MergeSubImage is the segmented image ISEG. Figures 5(a)-5(b) show a color image, IBIRD, and the segmented image in the RGB space, ISEG, which is a union of 6 objects, IOBJ(i),  i=1,…,6, shown in Figures 5(f)–5(k). Figure 5(e) shows the manually made benchmark, IBECHMARK.MergeSubImage merges the subimages according to the overlap sizes of their color components to construct some objects in the source image. In the process of the MergeSubImage, the FCE usually classifies an individual pixel into multiple subimages. It causes these subimages to share a common color area in the RGB space, and even probably a subimage may fully cover another subimage. The common area of two subimages implies two features: spatial proximity and color similarity, because the two subimages connect together and have a partition with the same color if they share the common area. Therefore, our algorithm uses this unique characteristic to merge subimages. MergeSubImage uses the union(IM(i)∩IM(l))/IM(i) to check the common area size of two subimages IM(i) and IM(l) in the RGB space. A bigger value of (IM(i)∩IM(l))/IM(i) means that IM(i) and IM(l) share a larger common area; that is, their color components are more similar. A data structure for maintaining the subimages IMi is a list that sorts {IM(i)} in a sending order of their sizes. Technical implementation of merging IMi is shown in Figure 6. The merging process starts with the last element IM(i) with i=I, located at the list tail, which is the subimage with the smallest area size, and checks IM(l) with each subimage {IM(l),l=i-1,…,1} via (IMi∩IMl)/IM(i). If (IM(i)∩IM(l))/IM(i)>τ, the algorithm merges IM(i) and IM(l) together, removes IM(i) from the list, and replaces IM(i) with IM(i)∪IM(l), where IM(l) size is bigger than IM(i) one. After processing all the subimages in the list, the list holds the partitioned objects {IOBJ(i)} from the input source image ISRC. MergeSubImage returns the partitioned objects {IOBJ(k)} as well as its union, IOBJ(k)∪IOBJk-1⋯IOBJ(1), as the segmented image ISEG.Figure 6
Process for merging subimagesIM(i) and IM(l).Table4 lists the extracted subimages from the source image IBIRD in Figure 5(a). MergeSub- Image merges these 20 subimages into 6 partitioned objects {IOBJ(i)} in Figures 5(f)–5(k). Table 4 also shows information on which and how many subimages are merged into which object. For example, the partitioned object IOBJ(1) shown in Figure 5(f) comes from a single subimage IM(11), while the partitioned object IOBJ(5) in Figure 5(j) is produced by merging 11 subimages {IM(9), IM(22), IM(23), IM(27), IM(29), IM(32), IM(34), IM(36), IM(38), IM(41), IM(43)}. Figure 5(b) shows the segmented image ISEG by a union operation of IOBJ(6)∪IOBJ(5)⋯IOBJ(1).Table 4
I
O
B
J
(
i
) merged by subimages created from IBIRD.
Segmented objects
Sub-images
I
O
B
J
(
1
)
I
M
(
11
)
I
O
B
J
(
2
)
I
M
(
2
), IM(19), IM(22)
I
O
B
J
(
3
)
I
M
(
22
)
I
O
B
J
(
4
)
I
M
(
22
), IM(22)
I
O
B
J
(
5
)
I
M
(
22
), IM(22), IM(22), IM(22), IM(22), IM(22), IM(22), IM(22), IM(22), IM(22), IM(22)
I
O
B
J
(
6
)
I
M
(
22
), IM(22)


## 6. Discussions and Conclusions

In this section, we present our selection of the membership functions’ parameters of the FCE defined in (4)–(7) and settings of the CreateSubImage and MergeSubImage procedures used for our tests. We empirically selected the parameters of the membership functions as α1=60, α2=200, ρM=20, ρM+ρU=255, and σ=90. A smaller α1 may more precisely segment color components in an input image but produces more extracted subimages. A slight change of α2 does not affect the segmentation result, so it is not sensitive. ρM should be at least less than 127.5. The threshold, σ, is important for classifying pixels into a Matched subimage IM(i) or an Unmatched subimage IU(i). We select the parameters in CreateSubImage and MergeSubImage as ε=0.01, γ1=0.25, γ2=500, and τ=0.6. The function of ε is to remove insignificant subimages with a “too small” area that may be caused by noise. Both parameters γ1 and γ2 are defined to remove another type of insignificant subimages, for example, ITEMP shown in Figure 4(b). This subimage is not extracted very well due to a “bad seed,” seedRGB=(64,28,28). Such a subimage is featured by its large area size and a large variance of its color components. The large variance indicates that the subimage holds a wide range of color distribution in the RGB space, so this subimage should be removed. τ is the threshold to check the overlap area over the subimage with a smaller area. In this paper, τ=0.6 herein means that the overlap area is 60% of the subimage with the smaller area. The bigger τ is, the more similar the color components of the two subimages are. Changing τ affects the number of partitioned objects generated by MergeSubImage. As an extreme example, τ=1.0 means that only if the larger subimage fully overlaps the smaller subimage, the two subimages will be merged together.In our implementation, the algorithm invokes the CreateSubImage procedure twice. For the first invocation, an input image,IIMG, to IniSeedList and UpdateSeedList, is ISRC-ICENTER, where ICENTER is extracted by seedCENTER=(127,127,127). Because the first call is to process a source image ISRC, the algorithm does not select a seed close to the RGB space center. As discussed in Section 4, the seed seedCENTER cannot differentiate color components which are far from the well-defined eight colors. For example, the extracted image ICENTER shown in Figure 4(a) holds many color components of six corn cobs. Because the algorithm does not select any seed from ICENTER, it is very possible that some of the pixels in the source image remain unclassified. In order to process the unclassified pixels, we create an image IUNCLASSIED=ISRC-∪IOBJ(i), as shown in Figure 4(c). The algorithm calls the CreateSubImage a second time to process IUNCLASSIED, but the input image to IniSeedList and UpdateSeedList is IUNCLASSIED itself. The variance of IUNCLASSIED is smaller than that of its original source image ISRC as most pixels with a large range of color distribution are classified after first calling CreateSubImage. For this reason, we halve the parameter α1=30 of the membership functions to process IUNCLASSIED. The proposed approach segments the color image ICORN in Figure 7(a) into 12 objects. Figure 7(b) shows the segmented image. Figures 7(f)–7(q) display the first 12 partitioned objects. IOBJ(3) and IOBJ(6) in Figures 7(h) and 7(k) are extracted from IUNCLASSIED in Figure 4(c).Figure 7
Source image,ICORN; segmented images in the RGB, HSV, and YUV space, ISEG, ISEG′; and ISEG′′; manually made benchmark, IBECHMARK; and partitioned objects in the RGB space, IOBJ(i).
(a)
I
C
O
R
N
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
N
C
H
M
A
R
K
(f)
I
O
B
J
(
1
)
(g)
I
O
B
J
(
2
)
(h)
I
O
B
J
(
3
)
(i)
I
O
B
J
(
4
)
(j)
I
O
B
J
(
5
)
(k)
I
O
B
J
(
6
)
(l)
I
O
B
J
(
7
)
(m)
I
O
B
J
(
8
)
(n)
I
O
B
J
(
9
)
(o)
I
O
B
J
(
1
0
)
(p)
I
O
B
J
(
1
1
)
(q)
I
O
B
J
(
1
2
)In order to test the robustness of the proposed algorithm, we applied the algorithm with the same settings defined in the first paragraph of this section to process the five color images{IBIRD,ICORN,ILAKE,IMAN,IBABOON}, in the RGB color space as shown in Figures 5(a), 7(a), 8(a), 9(a), and 10(a), respectively. Their corresponding segmented images are displayed in Figures 5(b), 7(b), 8(b), 9(b), and 10(b), respectively. The first four color images are from the database at UC Berkeley [23]. These selected color images represent different features and diverse scenes that might be taken under different light conditions. The source image IBIRD in Figure 5(a) holds a bird on a tree trunk under the sky. The source image ICORN in Figure 7(a) holds six corns with many small multicolor kernels and its segmented image holding the six colorful corns. The source image ILAKE in Figure 8(a) shows a lake and a mountain under the sky at evening. Figures 9(a) and 10(a) show the color images IMAN and IBABOON. Table 5 lists the subimages, and the partitioned objects from the 5 color images. 85 subimages are extracted from IBABOON, but the partitioned objects are only 6.Table 5
Statistics of subimages and partitioned objects of five color images.
ID
I
B
I
R
D
I
C
O
R
N
I
L
A
K
E
I
M
A
N
I
B
A
B
O
O
N
I
M
(
i
)
20
41
16
22
85
I
O
B
J
(
i
)
6
13
6
54
6Figure 8
Source image,ILAKE; segmented images in the RGB, HSV, and YUV space, ISEG, ISEG′, and ISEG′′; manually made benchmark, IBECHMARK; segmented images with Canny edge detection and Log edge detection, ICanny, ILog.
(a)
I
L
A
K
E
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
N
C
H
M
A
R
K
(f)
I
C
a
n
n
y
(g)
I
L
o
gFigure 9
Source image,IMAN; segmented images in the RGB, HSV, and YUV space, ISEG, ISEG′, and ISEG′′; manually made benchmark, IBECHMARK; segmented images with Canny edge detection and Log edge detection, ICanny, ILog.
(a)
I
M
A
N
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
N
C
H
M
A
R
K
(f)
I
C
a
n
n
y
(g)
I
L
o
gFigure 10
Source imageIBABOON and its segmented image ISEG.
(a)
I
B
A
B
O
O
N
(b)
I
S
E
GThere is a lack of a strong mathematical model for precisely expressing the human’s understanding of colors due to psychophysical perception. For a given color, different observers may perceive its color definition differently. That is why the FCE-based approach classifying an individual pixel into several subimages is reasonable. The proposed approach to color image segmentation is easy to use, and it does not apply any procedure where one needs to preprocess source images. Removing subimages with “too small” areas is filtering short edges and noise in a color image. The approach automatically partitions a color image into an appropriate number of meaningful objects or regions. The manually made benchmarks are downloaded from the database at UC Berkeley [23], as shown in Figures 5(e), 7(e), 8(e), 9(e), 11(e), and 12(e), respectively. These benchmarks display desired segmentation results represented by the images’ boundaries. Although the proposed approach does not reach the benchmarks’ results, it is able to partition the color images into meaningful objects, for example, the ICORN segmentation results as shown in Figure 7.Figure 11
Source image,IHORSE; segmented images in the RGB, HSV, and YUV space, ISEG, ISEG′, and ISEG′′; manually made benchmark, IBECHMARK; segmented images with Canny edge detection and Log edge detection, ICanny, ILog.
(a)
I
H
O
R
S
E
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
N
C
H
M
A
R
K
(f)
I
C
a
n
n
y
(g)
I
L
o
gFigure 12
Source image,IVEGGI; segmented images in the RGB, HSV, and YUV space ISEG, ISEG′, and ISEG′′; manually made benchmark, IBECHMARK; segmented images with Canny edge detection and Log edge detection, ICanny, ILog.
(a)
I
V
E
G
G
I
(b)
I
S
E
G
(c)
I
S
E
G
′
(d)
I
S
E
G
′
′
(e)
I
B
E
N
C
H
M
A
R
K
(f)
I
C
a
n
n
y
(g)
I
L
o
gIn this study, we also compared the performance of the FCE-based algorithm by segmenting the same test imagesIBIRD,ICORN,ILAKE,IMAN,IHORSE,IVEGGI in the RGB, HSV, and YUV spaces, respectively. In this case, we only transformed the color image representation from the RGB space into the HSV and YUV spaces according to (1) and (2). The test images are selected randomly from the database at UC Berkeley [23], as shown in Figures 5(a), 7(a), 8(a), 9(a), 11(a), and 12(a). Their corresponding segmented images in the RGB, HSV, and YUV spaces are displayed in Figures 5(b)–5(d), 7(b)–7(d), 8(b)–8(d), 9(b)–9(d), 11(b)–11(d), and 12(b)–12(d). We find that the segmented images in the HSV space ISEG′ are the most different from IBENCHMARK by comparison. So the FCE-based algorithm defined in the HSV space is not adopted. In some cases, the segmentation results in the YUV space are superior to the RGB spaces, such as the “face” of IMAN in Figure 9 and the “boxes” in IVEGGI in Figure 12. But in most cases, the segmented images in the RGB space ISEG are the most similar to IBENCHMARK. It is proven that the FCE-based algorithm defined in the RGB space is the most robust. The possible reason for the results is the transformation of the color image representation from the RGB space into the HSV and YUV spaces may cause color errors so that the segmentation is influenced.In order to show the advantages of the new approach to color image segmentation, we applied Canny edge detection and Log edge detection algorithms to segment the test imagesILAKE,IMAN,IHORSE,IVEGGI, respectively, and the segmented images were shown in Figures 8(f)-8(g), 9(f)-9(g), 11(f)-11(g), and 12(f)-12(g). Then we compared ICanny and ILog with the segmented images through the FCE-based approach, namely, ISEG, ISEG′, and ISEG′′, and meanwhile IBECHMARK can be used as a metric to evaluate the segmentations. As shown in Figures 8 and 9, the edges in ICanny and ILog are “too many” or “too few” compared with IBECHMARK, which proves that the segmentation effect with Canny edge detection and Log edge detection is inferior to the FCE-based approach. Although the segmentation effect with Log edge detection is superior to the FCE-based approach applied in the HSV and YUV spaces, it performs not well as the proposed approach applied in the RGB color space, as shown in Figures 11 and 12. Overall, compared with Canny edge detection and Log edge detection, the new approach based on FCE defined in the RGB color space presents the best performance in the color image segmentation.The approach is effective in tracking or following a certain color because the FCE-based approach does not need a precise color pattern to extract a set of colors. We apply this algorithm to extract chemical plumes in images taken undersea at very poor illumination conditions [26]. In this paper, we start selecting seeds near the eight well-defined colors, that is, at the vertices of the RGB space, and move their selection from the vertices to the RGB center. Our further research will investigate more effective methods for selecting seeds because seed selection plays an important role in achieving a good segmentation performance. Another issue we will address in our study is how to automatically determine the parameters of fuzzy sets based on an input color image.



---

*Source: 7347421-2017-09-25.xml*