# Image Retrieval Based on a Multi-Integration Features Model

Authors: Kai Chu; Guang-Hai Liu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Feature integration theory can be regarded as a perception theory, but the extraction of visual features using such a theory within the CBIR framework is a challenging problem. To address this problem, we extract the color and edge features based on a multi-integration features model and use these for image retrieval. A novel and highly simple but efficient visual feature descriptor, namely, a multi-integration features histogram, is proposed for image representation and content-based image retrieval. First, a color image is converted from the RGB to the HSV color space, and the color features and color differences are extracted. Then, the color differences are calculated to extract the edge features using a set of simple integration processes. Finally, combining the color, edge, and spatial layout features allows representing the image content. Experiments show that our method produces results comparable to existing and well-known methods on three datasets that contain 25,000 natural images. The performances are significantly better than that of the BOW histogram, local binary pattern histogram, histogram of oriented gradient, and multi-texton histogram, with performances similar to the color volume histogram.


---

## Body

## 1. Introduction

The CBIR technique originated in the 1990s and has become a research hotspot over the last thirty years. The CBIR has been used to describe the process of retrieving similar images from a large collection based on the image content (color, shape, and texture). The essence of the traditional CBIR is associated with similar image retrieval. Several image retrieval techniques have been developed through years of technical innovation, such as content-based and object-based methods, as well as methods based on machine or deep learning. The classic CBIR is markedly different from object recognition as most images are similar, rather than containing the same objects. Thus, training and testing sets cannot be distinguished in the classic CBIR.Feature extraction and representation is the most important issue in CBIR and has a close relationship with human perception, where color and edge cues are among the visual search components for perception stimuli and can express meaningful characteristics in images or scenes. Feature integration theory is a perceptual and attentional theory that explains how an individual combines pieces of observable information about an object to form a complete perception [1]. The first stage is preattention, in which we focus on one distinguishing attribute of the object. The second stage is focused attention, in which we take all the observed edge features and combine them to make a complete perception [1]. Representing image contents via extracting the color and edge features based on feature integration theory is a challenging problem. To address this problem, a novel and highly simple but efficient representation based on the multi-integration features model, namely the multi-integration features histogram (MIFH), is proposed for CBIR.The proposed model is based on similar image retrieval rather than object-based image retrieval or object classification. Our main contributions are summarized as follows. First, a new method is proposed to extract lower-level features, which simulates the preattention stage, where color differences are used as the novel distinguishing attribute of an object. Second, we take all of the observed edge features and combine them to make a complete perception using a set of simple integration processes to provide powerful information for image representation. Third, a new image representation is proposed for image retrieval that combines color, edge, and spatial layout into an integrational feature. These attributes improve the discriminative power for color, texture, and shape features.The rest of this paper is organized as follows. In Section2, we review existing visual feature description methods and image retrieval techniques. In Section 3, the multi-integration features model and MIFH method are presented. We conduct CBIR experiments in Section 4, while Section 5 concludes the paper.


## 2. Related Works

Low-level features are highly popular in pattern recognition techniques. Color, texture, and shape features are widely used attributes in saliency detection [2] and traditional content-based image retrieval [3–10]. In the 2000s, ISO developed a multimedia content description standard (MPEG-7) to represent information regarding its content. The color, texture, and shape descriptors are the core components in this standard [3]. In the early stages of CBIR development, the use of a color histogram to describe image content has been a highly popular method in CBIR, since the color histogram is invariant to the orientation and scale. This characteristic makes it powerful for image classification and CBIR [3, 5–7].Texture is also a highly popular visual feature for CBIR as it yields information regarding the spatial arrangement of the color or intensity in an image or region. The texture descriptors can be used to characterize image textures or regions, where the local binary pattern (LBP) is a famous texture descriptor [11]. In recent years, some LBP methods have been proposed for content-based image retrieval [12–14], such as the multichannel decoded LBP method [12] and the edge-local binary pattern (edgeLBP) approach [14]. In recent years, many algorithms have been proposed that combine multivisual cues [3–10, 12, 13] and have been utilized for various tasks. Moreover, simulating the mechanism of the human cortex to extract visual features provides a new avenue for CBIR. A set of Gabor filters with different frequencies and orientations can be used to extract useful visual features from an image, such as texture and edge information. This is similar to the perception of the human visual system and has been found to be particularly appropriate for texture representation and discrimination [15–18]. Liu et al. proposed the intensity variation descriptor [19] and the gradient-structures histogram [20] to represent image content and used them for image retrieval. The proposed representation methods have the advantages of being histogram-based and having the power to discriminate spatial layout, color and edge cues.Shape features are widely used in object recognition and image retrieval because they contain important semantic information. Humans have the ability to recognize objects through shape; however, shape can only be extracted from a set of accurate segmentation methods, which is a complicated process. In many image processing cases, shape descriptors may also be required to be invariant to translations, rotations, and scale. To avoid the requirement of accurate segmentation, some feature descriptors exhibit strength in object recognition, such as shape context [21], scale-invariant feature transform (SIFT) descriptors [22], histogram of oriented gradients [23], and SURF descriptors [24]. The histogram of oriented gradient (HOG) is a feature descriptor that is widely used for object detection. Object-based image retrieval has become increasingly popular since 2004 when Lowe proposed the SIFT descriptors. This type of image retrieval usually uses BOW methods to represent local features. In subsequent years, various BOW-based methods have been proposed for object-based image retrieval, object recognition, and other applications [25–28]. Ahmed et al. proposed a new approach for CBIR by fashioning the spatial color information with shape and object features [29]. Pradhan et al. proposed a new method for CBIR that uses color and shape features [30, 31]. In some cases, the representation methods using dimensionality reduction [32] and structured optimal graph [33] have the power to extract the shape feature.In recent years, deep learning techniques represented by convolutional neural networks (CNNs) have been shown to be effective at various vision tasks [17, 18, 34–39]. In early works, the approach of applying CNNs for image retrieval established the fully connected layer activation functions to be the global image descriptors [17, 35–37]. Deep learning approaches represented by CNNs have demonstrated a superior performance over hand-crafted features. For instance, Qayyum et al. proposed a framework of deep learning for CBMIR systems using a deep CNN that was trained for classification of medical imagery [38]. Tzelepi and Tefas employed a deep CNN model to obtain feature representation from the activations of convolutional layers using max pooling for CBIR [18]. Alzu’bi et al. introduced compact bilinear CNN-based architectures for several CBIR tasks using two parallel feature extractors without prior knowledge of the semantic metadata of image content [39].Deep learning techniques can provide better performance overall in image retrieval and object recognition but require long training times. These approaches also do not typically mimic the mechanism of the human brain or the perception mechanism. Extracting visual features and representing image contents that mimic the perception mechanism within the CBIR framework requires further study.


## 3. The Multi-Integration Features Model and Representation

In digital image processing, color feature extraction, and edge detection play an important role in feature extraction and includes high-level concepts [40, 41]. In applications of CBIR and object recognition, color can provide powerful information for feature extraction and representation [2, 9]; even in the absence of shape information, combining color features with other visual features, such as textures, edge cues, and spatial attributes, is a popular technique to improve the image retrieval performance [19, 20]. Color and edge cues are visual search components for stimuli perception that can express meaningful characteristics of images or scenes. To represent the image content by extracting color and edge features based on feature integration theory, we propose a multi-integration features model to represent image content and use it for CBIR. The architecture overview of the process is shown in Figure 1.Figure 1
The architecture overview of the multi-integration features model.There are three main components of the architecture shown in Figure1. The first component is the preattention stage which is highlighted with the red solid line, in which we focus on one distinguishing attribute of the object using color differences. The second component is the focused attention, which is highlighted with the purple dotted line, in which we take all the observed edge features and combine them to make a complete perception. Three integrations of the edge features are used with a set of simple integration processes. For the third component, we propose a novel and highly simple but efficient visual descriptor to represent an image, which is used for CBIR. The E1, E2, and E3 integrations, color map, and log transformation processes occur simultaneously in the histogram.


### 3.1. Color Feature Extraction and the Initial Feature Integration

The HSV color space can mimic the human color perception well; therefore, many researchers have used it for color quantization [6, 8, 10, 19, 20]. The HSV color space is based on the cylindrical coordinate system and is defined in terms of the three components of hue (H), saturation (S), and value (V), where S is the distance perpendicular to the rod, V is the central rod, and H is the angle around the rod [19, 20, 40, 41]. In the proposed MIFH method, the color feature is extracted using color quantization.To simplify the color quantization operation in the HSV color space, each of theH, S, and V color channels are uniformly quantized into 6, 3, and 3 bins, respectively, which results in a total of 6 × 3 × 3 = 54 color combinations. Let M0x,y be the color combination or the color map as M0x,y=w,w∈0,1,…,NC−1, where w denotes the index value of the color map in the pixel coordinate x,y, and NC=54. The M0x,y is used to calculate the color features in the proposed model.To perform the initial feature integration, we focus on one distinguishing attribute of the object using the color difference, which we calculate in the HSV color space for the integration processes. The initial feature integration involves the characteristics of neurons that it receives a set of input signals and produces output, and the activation and excitation states of the neurons are simulated by using a threshold [42]. In order to simulate such characteristics, the sigmoid function sigmoid. is used as the active function in the initial feature integration, and it can map the input into an output in the range of 0,1.We then transform the cylindrical coordinate system to Cartesian coordinates. LetH,S,V be a random dot in cylindrical coordinates, and h,s,v is the transformation of H,S,V in Cartesian coordinates, where h=S⋅cosH, s=S⋅sinH, and v=V. In the input image, the central coordinates are denoted as x0,y0, and the coordinates for the eight neighboring pixels are denoted as xi,yi,i∈1,2,3,…,8. The color difference map is defined as follows:(1)Cx0,y0=sigmoid∑i=18ΔHi2+ΔSi2+ΔVi2+b,ΔHi=hxi,xi−hx0,y0,ΔSi=sxi,xi−sx0,y0,ΔVi=vxi,xi−vx0,y0,where ΔHi, ΔSi, and ΔVidenote the color differences between the pixel locations x0,y0 and xi,yi for the H, S, and V components. In the active function sigmoid., the bias parameter is b=1.0, and the value of the bias can measure how easy it is for a neuron to generate positive (negative) excitation. The color difference for the local regions contains rich information as it can measure the degree of color differences. Figure 2 shows that the edge and important regions can be simply detected using such operations. At the same time, the information for the three-color channels was also combined into a color difference map Cx,y.Figure 2
Illustration for the computation of color differences. (a) The center point and its eight neighborhoods. (b) Color difference map derived from an input image (top right), where Δd denotes the distance between the center point and its neighborhood points. The coordinates of points wi are denoted as xi,yi,i=0,1,2,…,8.
(a)
(b)A subsampling operation is implemented in the color difference mapCx,y with a scale of two, and the subsampling image is denoted as Sx,y. The size of Sx,y is one quarter that of Cx,y.


### 3.2. First Integration of the Edge Features

When extracting edge features forSx,y, we adopt the horizontal and vertical Sobel kernels as the weight values between feature maps. The Sobel operator is less sensitive to noise than other gradient operators or edge detectors while also being relatively efficient [40, 41]. These kernels are defined as follows:(2)gx=−10+1−20+2−10+1,gy=−1−2−1000+1+2+1.We obtain the edge mapG0x,y using the Sobel operation G. which is denoted as(3)G0x,y=GSx,y.To extract more edge features, we need to perform the same actionsG. three times to obtain the remaining three edge maps again as(4)Gix,y=GKix,y,i∈1,2,3,whereKix,y=Sx,y+∑k=1iGk−1x,y.We obtain the four edge feature maps and combine them into a gradient imageE1x,y, which can be calculated as(5)E1x,y=∑i=03Gix,y2.Then, the subsampling operation is implemented inGix,y,i∈0,1,2,3 with a scale of two, and the subsampling feature maps are denoted as Gix,y,i∈0,1,2,3. The size of Gix,y is one quarter that of Gix,y.


### 3.3. Second Integration of the Edge Features

For every feature mapGix,y,i∈0,1,2,3, the same actions G. as the equations (3) and (4) need to be performed. As a result, a total of 16 edge maps Gi,jx,y,i,j∈0,1,2,3 were obtained. We first combine them into four gradient images Fix,y,i∈0,1,2,3 using (5), which can be calculated as(6)Fix,y=∑j=03Gi,jx,y2,i∈0,1,2,3.The four gradient imagesFix,y,i∈0,1,2,3 are further combined into a single gradient image E2x,y, which can be calculated as(7)E2x,y=∑i=03Fix,y.Then, a subsampling operation is implemented forFix,y,i∈0,1,2,3, where the scale is two, and the subsampling feature maps are denoted as ℱix,y,i∈0,1,2,3. The size of ℱix,y is one quarter of Fix,y.


### 3.4. Third Integration of the Edge Features

We combine the feature mapsℱix,y,i∈0,1,2,3 into a single gradient image E3x,y with(8)E3x,y=∑i=03ℱix,y.To represent the image content, we uniformly quantize the gradient imagesE1x,y, E2x,y, and E3x,y in different steps, where their index maps are defined as M1x,y, M2x,y, and M3x,y, respectively.In the index mapsMkx,y, k=1,2,3,, and their index values are Mkx,y=v,v∈0,1,…,Nv−1. We set the number of bins to Nv=16. In this study, the color index map M0x,y and the edge index maps Mkx,y,k=1,2,3 are used to calculate the visual features in the proposed model.


### 3.5. Feature Representation

The spatial layout information and histogram-based methods are widely used in CBIR and to describe visual features. Here, co-occurrence is adopted for image representation. Letx,y and x+Δx,y+Δy be two neighboring pixel locations in an image, where Δx and Δy denote the offsets of the x-axis and the y-axis, respectively. Two neighboring pixels have a distance of d=maxΔx,Δy. We use N. to denote the co-occurring number of the two-pixel values and then define the co-occurring histogram of the color map and three edge maps as follows:(9)HkMkx,yNMkx,y=v1∧Mkx+Δx,y+Δy=v2,whereMkx,y=Mkx+Δx,y+Δy=v1=v2,k=0,1,2,3.Following the concatenation ofHk,k∈0,1,2,3, we obtain the histogram hi,i∈0,1,…,Nc+3Nv−1. The subscript of the histogram is denoted as i,i∈0,1,…,NC+3Nv−1. A log transformation is applied to adjust the dynamic range in the histograms hi. We then define the MIFH as follows:(10)Hi=loghi,loghi>0.0,0.0,else.In image processing, the log function is important as it compresses the dynamic range of images that have large variations in their pixel values [41]. The log transformation expands the values for lower-level pixels in an image while compressing the higher-level values, thus emphasizing the lower-level pixels. In the proposed method, the use of a log function can reduce the effects of errors as derived from feature quantization and contains the spatial information obtained using the co-occurring neighboring pixels. Moreover, the proposed method can discriminate color, texture, and shape features.


## 3.1. Color Feature Extraction and the Initial Feature Integration

The HSV color space can mimic the human color perception well; therefore, many researchers have used it for color quantization [6, 8, 10, 19, 20]. The HSV color space is based on the cylindrical coordinate system and is defined in terms of the three components of hue (H), saturation (S), and value (V), where S is the distance perpendicular to the rod, V is the central rod, and H is the angle around the rod [19, 20, 40, 41]. In the proposed MIFH method, the color feature is extracted using color quantization.To simplify the color quantization operation in the HSV color space, each of theH, S, and V color channels are uniformly quantized into 6, 3, and 3 bins, respectively, which results in a total of 6 × 3 × 3 = 54 color combinations. Let M0x,y be the color combination or the color map as M0x,y=w,w∈0,1,…,NC−1, where w denotes the index value of the color map in the pixel coordinate x,y, and NC=54. The M0x,y is used to calculate the color features in the proposed model.To perform the initial feature integration, we focus on one distinguishing attribute of the object using the color difference, which we calculate in the HSV color space for the integration processes. The initial feature integration involves the characteristics of neurons that it receives a set of input signals and produces output, and the activation and excitation states of the neurons are simulated by using a threshold [42]. In order to simulate such characteristics, the sigmoid function sigmoid. is used as the active function in the initial feature integration, and it can map the input into an output in the range of 0,1.We then transform the cylindrical coordinate system to Cartesian coordinates. LetH,S,V be a random dot in cylindrical coordinates, and h,s,v is the transformation of H,S,V in Cartesian coordinates, where h=S⋅cosH, s=S⋅sinH, and v=V. In the input image, the central coordinates are denoted as x0,y0, and the coordinates for the eight neighboring pixels are denoted as xi,yi,i∈1,2,3,…,8. The color difference map is defined as follows:(1)Cx0,y0=sigmoid∑i=18ΔHi2+ΔSi2+ΔVi2+b,ΔHi=hxi,xi−hx0,y0,ΔSi=sxi,xi−sx0,y0,ΔVi=vxi,xi−vx0,y0,where ΔHi, ΔSi, and ΔVidenote the color differences between the pixel locations x0,y0 and xi,yi for the H, S, and V components. In the active function sigmoid., the bias parameter is b=1.0, and the value of the bias can measure how easy it is for a neuron to generate positive (negative) excitation. The color difference for the local regions contains rich information as it can measure the degree of color differences. Figure 2 shows that the edge and important regions can be simply detected using such operations. At the same time, the information for the three-color channels was also combined into a color difference map Cx,y.Figure 2
Illustration for the computation of color differences. (a) The center point and its eight neighborhoods. (b) Color difference map derived from an input image (top right), where Δd denotes the distance between the center point and its neighborhood points. The coordinates of points wi are denoted as xi,yi,i=0,1,2,…,8.
(a)
(b)A subsampling operation is implemented in the color difference mapCx,y with a scale of two, and the subsampling image is denoted as Sx,y. The size of Sx,y is one quarter that of Cx,y.


## 3.2. First Integration of the Edge Features

When extracting edge features forSx,y, we adopt the horizontal and vertical Sobel kernels as the weight values between feature maps. The Sobel operator is less sensitive to noise than other gradient operators or edge detectors while also being relatively efficient [40, 41]. These kernels are defined as follows:(2)gx=−10+1−20+2−10+1,gy=−1−2−1000+1+2+1.We obtain the edge mapG0x,y using the Sobel operation G. which is denoted as(3)G0x,y=GSx,y.To extract more edge features, we need to perform the same actionsG. three times to obtain the remaining three edge maps again as(4)Gix,y=GKix,y,i∈1,2,3,whereKix,y=Sx,y+∑k=1iGk−1x,y.We obtain the four edge feature maps and combine them into a gradient imageE1x,y, which can be calculated as(5)E1x,y=∑i=03Gix,y2.Then, the subsampling operation is implemented inGix,y,i∈0,1,2,3 with a scale of two, and the subsampling feature maps are denoted as Gix,y,i∈0,1,2,3. The size of Gix,y is one quarter that of Gix,y.


## 3.3. Second Integration of the Edge Features

For every feature mapGix,y,i∈0,1,2,3, the same actions G. as the equations (3) and (4) need to be performed. As a result, a total of 16 edge maps Gi,jx,y,i,j∈0,1,2,3 were obtained. We first combine them into four gradient images Fix,y,i∈0,1,2,3 using (5), which can be calculated as(6)Fix,y=∑j=03Gi,jx,y2,i∈0,1,2,3.The four gradient imagesFix,y,i∈0,1,2,3 are further combined into a single gradient image E2x,y, which can be calculated as(7)E2x,y=∑i=03Fix,y.Then, a subsampling operation is implemented forFix,y,i∈0,1,2,3, where the scale is two, and the subsampling feature maps are denoted as ℱix,y,i∈0,1,2,3. The size of ℱix,y is one quarter of Fix,y.


## 3.4. Third Integration of the Edge Features

We combine the feature mapsℱix,y,i∈0,1,2,3 into a single gradient image E3x,y with(8)E3x,y=∑i=03ℱix,y.To represent the image content, we uniformly quantize the gradient imagesE1x,y, E2x,y, and E3x,y in different steps, where their index maps are defined as M1x,y, M2x,y, and M3x,y, respectively.In the index mapsMkx,y, k=1,2,3,, and their index values are Mkx,y=v,v∈0,1,…,Nv−1. We set the number of bins to Nv=16. In this study, the color index map M0x,y and the edge index maps Mkx,y,k=1,2,3 are used to calculate the visual features in the proposed model.


## 3.5. Feature Representation

The spatial layout information and histogram-based methods are widely used in CBIR and to describe visual features. Here, co-occurrence is adopted for image representation. Letx,y and x+Δx,y+Δy be two neighboring pixel locations in an image, where Δx and Δy denote the offsets of the x-axis and the y-axis, respectively. Two neighboring pixels have a distance of d=maxΔx,Δy. We use N. to denote the co-occurring number of the two-pixel values and then define the co-occurring histogram of the color map and three edge maps as follows:(9)HkMkx,yNMkx,y=v1∧Mkx+Δx,y+Δy=v2,whereMkx,y=Mkx+Δx,y+Δy=v1=v2,k=0,1,2,3.Following the concatenation ofHk,k∈0,1,2,3, we obtain the histogram hi,i∈0,1,…,Nc+3Nv−1. The subscript of the histogram is denoted as i,i∈0,1,…,NC+3Nv−1. A log transformation is applied to adjust the dynamic range in the histograms hi. We then define the MIFH as follows:(10)Hi=loghi,loghi>0.0,0.0,else.In image processing, the log function is important as it compresses the dynamic range of images that have large variations in their pixel values [41]. The log transformation expands the values for lower-level pixels in an image while compressing the higher-level values, thus emphasizing the lower-level pixels. In the proposed method, the use of a log function can reduce the effects of errors as derived from feature quantization and contains the spatial information obtained using the co-occurring neighboring pixels. Moreover, the proposed method can discriminate color, texture, and shape features.


## 4. Experimental Results

In the experiments, we follow the traditional CBIR (similar image retrieval) rather than object-based or similar types of image retrieval. We compare the MIFH method with the color volume histograms [4], local binary pattern (LBP) [11], multi-texton histogram (MTH) [6], histogram of oriented gradients (HOG) [23], and BOW histogram [25]. Besides, we also compare the MIFH method with some extend methods which are derived from our previous works (the multi-texton histogram [6] and the micro-structure descriptor [8]), including CPV-THF [43], STH [44], and CMSD [45] methods. In the comparison experiment, the vector dimensionality for the MIFH, LBPH, BOW, MTH, and CVH methods are 102, 256, 1000, 82, and 104 bins, respectively. The vector dimensionality for the CPV-THF [43], STH [44], and CMSD [45] methods are 242, 172, and 88 bins, respectively. In the HOG descriptor, there are nine bins with a block size of three and a cell size of six. For a fair comparison, the MIFH, LBPH, BOW, MTH, and CVH methods adopt the L1 distance as the similarity measure or their original distance measures.


### 4.1. Benchmark Datasets

The Corel image dataset is the most commonly used dataset to test the performance of the CBIR. In Corel image dataset, every category in the Corel gallery contains 100 images, on which digital zoom can be used to obtain various image sizes. To measure the image retrieval performance, we select categories to build the Corel-10K and Corel-5K datasets and used them in the CBIR, where Corel-5K contains 50 categories and 5,000 images, and Corel-10K dataset contains 100 categories and 10,000 images which contain all images of Corel-5K dataset.The third dataset is the GHIM-10K with 10,000 images. All images in this dataset are sourced from the web or a camera and were collected byGuang-Hai Liu. The database contains 20 categories, such as building, sunset, fish, flower, car, mountain, and tiger. In the GHIM-10K dataset, each category contains 500 images at 400 × 300 or 300 × 400 in JPEG format. Corel-10K, Corel-5K, and GHIM-10K datasets can be downloaded from http://www.ci.gxnu.edu.cn/cbir/Dataset.aspx.Three subsets, consisting of ten percent of each dataset were used as the query images. The performance is evaluated using the average results for each query in terms of precision and recall.


### 4.2. Performance Measurements

Selecting the metric for the performance evaluation is critical to compare CBIR experiments. In many CBIR cases [3–10], precision and recall metrics are popular ways to evaluate the performance of CBIR or other text retrieval techniques [46].On the Corel datasets, each category contains 100 images and is denoted ask=100, while k=500 for the GHIM-10K dataset. If there are n retrieved images and m images within the retrieved images are relevant to the query for the image retrieval, then the precision and recall metrics can be defined as follows [46]:(11)Precision=mn,Recall=mk.We setn=12 in our CBIR system, i.e., there are 12 retrieved images in the image retrieval.


### 4.3. Retrieval Performance

Feature selection is a very important issue in image representation. For example, edges and colors are widely used features in content-based image retrieval. In image representation, low-level features are combined into the proposed algorithm where quantization is used to extract the features. Various feature quantizations are important factors in CBIR experiments. Feature quantization has a greater contribution to the performance of content-based image retrieval because the proposed MIFH method is a histogram-based method. Thus, the number of edge bins and color feature bins can directly influence the retrieval results.


#### 4.3.1. Investigation of Vector Dimensionality

To investigative the performance of the MIFH method using different histogram bins, we set the bin numbers of the color feature to 54, 72, 108, and 192 bins, respectively. For the first, second, and third integrations, the bins of the edge maps are set to 16, 32, and 64 bins, respectively. As shown in Figure3, in most cases, the CBIR precision reduces with an increasing number of color bins on the Corel-10K dataset.Figure 3
Performances of different bin numbers for the color and edge features on the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The GHIM-10K dataset differs because in many cases where the color bins increase, the CBIR precision increases. An exception is when there are 32 and 64 edge bins, as the precision increases when more color bins are added. Moreover, varying the edge bins for the first, second, and third integrations does not clearly increase or decrease the precision of the MIFH method.In Figure3, the precision of the MIFH method varies from approximately 49% to 53% on the Corel-10K dataset, whereas the precision for the GHIM-10K dataset varies from approximately 54% to 59%. The Corel-10K dataset is older and contains images with colors that are not as bright as the GHIM-10K dataset, which is relatively new. Color feature is an important factor in the proposed method, and the degree of color brightness affects the retrieval process for the proposed method. The precision change of the MIFH method does not present a clear trend for the two datasets. Thus, we reduce the vector dimensionality and set the number of color bins, the edge bins of the first integration, the edge bins of the second integration, and the edge bins of the third integration to 54, 16, 16, and 16 bins, respectively. In this case, the total vector dimensionality is 54 + 16+16 + 16 = 102 bins.


#### 4.3.2. Contribution of Edge and Color

Different experiments are implemented using various combinations of visual features to investigate the contributions of edges and colors within the MIFH method, where the quantization levels of the colors and edges are 54 and 16 bins, respectively.On the Corel-10K dataset, the precision and recall when only using the color feature are 46.48% and 5.58%, respectively. Using only the edge features of the first, second, and third integrations gives poor precision and recall. Compared with the edge features, the color features significantly increase the performance of the proposed method. If we combine these features, the performance significantly increases.Figure4 shows that the performances for both Corel-10K and GHIM-10K datasets when using color and edge features are better than using only the color features or edge features individually. The E1, E2, and E3 denote the edge features of the first, second, third integration, respectively. With more integrations, using the edge map for only a single integration does not improve the performance.Figure 4
Contribution of the edge, color, and the number of integrations, where there are 54 color bins and 16 bins for each of the edge maps in the first, second, third integrations. TheE1, E2, and E3 denote the edge features of the first, second, and third integrations, respectively. The data are for the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The repetition of the Sobel operation when calculating the edge gradient is an important factor in our method. Table1 shows that the precision of the MIFH method increases with increasing repetitions of the Sobel operation on the Corel-10K dataset. The GHIM-10K dataset differs; the repetitions of the Sobel operation have not clearly increased or decreased the precision of the MIFH method.Table 1
Repetitions of the Sobel operation when calculating the edge gradient.
Dataset
Performance (%)
Repetitions
1
2
3
4
5
Corel-10K
Precision
52.11
51.95
52.62
52.96
51.06
Recall
6.25
6.23
6.31
6.36
6.13
GHIM-10K
Precision
57.71
57.10
56.23
56.48
56.64
Recall
1.39
1.37
1.35
1.36
1.36The best precision for the MIFH method is achieved on Corel-10K datasets when the Sobel operation is repeated four times. With more repeated Sobel operations, the number of calculations also increase; thus, we set the final repetition of the Sobel operation to four in the proposed method.To investigate the performance of the MIFH method using different bias parameters in calculating the color difference map, sigmoid function sigmoid (.) was utilized as the active function. Various bias parameters can result in different color difference maps and may further lead to different retrieval performance. As can be seen from Table2, the best precision for the MIFH method is achieved on the Corel-10K dataset when the bias parameter b=1.0. On GHIM-10K dataset, the precisions of MIFH method do not have much difference between the bias parameter b=1.0 and b=2.0. Thus, the bias parameter b=1.0 was utilized in calculating the color difference map.Table 2
The bias parameter in calculating the sigmoid function sigmoid (.).
Dataset
Performance (%)
The bias parameter
0.0
1.0
2.0
3.0
4.0
Corel-10K
Precision
48.92
52.96
51.82
51.33
50.58
Recall
5.87
6.36
6.22
6.16
6.07
GHIM-10K
Precision
55.4
56.48
56.65
55.83
54.89
Recall
1.33
1.36
1.37
1.34
1.32


#### 4.3.3. Performance Comparisons

In image representation, the color and edge features are extracted using the MIFH method, and it is natural that the MTH method can represent color and edge orientation features as well. Therefore, it stands to reason that both the MIFH and MTH methods can represent color texture features. The LBPH method is a well-known texture features descriptor that can also represent color texture features by extracting the LBP features along the R, G, and B color channels simultaneously. The HOG method is a well-known descriptor that is widely used in object recognition [23]. The technique counts the occurrences of gradient orientations in localized portions of an image. The image retrieval performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets are shown in Table 3.Table 3
Performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets.
Dataset
Metric
Method
STH
CPV-THF
CMSD
BOW
LBPH
HOG
MTH
CVH
MIFH
Corel-10K
Precision
48.03
52.28
50.25
30.36
37.24
27.95
40.87
48.58
52.96
Recall
5.76
6.27
6.03
3.64
4.47
3.35
4.91
5.83
6.36
GHIM-10K
Precision
—
—
—
39.67
46.89
23.96
53.13
57.48
56.48
Recall
—
—
—
0.96
1.13
0.57
1.28
1.38
1.36
Corel-5K
Precision
60.28
63.90
63.14
—
43.82
35.75
49.98
60.13
60.16
Recall
7.23
7.66
7.57
—
5.26
4.29
6.00
7.21
7.21The performances of the MIFH method are better than those of the BOW, MTH, HOG, and LBPH based on the experimental data. The precision of the MIFH method is higher than the STH, CPV-THF, CMSD, BOW, LBPH, HOG, MTH, and CVH methods at 4.93%, 0.68%, 2.71%, 22.60%, 15.72%, 25.01%, 12.09%, and 4.38%, respectively, on the Corel-10K dataset. On the GHIM-10K dataset, the precision of the MIFH method is higher than the BOW, LBPH, HOG, and MTH methods at 16.81%, 9.59%, 32.52%, and 3.35%, respectively. Nevertheless, the MIFH method is lower than the CVH method. On the Corel-5K dataset, the precision of the MIFH method is higher than the MTH, HOG, and LBPH methods. Considering the discrimination power of the color texture features, we confirm that the MIFH method is better than the LBPH method.In order to determine the most suitable distance distances or similarity metrics, Canberra,x2 statistics, L1, histogram intersection, and Chebyshev distances were adopted in the CBIR experiments. As can be seen from Table 4, L1 distance gives much better results than other metrics such as Canberra, x2 statistics, Chebyshev, Cosine, and histogram intersection. The histogram intersection gives the worst results on the two datasets. L1 distance is the simplest distance and more suitable for large scale datasets; thus, we were adopted it as the final distance in the proposed MIFH method.Table 4
The performance of the proposed method according to different distance or similarity metrics.
Dataset
Performance (%)
Distance or similarity metrics
Canberra
x
2 statistics
L
1
Histogram intersection
Cos
Chebyshev
Corel-10K
Precision
44.63
45.42
52.96
24.03
48.61
35.16
Recall
5.35
5.45
6.36
2.88
5.83
4.22
GHIM-10K
Precision
50.33
51.4
56.48
31.01
54.87
39.98
Recall
1.21
1.23
1.36
0.74
1.32
0.96


## 4.1. Benchmark Datasets

The Corel image dataset is the most commonly used dataset to test the performance of the CBIR. In Corel image dataset, every category in the Corel gallery contains 100 images, on which digital zoom can be used to obtain various image sizes. To measure the image retrieval performance, we select categories to build the Corel-10K and Corel-5K datasets and used them in the CBIR, where Corel-5K contains 50 categories and 5,000 images, and Corel-10K dataset contains 100 categories and 10,000 images which contain all images of Corel-5K dataset.The third dataset is the GHIM-10K with 10,000 images. All images in this dataset are sourced from the web or a camera and were collected byGuang-Hai Liu. The database contains 20 categories, such as building, sunset, fish, flower, car, mountain, and tiger. In the GHIM-10K dataset, each category contains 500 images at 400 × 300 or 300 × 400 in JPEG format. Corel-10K, Corel-5K, and GHIM-10K datasets can be downloaded from http://www.ci.gxnu.edu.cn/cbir/Dataset.aspx.Three subsets, consisting of ten percent of each dataset were used as the query images. The performance is evaluated using the average results for each query in terms of precision and recall.


## 4.2. Performance Measurements

Selecting the metric for the performance evaluation is critical to compare CBIR experiments. In many CBIR cases [3–10], precision and recall metrics are popular ways to evaluate the performance of CBIR or other text retrieval techniques [46].On the Corel datasets, each category contains 100 images and is denoted ask=100, while k=500 for the GHIM-10K dataset. If there are n retrieved images and m images within the retrieved images are relevant to the query for the image retrieval, then the precision and recall metrics can be defined as follows [46]:(11)Precision=mn,Recall=mk.We setn=12 in our CBIR system, i.e., there are 12 retrieved images in the image retrieval.


## 4.3. Retrieval Performance

Feature selection is a very important issue in image representation. For example, edges and colors are widely used features in content-based image retrieval. In image representation, low-level features are combined into the proposed algorithm where quantization is used to extract the features. Various feature quantizations are important factors in CBIR experiments. Feature quantization has a greater contribution to the performance of content-based image retrieval because the proposed MIFH method is a histogram-based method. Thus, the number of edge bins and color feature bins can directly influence the retrieval results.


### 4.3.1. Investigation of Vector Dimensionality

To investigative the performance of the MIFH method using different histogram bins, we set the bin numbers of the color feature to 54, 72, 108, and 192 bins, respectively. For the first, second, and third integrations, the bins of the edge maps are set to 16, 32, and 64 bins, respectively. As shown in Figure3, in most cases, the CBIR precision reduces with an increasing number of color bins on the Corel-10K dataset.Figure 3
Performances of different bin numbers for the color and edge features on the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The GHIM-10K dataset differs because in many cases where the color bins increase, the CBIR precision increases. An exception is when there are 32 and 64 edge bins, as the precision increases when more color bins are added. Moreover, varying the edge bins for the first, second, and third integrations does not clearly increase or decrease the precision of the MIFH method.In Figure3, the precision of the MIFH method varies from approximately 49% to 53% on the Corel-10K dataset, whereas the precision for the GHIM-10K dataset varies from approximately 54% to 59%. The Corel-10K dataset is older and contains images with colors that are not as bright as the GHIM-10K dataset, which is relatively new. Color feature is an important factor in the proposed method, and the degree of color brightness affects the retrieval process for the proposed method. The precision change of the MIFH method does not present a clear trend for the two datasets. Thus, we reduce the vector dimensionality and set the number of color bins, the edge bins of the first integration, the edge bins of the second integration, and the edge bins of the third integration to 54, 16, 16, and 16 bins, respectively. In this case, the total vector dimensionality is 54 + 16+16 + 16 = 102 bins.


### 4.3.2. Contribution of Edge and Color

Different experiments are implemented using various combinations of visual features to investigate the contributions of edges and colors within the MIFH method, where the quantization levels of the colors and edges are 54 and 16 bins, respectively.On the Corel-10K dataset, the precision and recall when only using the color feature are 46.48% and 5.58%, respectively. Using only the edge features of the first, second, and third integrations gives poor precision and recall. Compared with the edge features, the color features significantly increase the performance of the proposed method. If we combine these features, the performance significantly increases.Figure4 shows that the performances for both Corel-10K and GHIM-10K datasets when using color and edge features are better than using only the color features or edge features individually. The E1, E2, and E3 denote the edge features of the first, second, third integration, respectively. With more integrations, using the edge map for only a single integration does not improve the performance.Figure 4
Contribution of the edge, color, and the number of integrations, where there are 54 color bins and 16 bins for each of the edge maps in the first, second, third integrations. TheE1, E2, and E3 denote the edge features of the first, second, and third integrations, respectively. The data are for the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The repetition of the Sobel operation when calculating the edge gradient is an important factor in our method. Table1 shows that the precision of the MIFH method increases with increasing repetitions of the Sobel operation on the Corel-10K dataset. The GHIM-10K dataset differs; the repetitions of the Sobel operation have not clearly increased or decreased the precision of the MIFH method.Table 1
Repetitions of the Sobel operation when calculating the edge gradient.
Dataset
Performance (%)
Repetitions
1
2
3
4
5
Corel-10K
Precision
52.11
51.95
52.62
52.96
51.06
Recall
6.25
6.23
6.31
6.36
6.13
GHIM-10K
Precision
57.71
57.10
56.23
56.48
56.64
Recall
1.39
1.37
1.35
1.36
1.36The best precision for the MIFH method is achieved on Corel-10K datasets when the Sobel operation is repeated four times. With more repeated Sobel operations, the number of calculations also increase; thus, we set the final repetition of the Sobel operation to four in the proposed method.To investigate the performance of the MIFH method using different bias parameters in calculating the color difference map, sigmoid function sigmoid (.) was utilized as the active function. Various bias parameters can result in different color difference maps and may further lead to different retrieval performance. As can be seen from Table2, the best precision for the MIFH method is achieved on the Corel-10K dataset when the bias parameter b=1.0. On GHIM-10K dataset, the precisions of MIFH method do not have much difference between the bias parameter b=1.0 and b=2.0. Thus, the bias parameter b=1.0 was utilized in calculating the color difference map.Table 2
The bias parameter in calculating the sigmoid function sigmoid (.).
Dataset
Performance (%)
The bias parameter
0.0
1.0
2.0
3.0
4.0
Corel-10K
Precision
48.92
52.96
51.82
51.33
50.58
Recall
5.87
6.36
6.22
6.16
6.07
GHIM-10K
Precision
55.4
56.48
56.65
55.83
54.89
Recall
1.33
1.36
1.37
1.34
1.32


### 4.3.3. Performance Comparisons

In image representation, the color and edge features are extracted using the MIFH method, and it is natural that the MTH method can represent color and edge orientation features as well. Therefore, it stands to reason that both the MIFH and MTH methods can represent color texture features. The LBPH method is a well-known texture features descriptor that can also represent color texture features by extracting the LBP features along the R, G, and B color channels simultaneously. The HOG method is a well-known descriptor that is widely used in object recognition [23]. The technique counts the occurrences of gradient orientations in localized portions of an image. The image retrieval performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets are shown in Table 3.Table 3
Performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets.
Dataset
Metric
Method
STH
CPV-THF
CMSD
BOW
LBPH
HOG
MTH
CVH
MIFH
Corel-10K
Precision
48.03
52.28
50.25
30.36
37.24
27.95
40.87
48.58
52.96
Recall
5.76
6.27
6.03
3.64
4.47
3.35
4.91
5.83
6.36
GHIM-10K
Precision
—
—
—
39.67
46.89
23.96
53.13
57.48
56.48
Recall
—
—
—
0.96
1.13
0.57
1.28
1.38
1.36
Corel-5K
Precision
60.28
63.90
63.14
—
43.82
35.75
49.98
60.13
60.16
Recall
7.23
7.66
7.57
—
5.26
4.29
6.00
7.21
7.21The performances of the MIFH method are better than those of the BOW, MTH, HOG, and LBPH based on the experimental data. The precision of the MIFH method is higher than the STH, CPV-THF, CMSD, BOW, LBPH, HOG, MTH, and CVH methods at 4.93%, 0.68%, 2.71%, 22.60%, 15.72%, 25.01%, 12.09%, and 4.38%, respectively, on the Corel-10K dataset. On the GHIM-10K dataset, the precision of the MIFH method is higher than the BOW, LBPH, HOG, and MTH methods at 16.81%, 9.59%, 32.52%, and 3.35%, respectively. Nevertheless, the MIFH method is lower than the CVH method. On the Corel-5K dataset, the precision of the MIFH method is higher than the MTH, HOG, and LBPH methods. Considering the discrimination power of the color texture features, we confirm that the MIFH method is better than the LBPH method.In order to determine the most suitable distance distances or similarity metrics, Canberra,x2 statistics, L1, histogram intersection, and Chebyshev distances were adopted in the CBIR experiments. As can be seen from Table 4, L1 distance gives much better results than other metrics such as Canberra, x2 statistics, Chebyshev, Cosine, and histogram intersection. The histogram intersection gives the worst results on the two datasets. L1 distance is the simplest distance and more suitable for large scale datasets; thus, we were adopted it as the final distance in the proposed MIFH method.Table 4
The performance of the proposed method according to different distance or similarity metrics.
Dataset
Performance (%)
Distance or similarity metrics
Canberra
x
2 statistics
L
1
Histogram intersection
Cos
Chebyshev
Corel-10K
Precision
44.63
45.42
52.96
24.03
48.61
35.16
Recall
5.35
5.45
6.36
2.88
5.83
4.22
GHIM-10K
Precision
50.33
51.4
56.48
31.01
54.87
39.98
Recall
1.21
1.23
1.36
0.74
1.32
0.96


## 4.3.1. Investigation of Vector Dimensionality

To investigative the performance of the MIFH method using different histogram bins, we set the bin numbers of the color feature to 54, 72, 108, and 192 bins, respectively. For the first, second, and third integrations, the bins of the edge maps are set to 16, 32, and 64 bins, respectively. As shown in Figure3, in most cases, the CBIR precision reduces with an increasing number of color bins on the Corel-10K dataset.Figure 3
Performances of different bin numbers for the color and edge features on the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The GHIM-10K dataset differs because in many cases where the color bins increase, the CBIR precision increases. An exception is when there are 32 and 64 edge bins, as the precision increases when more color bins are added. Moreover, varying the edge bins for the first, second, and third integrations does not clearly increase or decrease the precision of the MIFH method.In Figure3, the precision of the MIFH method varies from approximately 49% to 53% on the Corel-10K dataset, whereas the precision for the GHIM-10K dataset varies from approximately 54% to 59%. The Corel-10K dataset is older and contains images with colors that are not as bright as the GHIM-10K dataset, which is relatively new. Color feature is an important factor in the proposed method, and the degree of color brightness affects the retrieval process for the proposed method. The precision change of the MIFH method does not present a clear trend for the two datasets. Thus, we reduce the vector dimensionality and set the number of color bins, the edge bins of the first integration, the edge bins of the second integration, and the edge bins of the third integration to 54, 16, 16, and 16 bins, respectively. In this case, the total vector dimensionality is 54 + 16+16 + 16 = 102 bins.


## 4.3.2. Contribution of Edge and Color

Different experiments are implemented using various combinations of visual features to investigate the contributions of edges and colors within the MIFH method, where the quantization levels of the colors and edges are 54 and 16 bins, respectively.On the Corel-10K dataset, the precision and recall when only using the color feature are 46.48% and 5.58%, respectively. Using only the edge features of the first, second, and third integrations gives poor precision and recall. Compared with the edge features, the color features significantly increase the performance of the proposed method. If we combine these features, the performance significantly increases.Figure4 shows that the performances for both Corel-10K and GHIM-10K datasets when using color and edge features are better than using only the color features or edge features individually. The E1, E2, and E3 denote the edge features of the first, second, third integration, respectively. With more integrations, using the edge map for only a single integration does not improve the performance.Figure 4
Contribution of the edge, color, and the number of integrations, where there are 54 color bins and 16 bins for each of the edge maps in the first, second, third integrations. TheE1, E2, and E3 denote the edge features of the first, second, and third integrations, respectively. The data are for the (a) Corel-10K and (b) GHIM-10K datasets.
(a)
(b)The repetition of the Sobel operation when calculating the edge gradient is an important factor in our method. Table1 shows that the precision of the MIFH method increases with increasing repetitions of the Sobel operation on the Corel-10K dataset. The GHIM-10K dataset differs; the repetitions of the Sobel operation have not clearly increased or decreased the precision of the MIFH method.Table 1
Repetitions of the Sobel operation when calculating the edge gradient.
Dataset
Performance (%)
Repetitions
1
2
3
4
5
Corel-10K
Precision
52.11
51.95
52.62
52.96
51.06
Recall
6.25
6.23
6.31
6.36
6.13
GHIM-10K
Precision
57.71
57.10
56.23
56.48
56.64
Recall
1.39
1.37
1.35
1.36
1.36The best precision for the MIFH method is achieved on Corel-10K datasets when the Sobel operation is repeated four times. With more repeated Sobel operations, the number of calculations also increase; thus, we set the final repetition of the Sobel operation to four in the proposed method.To investigate the performance of the MIFH method using different bias parameters in calculating the color difference map, sigmoid function sigmoid (.) was utilized as the active function. Various bias parameters can result in different color difference maps and may further lead to different retrieval performance. As can be seen from Table2, the best precision for the MIFH method is achieved on the Corel-10K dataset when the bias parameter b=1.0. On GHIM-10K dataset, the precisions of MIFH method do not have much difference between the bias parameter b=1.0 and b=2.0. Thus, the bias parameter b=1.0 was utilized in calculating the color difference map.Table 2
The bias parameter in calculating the sigmoid function sigmoid (.).
Dataset
Performance (%)
The bias parameter
0.0
1.0
2.0
3.0
4.0
Corel-10K
Precision
48.92
52.96
51.82
51.33
50.58
Recall
5.87
6.36
6.22
6.16
6.07
GHIM-10K
Precision
55.4
56.48
56.65
55.83
54.89
Recall
1.33
1.36
1.37
1.34
1.32


## 4.3.3. Performance Comparisons

In image representation, the color and edge features are extracted using the MIFH method, and it is natural that the MTH method can represent color and edge orientation features as well. Therefore, it stands to reason that both the MIFH and MTH methods can represent color texture features. The LBPH method is a well-known texture features descriptor that can also represent color texture features by extracting the LBP features along the R, G, and B color channels simultaneously. The HOG method is a well-known descriptor that is widely used in object recognition [23]. The technique counts the occurrences of gradient orientations in localized portions of an image. The image retrieval performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets are shown in Table 3.Table 3
Performances of the compared methods on the Corel-10K, Corel-5K, and GHIM-10K datasets.
Dataset
Metric
Method
STH
CPV-THF
CMSD
BOW
LBPH
HOG
MTH
CVH
MIFH
Corel-10K
Precision
48.03
52.28
50.25
30.36
37.24
27.95
40.87
48.58
52.96
Recall
5.76
6.27
6.03
3.64
4.47
3.35
4.91
5.83
6.36
GHIM-10K
Precision
—
—
—
39.67
46.89
23.96
53.13
57.48
56.48
Recall
—
—
—
0.96
1.13
0.57
1.28
1.38
1.36
Corel-5K
Precision
60.28
63.90
63.14
—
43.82
35.75
49.98
60.13
60.16
Recall
7.23
7.66
7.57
—
5.26
4.29
6.00
7.21
7.21The performances of the MIFH method are better than those of the BOW, MTH, HOG, and LBPH based on the experimental data. The precision of the MIFH method is higher than the STH, CPV-THF, CMSD, BOW, LBPH, HOG, MTH, and CVH methods at 4.93%, 0.68%, 2.71%, 22.60%, 15.72%, 25.01%, 12.09%, and 4.38%, respectively, on the Corel-10K dataset. On the GHIM-10K dataset, the precision of the MIFH method is higher than the BOW, LBPH, HOG, and MTH methods at 16.81%, 9.59%, 32.52%, and 3.35%, respectively. Nevertheless, the MIFH method is lower than the CVH method. On the Corel-5K dataset, the precision of the MIFH method is higher than the MTH, HOG, and LBPH methods. Considering the discrimination power of the color texture features, we confirm that the MIFH method is better than the LBPH method.In order to determine the most suitable distance distances or similarity metrics, Canberra,x2 statistics, L1, histogram intersection, and Chebyshev distances were adopted in the CBIR experiments. As can be seen from Table 4, L1 distance gives much better results than other metrics such as Canberra, x2 statistics, Chebyshev, Cosine, and histogram intersection. The histogram intersection gives the worst results on the two datasets. L1 distance is the simplest distance and more suitable for large scale datasets; thus, we were adopted it as the final distance in the proposed MIFH method.Table 4
The performance of the proposed method according to different distance or similarity metrics.
Dataset
Performance (%)
Distance or similarity metrics
Canberra
x
2 statistics
L
1
Histogram intersection
Cos
Chebyshev
Corel-10K
Precision
44.63
45.42
52.96
24.03
48.61
35.16
Recall
5.35
5.45
6.36
2.88
5.83
4.22
GHIM-10K
Precision
50.33
51.4
56.48
31.01
54.87
39.98
Recall
1.21
1.23
1.36
0.74
1.32
0.96


## 5. Results and Discussion

Color and edge cues are among the visual search components for perception stimuli and can express meaningful characteristics in images or scenes. Combining the color and edge cues can help to describe image content. Many descriptors can be used to characterize the texture characteristics, where the local binary pattern (LBP) is a well-known texture descriptor that can represent the local structures of image, but it cannot combine color and edge cues well. The MTH method is our prior work and was developed for the traditional CBIR, and it can represent the frequency for color and edge orientation information using a special histogram. CPV-THF, STH, and CMSD methods were derived from our previous works (the multi-texton histogram and the micro-structure descriptor), but those extend methods have not extracted the color and edge features based on feature integration theory.The CVH method incorporates the advantages of histogram-based methods, as it takes the spatial information of neighboring pixels into account and has a performance that is similar with the MIFH method. It has the ability to discriminate color and edge features.The BOW and HOG methods do not achieve satisfactory results on either dataset for the content-based image retrieval. The HOG method is a well-known descriptor that was developed for object recognition. Object recognition and content-based image retrieval are two different applications of pattern recognition. The task of object-based image retrieval is finding images that contain the same object or scene as the query image, whereas the classic CBIR is finding images that contain similar content as the query image. In some cases, using an object recognition-based method for content-based image retrieval cannot obtain satisfactory results compared with the application of object recognition.The HOG method counts the occurrences of the gradient orientation to represent the image content. While the gradient orientation is a highly important cue in image representation, the discriminative power of only using the gradient orientation cue is a limiting factor in CBIR. In the BOW histogram, visual words obtained using the vector quantization of local features descriptors (e.g., Sift, SURF, ORB, and other local descriptors) require heavy computational burdens and may result in information loss.Color is the most basic quality of visual content, so it is expected that the MIFH method has the ability to discriminate color features. In addition, edge cues are also contained in the MIFH method. This approach should have the ability to represent the general shape and texture features normally. To illustrate the representational ability of the visual features, we show two retrieval examples for the MIFH method in Figures5(a) and 5(b). Both examples are used to show the visual effects of color and texture or other features, rather than showing whether the performances are good or not because not all images can return such good effects.Figure 5
Two retrieval examples on the Corel-10K and GHIM-10K datasets using the MIFH method. The queries are for (a) flower and (b) motorcycle. In the two examples, the top-left image is the query image, and the similar images include the query image itself in the performance metric in terms of precision and recall.
(a)
(b)In Figure5, most of the returned images are correctly retrieved and ranked within the top 12 images, and all of the top retrieved images exhibit a good match for both the shape and color texture to the queries. In addition, texture is a highly popular visual feature for CBIR that yields information regarding the spatial arrangement of the color or intensity of an image or region.


## 6. Conclusions

Feature extraction and representation is an important issue in CBIR and has a close relationship with human perception. Color and edge cues are visual search components for stimuli perception that can express meaningful characteristics of images or scenes. Representing image contents via extracting color and edge features based on feature integration theory is a challenging problem. To address this problem, a novel and highly simple but efficient representation based on the multi-integration features model, called the MIFH, is proposed for CBIR using the feature integration theory.The MIFH method provides powerful information in image representation using a set of simple integration processes. Such attributes improve the discriminative power of the MIFH method and enable a higher performance compared with the BOW, LBP, HOG, and MTH approaches.



---

*Source: 1461459-2020-03-09.xml*