# Image Retrieval Using the Intensity Variation Descriptor

Authors: Zhao Wei; Guang-Hai Liu
Journal: Mathematical Problems in Engineering
 (2020)

---

## Abstract

Variations between image pixel characteristics contain a wealth of information. Extraction of such cues can be used to describe image content. In this paper, we propose a novel descriptor, called the intensity variation descriptor (IVD), to represent variations in colour, edges, and intensity and apply it to image retrieval. The highlights of the proposed method are as follows. (1) The IVD combines the advantages of the HSV and RGB colour spaces. (2) It can simulate the lateral inhibition mechanism and orientation-selective mechanism to determine an optimal direction and spatial layout. (3) An extended weighted L1 distance metric is proposed to calculate the similarity of images. It does not require complex operations such as square or square root and leads to good performance. Comparative experiments on two Corel datasets containing 15,000 images show that the proposed method performs better than the SoC-GMM, CPV-THF, and STH methods and provides good matching of texture, colour, and shape.


---

## Body

## 1. Introduction

With the increasing number of images uploaded to the Internet, there is an urgent need to quickly and efficiently retrieve images from large-scale collections. This situation has triggered researchers to propose a variety of content-based image retrieval (CBIR) methods. Most of these methods first extract primary visual features such as colours, edges, and intensities. They then design feature models such as attention models or local pattern models to extract useful information, which is used to construct a feature vector histogram for image matching.In recent years, use of the computational visual attention model to extract visual features has become a popular method for image retrieval. Liu et al. proposed a bar-shaped structure for image content analysis that combines the visual attention and orientation-selective mechanisms [1]. The bar-shaped structures emphasise the salient structural information in the primary visual features, which significantly improves the image retrieval performance. However, it only takes into account the associations between the bar-shaped structures, whose three pixels are invariant in intensity. The multitrend structure descriptor defines three trends in local structure [2], which comprise the increasing, decreasing, and invariant trends of the three pixels according to the base of the microstructure descriptor [3]. The dLBP method [4] encodes the intensity variation of the three pixels into two bits, providing a total of eight kinds of variation. These methods have demonstrated excellent performance in image retrieval and object detection, but do not efficiently extract information on variations in colour, edges, and intensity. In order to efficiently extract these variations, we propose an intensity variation descriptor (IVD) based on dLBP [4], which can effectively combine the advantages of the HSV and RGB colour spaces and has the power to discriminate texture and spatial structure.The proposed method has the following highlights. (1) The IVD can combine the advantages of the HSV and RGB colour spaces, which is more efficient than using a single colour space. (2) It can simulate the lateral inhibition mechanism and orientation-selective mechanism to determine the optimal direction and spatial layout. (3) We propose an extended weighted L1 distance, called W1 distance, to calculate the similarity between images and provide good CBIR performance.The rest of this paper is organised as follows. In Section2, we briefly review related works. The proposed method is presented in Section 3. We conduct CBIR experiments in Section 4, while Section 5 concludes the paper.


## 2. Related Works

In the field of image retrieval, various methods have been proposed to extract image features. These can be divided into two categories according to the type of features: global feature methods and local feature methods. Global feature methods extract the colour, texture, shape, and other features as essential visual features. It is simple to calculate global features with low feature dimensions. The local features are extracted from the positions of the image that are not easy to change. Therefore, it is robust to occlusion, changes in illumination and background, and geometric transformation [5, 6]. However, local feature methods use algorithms that are usually too complex and produce high-dimensional feature vectors.The colour histogram is one of the most commonly used global feature extraction methods. It is robust to scale, rotation, and noise. However, it lacks a spatial layout and completely different images may have the same histogram distributions. Some methods have been proposed to address such problems, such as colour correlograms [7] and colour coherence vectors [8]. Texture is an image feature with a certain spatial structure that appears repeatedly. There are some well-known texture description methods, such as GLCM [9], LBP [10], Gabor [11], and wavelet [12]. Shape is also one of the most important features used to describe an image, and human beings can directly judge a category by its shape. Classical shape descriptors include Zernike moments [13], curvature scale space (CSS), and angular radial transform [14]. Among local feature methods, SIFT [15] is one of the most classical. It can achieve scale and rotation invariance by extracting certain points with invariant features in different scale spaces of an image. Due to the complexity of the algorithm and the high feature dimensions, some improved methods have been proposed, including SURF [16] and PCA-SIFT [17]. In addition, there are other local feature methods such as that of Harris and Stephens [18] and oriented FAST and rotated BRIEF (ORB) [19]. The BOW model [20] uses SIFT to extract the local features and then clusters them into visual words via a clustering algorithm. Finally, a dictionary composed of these words retrieves the corresponding results through similarity matching.In recent years, the use of a single feature for image retrieval has failed to meet the requirements of large-scale image datasets. Accordingly, researchers began to propose image retrieval methods based on multifeature fusion. The most common fusion method is the combination of colour and texture. For example, a microstructure descriptor represents image local features by computing edge orientation similarities and underlying colours [3]. The structure element descriptor (SED) [21] combines colour and texture features using five structure elements denoting five directions. Liu and Yang proposed the colour difference histogram to integrate colour and edge orientation information based on perceptually uniform colour differences and used it for image retrieval [22]. Recently, many image retrieval methods based on LBP variants have been proposed [23–26]. Discrete wavelet transform, LBP, and grey level co-occurrence matrixes can be used to exploit multiresolution analysis and to enhance image directional information [26–30]. Simulation of human perception and visual attention have been adopted in some multistage image retrieval frameworks [31–41]. The manifold ranking method has been used to match similar feature vectors [42, 43]. The WAS method [44] was proposed to integrate the information of adjacent images and compute their similarity. The SFW graph [35] is an efficient graph-matching method used to solve the pairwise image matching problem and reduce computational complexity.Recently, deep learning has also been developed in the field of image retrieval, especially with image retrieval methods based on the convolutional neural networks (CNN). CNN-based methods use a pretrained or fine-tuned convolutional neural network to extract global features from the fully connected layer or local features from the intermediate layer for image retrieval [45–48]. Deep learning methods can achieve excellent retrieval results, but achieving reasonable results requires a lot of effort, including (1) knowledge and experience of network architecture, (2) huge amounts of training data, and (3) much computer time.In this paper, we propose a simple yet efficient image retrieval method. Our method directly utilises low-level visual features to represent images and does not require complex operations such as modelling, training, clustering, and segmentation.


## 3. The Proposed Method

Color, intensity, and edges are considered as the primary visual features which are commonly used in CBIR. Here, we propose a novel descriptor, called the intensity variation descriptor (IVD), to represent variations in colour, edges, and intensity, and apply it to image retrieval. The flow diagram of the proposed method is shown in Figure1. The IVD is built based on the intensity variations using colour, edges, and intensity in a certain direction. With the intensity variation serving as a bridge, the IVD extracts feature by simulating the lateral inhibition mechanism and orientation-selective mechanism and it effectively integrates colour, edges, and intensity for image retrieval.Figure 1
Flow diagram of the proposed method.In the proposed method, firstly, the input image is converted to the HSV colour space from the RGB colour space, and the H, S, and V components are uniform quantized. Then, the R, G, and B components of each pixel in the RGB colour space are taken as inputs for the IVD, and the output intensity variations are used as the feature values of the corresponding HSV-quantized colours. In addition, the edges and intensity information are extracted from the V component and used for quantization. The IVD is used to describe the intensity variation of the quantized edges and intensities in a certain direction, which is selected by the proposed local direction detection method. Finally, the intensity variation values of colours, edges, and intensities are integrated into the feature vector. After similarity matching by W1 distance, the 12 images look like the query image most are returned.


### 3.1. The Intensity Variation Descriptor (IVD)

As mentioned before, there is a variety of pixel intensity variations. The bar-shaped structure considers the intensity invariant on three pixels [1]. The multitrend structure descriptor adds two situations, where the intensities of three pixels increase and decrease in turn from left to right based on the bar-shaped structure [1] and microstructure [3]. The LBP combines the situations of the invariant and increases into one situation, so there are four variations in LBP, as shown in Figure 2(c). The dLBP considers the intensity difference variation (IDV) based on LBP, as shown in Figure 2(d). Therefore, dLBP contains eight variations which come from four intensity variations (IVs) and two IDVs.Figure 2
Intensity variation and intensity difference variation. (a) Bar-shaped structure, (b) multitrend structure, (c) LBP, (d) dLBP, (e) the proposed IVD, and (f) legend.In order to efficiently extract more intensity variation information, an intensity variation descriptor (IVD) is proposed to extract intensity variation and intensity difference variation information based on the bar-shaped structure [1] and dLBP [4], as shown in Figure 2(e).Let there be three intensity values,I1, I2, and I3. We define the intensity difference between them as follows:(1)d1=I1−I2,d2=I2−I3.The weight of the intensity variation (IV) can be defined as follows:(2)Iv=23×sd1+sd2,sx=0x>01x<02x=0.The weight of the intensity difference variation (IDV) can be denoted as follows:(3)Idv=22−sd1−d2.After combining the weights of the IV and IDV, we define the intensity variation value as follows:(4)IVDI1,I2,I3=Iv×Idv.


### 3.2. Extraction of Visual Features

Human eyes are sensitive to colour and edges. Colour can provide rich information and is the most direct visual feature of images. Edges can represent the boundaries of image contents and textural structures. Therefore, colours and edges are utilised to represent image features in the IVD method.Both the HSV and Lab colour spaces imitate human colour perception well [22, 38]. The Lab colour space is popular in calculating colour differences for image representation [22] and saliency detection [38], while the HSV colour space is more suitable for extracting colour information based on colour quantization [1, 3, 49]. Therefore, we adopted the HSV colour space for extracting colour features with the proposed method, where the H, S, and V components are uniformly quantized into 6, 3, and 3 bins, respectively. This results in a total of 6 × 3 × 3 = 54 colour bins. We define the colour bins as a colour map Cx,y, where Cx,y=s, s∈0,1,…,NC−1, and NC=54.Sobel operators have good noise-suppression characteristics [50] and are simple to calculate. We used them to detect edge cues on the V component. We can obtain an edge amplitude map Ax,y and edge orientation map Ox,y by using uniform quantization, Ax,y=t, t∈0,1,…,NA−1, where NA=9, and Ox,y=u, u∈0,1,…,NO−1, where NO=36.TheV component is also utilised to represent intensity. An intensity map Vx,y can be obtained using the above operation. We defined it as Vx,y=v, v∈0,1,…,NV−1, where NV=16.


### 3.3. Local Direction Detection

In the IVD method, three intensity values are input to obtain the intensity variation. As shown in Figure3, in a 3×3 neighbourhood, the intensity values of three pixels in four directions can meet the requirements of IVD. Unfortunately, the calculation of IVD in four directions is too computationally intensive. Hubel and Wiesel showed that many simple cells in the visual cortex have an orientation-selective mechanism [51, 52], which makes them respond only to lines with a certain orientation [53]. Inspired by this view, we try to select three intensity values in a certain direction as the input of IVD, whose direction is the texture direction of the neighbourhood.Figure 3
Example of IVD with a 3 × 3 neighbourhood.Let anm×n region in an image and its grey value be gx,y. The grey values of pixels adjacent to gx,y in the 0°, 45°, 90°, and 135° directions are denoted as gx+1,y, gx+1,y−1, gx,y+1, and gx+1,y+1, respectively. Therefore, the average grey value difference of the local neighbourhood is denoted as follows:(5)Gα=∑x=0m−2∑y=0n−1gx,y−gx+1,ym−1×n,α=0°,∑x=0m−2∑y=1n−1gx,y−gx+1,y−1m−1×n−1,α=45°,∑x=0m−1∑y=0n−2gx,y−gx,y+1m×n−1,α=90°,∑x=0m−2∑y=0n−2gx,y−gx+1,y+1m−1×n−1,α=135°,where m and n are the width and height of the region, respectively. m−1×n, m−1×n−1, m×n−1, and m−1×n−1 are the numbers of pixel pairs in four directions, respectively. In this paper, we set m=n=3. In a certain direction, we consider Gα as a numerator and the direction perpendicular to Gα as a denominator. We define the ratio fα between them as follows:(6)fα=G0°G90°+1,α=0°,G45°G135°+1,α=45°,G90°G0°+1,α=90°,G135°G45°+1,α=135°,where adding one to the denominator prevents it becoming a zero value. The direction corresponding to the minimum of fα is considered the direction of the 3×3 region.Using the above implementation, we can determine the regional direction of the edge amplitude mapA, the edge orientation map O, and the intensity map V, where the intensity values of the three pixels in this direction are utilised as the input for the IVD.


### 3.4. Image Representation

In order to represent colour features efficiently, both HSV and RGB colour spaces are utilised to extract colour information. The pixels of theR, G, and B channels are utilised as the input values of the IVD, and the output of the IVD is utilised as the histogram values. The colour features are represented as follows:(7)HCCx,y=∑∑IVDRx,y,Gx,y,Bx,y.In the representation of edges and intensity features, we determine theα direction of a 3×3 region within the edge and intensity feature map, and then use the three intensity values in the α direction as the input of the IVD. Furthermore, we use the lateral inhibition formula to combine the spatial information output by the IVD [54].(8)R=IVD0α+∑i=18γ×IVDiα,where γ is the lateral inhibition coefficient, IVD0α is the output value of the IVD at the centre pixel, and IVDiα, i∈1,2,…,8 is the output value of the IVD about eight pixels adjacent to the central pixel. We define γ as follows:(9)γ=1,ifIVD0=IVDi,0.1,others.The histogram of the edge amplitudeA is defined as follows:(10)HAAx,y=∑∑19IVD0αA0α−1,A0α,A0α+1+∑i=18γ×IVDiαAiα−1,Aiα,Aiα+1,where A0α is the edge amplitude value of the centre pixel of the 3×3 region, A0α−1 and A0α+1 are the surrounding edge amplitude values of A0α in the α direction, Aiα are the edge amplitude values of the eight pixels adjacent to the central pixel A0α, and Aiα−1 and Aiα+1 are the surrounding edge amplitude values of Aiα in direction α.Similarly, the histograms of edge orientationO and intensity map V are defined as follows:(11)HOOx,y=∑∑19IVD0αO0α−1,O0α,O0α+1+∑i=18γ×IVDiαOiα−1,Oiα,Oiα+1,HVVx,y=∑∑19IVD0αV0α−1,V0α,V0α+1+∑i=18γ×IVDiαViα−1,Viα,Viα+1.CombiningHCCx,y, HAAx,y, HOOx,y, and HVVx,y, the final histograms H can be obtained:(12)H=HC0,HC1,…,HCNC−1,HA0,HA1,…,HANA−1,HO0,HO1,…,HONO−1,HV0,HV1,…,HVNV−1.


## 3.1. The Intensity Variation Descriptor (IVD)

As mentioned before, there is a variety of pixel intensity variations. The bar-shaped structure considers the intensity invariant on three pixels [1]. The multitrend structure descriptor adds two situations, where the intensities of three pixels increase and decrease in turn from left to right based on the bar-shaped structure [1] and microstructure [3]. The LBP combines the situations of the invariant and increases into one situation, so there are four variations in LBP, as shown in Figure 2(c). The dLBP considers the intensity difference variation (IDV) based on LBP, as shown in Figure 2(d). Therefore, dLBP contains eight variations which come from four intensity variations (IVs) and two IDVs.Figure 2
Intensity variation and intensity difference variation. (a) Bar-shaped structure, (b) multitrend structure, (c) LBP, (d) dLBP, (e) the proposed IVD, and (f) legend.In order to efficiently extract more intensity variation information, an intensity variation descriptor (IVD) is proposed to extract intensity variation and intensity difference variation information based on the bar-shaped structure [1] and dLBP [4], as shown in Figure 2(e).Let there be three intensity values,I1, I2, and I3. We define the intensity difference between them as follows:(1)d1=I1−I2,d2=I2−I3.The weight of the intensity variation (IV) can be defined as follows:(2)Iv=23×sd1+sd2,sx=0x>01x<02x=0.The weight of the intensity difference variation (IDV) can be denoted as follows:(3)Idv=22−sd1−d2.After combining the weights of the IV and IDV, we define the intensity variation value as follows:(4)IVDI1,I2,I3=Iv×Idv.


## 3.2. Extraction of Visual Features

Human eyes are sensitive to colour and edges. Colour can provide rich information and is the most direct visual feature of images. Edges can represent the boundaries of image contents and textural structures. Therefore, colours and edges are utilised to represent image features in the IVD method.Both the HSV and Lab colour spaces imitate human colour perception well [22, 38]. The Lab colour space is popular in calculating colour differences for image representation [22] and saliency detection [38], while the HSV colour space is more suitable for extracting colour information based on colour quantization [1, 3, 49]. Therefore, we adopted the HSV colour space for extracting colour features with the proposed method, where the H, S, and V components are uniformly quantized into 6, 3, and 3 bins, respectively. This results in a total of 6 × 3 × 3 = 54 colour bins. We define the colour bins as a colour map Cx,y, where Cx,y=s, s∈0,1,…,NC−1, and NC=54.Sobel operators have good noise-suppression characteristics [50] and are simple to calculate. We used them to detect edge cues on the V component. We can obtain an edge amplitude map Ax,y and edge orientation map Ox,y by using uniform quantization, Ax,y=t, t∈0,1,…,NA−1, where NA=9, and Ox,y=u, u∈0,1,…,NO−1, where NO=36.TheV component is also utilised to represent intensity. An intensity map Vx,y can be obtained using the above operation. We defined it as Vx,y=v, v∈0,1,…,NV−1, where NV=16.


## 3.3. Local Direction Detection

In the IVD method, three intensity values are input to obtain the intensity variation. As shown in Figure3, in a 3×3 neighbourhood, the intensity values of three pixels in four directions can meet the requirements of IVD. Unfortunately, the calculation of IVD in four directions is too computationally intensive. Hubel and Wiesel showed that many simple cells in the visual cortex have an orientation-selective mechanism [51, 52], which makes them respond only to lines with a certain orientation [53]. Inspired by this view, we try to select three intensity values in a certain direction as the input of IVD, whose direction is the texture direction of the neighbourhood.Figure 3
Example of IVD with a 3 × 3 neighbourhood.Let anm×n region in an image and its grey value be gx,y. The grey values of pixels adjacent to gx,y in the 0°, 45°, 90°, and 135° directions are denoted as gx+1,y, gx+1,y−1, gx,y+1, and gx+1,y+1, respectively. Therefore, the average grey value difference of the local neighbourhood is denoted as follows:(5)Gα=∑x=0m−2∑y=0n−1gx,y−gx+1,ym−1×n,α=0°,∑x=0m−2∑y=1n−1gx,y−gx+1,y−1m−1×n−1,α=45°,∑x=0m−1∑y=0n−2gx,y−gx,y+1m×n−1,α=90°,∑x=0m−2∑y=0n−2gx,y−gx+1,y+1m−1×n−1,α=135°,where m and n are the width and height of the region, respectively. m−1×n, m−1×n−1, m×n−1, and m−1×n−1 are the numbers of pixel pairs in four directions, respectively. In this paper, we set m=n=3. In a certain direction, we consider Gα as a numerator and the direction perpendicular to Gα as a denominator. We define the ratio fα between them as follows:(6)fα=G0°G90°+1,α=0°,G45°G135°+1,α=45°,G90°G0°+1,α=90°,G135°G45°+1,α=135°,where adding one to the denominator prevents it becoming a zero value. The direction corresponding to the minimum of fα is considered the direction of the 3×3 region.Using the above implementation, we can determine the regional direction of the edge amplitude mapA, the edge orientation map O, and the intensity map V, where the intensity values of the three pixels in this direction are utilised as the input for the IVD.


## 3.4. Image Representation

In order to represent colour features efficiently, both HSV and RGB colour spaces are utilised to extract colour information. The pixels of theR, G, and B channels are utilised as the input values of the IVD, and the output of the IVD is utilised as the histogram values. The colour features are represented as follows:(7)HCCx,y=∑∑IVDRx,y,Gx,y,Bx,y.In the representation of edges and intensity features, we determine theα direction of a 3×3 region within the edge and intensity feature map, and then use the three intensity values in the α direction as the input of the IVD. Furthermore, we use the lateral inhibition formula to combine the spatial information output by the IVD [54].(8)R=IVD0α+∑i=18γ×IVDiα,where γ is the lateral inhibition coefficient, IVD0α is the output value of the IVD at the centre pixel, and IVDiα, i∈1,2,…,8 is the output value of the IVD about eight pixels adjacent to the central pixel. We define γ as follows:(9)γ=1,ifIVD0=IVDi,0.1,others.The histogram of the edge amplitudeA is defined as follows:(10)HAAx,y=∑∑19IVD0αA0α−1,A0α,A0α+1+∑i=18γ×IVDiαAiα−1,Aiα,Aiα+1,where A0α is the edge amplitude value of the centre pixel of the 3×3 region, A0α−1 and A0α+1 are the surrounding edge amplitude values of A0α in the α direction, Aiα are the edge amplitude values of the eight pixels adjacent to the central pixel A0α, and Aiα−1 and Aiα+1 are the surrounding edge amplitude values of Aiα in direction α.Similarly, the histograms of edge orientationO and intensity map V are defined as follows:(11)HOOx,y=∑∑19IVD0αO0α−1,O0α,O0α+1+∑i=18γ×IVDiαOiα−1,Oiα,Oiα+1,HVVx,y=∑∑19IVD0αV0α−1,V0α,V0α+1+∑i=18γ×IVDiαViα−1,Viα,Viα+1.CombiningHCCx,y, HAAx,y, HOOx,y, and HVVx,y, the final histograms H can be obtained:(12)H=HC0,HC1,…,HCNC−1,HA0,HA1,…,HANA−1,HO0,HO1,…,HONO−1,HV0,HV1,…,HVNV−1.


## 4. Experimental Results

In order to validate the performance of the proposed method, some state-of-the-art methods were selected for comparison: SoC-GMM [55], LDP [56], SSH [1], CPV-THF [57], and STH [58]. A distance metric and benchmark dataset were required for the comparisons. In these experiments, we propose using an improved distance to evaluate performance based on the average results of each query in terms of precision, recall, and F-measure, respectively.


### 4.1. Datasets

Two datasets were used for CBIR. (1) One was the Corel-5K dataset, which contains 5000 images with diverse content such as bark, food textures, waves, microscopic objects, and trees. It contains 50 categories, each with 100 images sized 192 × 128 or 128 × 192 pixels in JPEG format. (2) The second was the Corel-10K dataset containing 10,000 images of the same size as those in the Corel-5K dataset. It has various contents such as flowers, horses, fish, beaches, and mountains. It contains 100 categories, with 100 images in each category. The Corel-5K dataset is a subset of the Corel-10K dataset.


### 4.2. Distance Metric

In the CBIR experiments, we propose an improved distance formula based on CDH [22], namely, the W1 distance, to implement image matching. It can be considered as an extended weighted L1 distance. Let T and Q be the feature vectors of template and query images, respectively. Both are K-dimensional feature vectors: T=T1,T2,…,TK and Q=Q1,Q2,…,QK. The W1 distance between the template and query images is simply calculated as follows:(13)DT,Q=∑i=1KTi−QiTi+Qi+w×uT,where uT=∑i=1KTi/K. The W1 distance has the best performance when w=0.01.


### 4.3. Performance Measures

In comparative CBIR experiments, deciding what kind of performance evaluation metric to use is important. In this paper, the precision, recall, andF-measure metrics were utilised to evaluate the effectiveness of the proposed method. They are defined as follows:(14)P=INN,R=INM,F=1+β2×R×Pβ2×P+R,where IN is the number of images retrieved in the top N positions that are similar to the query image, N is the total number of images retrieved, and M is the total number of images in the database that are similar to the query. Parameter β allows one to weight either precision or recall more heavily.In the Corel-5K and Corel-10K datasets, we setN=12, M=100, and β=1.


### 4.4. Retrieval Performance

The vector dimensionality has a very important impact on performance. In general, with the increase of dimensions, the retrieval performance of an algorithm will improve. However, exorbitant dimensions increase the computational burden. Therefore, choosing an appropriate vector dimensionality not only produces efficient results but also avoids excessive computational cost.In the CBIR experiments, hue (H) is uniformly quantified into 6, 8, and 12 bins, and Saturation (S) and Value (V) are both quantified as 3 bins in HSV colour space. Thus, the quantization numbers of colours are 6 × 3 × 3 = 54 bins, 8 × 3 × 3 = 72 bins, and 12 × 3 × 3 = 108 bins. At the same time, the quantization numbers of edge orientations are 9 bins, 18 bins, 36 bins, 45 bins, 60 bins, and 90 bins, and the quantization numbers of intensity are 16 bins, 32 bins, and 64 bins.As shown in Figure4, the precision did not change much as the quantization number of colours increased. Overall, precision declined with increases in the quantization number of intensity and increased with increases in the quantization number of edge orientation. When the quantization number of edge orientation increased from 18 to 36, the precision increased by more than 1%. When the quantization number of edge orientation was fixed to 36 bins, the quantization number of colour and greyscale were 54 bins and 16 bins, respectively, and the vector dimensionality was the smallest and the precision was the highest. Therefore, we set the quantization numbers of colours as 54 bins and 16 bins.Figure 4
Precision using various quantization numbers of colour, edge orientation, and intensity on the Corel-10K dataset: (a) the quantization number of greyscale is 16 bins, (b) the quantization number of greyscale is 32 bins, and (c) the quantization number of greyscale is 64 bins.
(a)
(b)
(c)We further evaluated the effects of the quantization numbers of edge amplitude. The quantization numbers of edge amplitude were 9 bins, 18 bins, 36 bins, 45 bins, 60 bins, and 90 bins. In Figure5, the precision, recall, and F-measures decrease with increases in the quantization number of edge amplitude. According to the results, we set the quantization number of edge amplitude to 9 bins. Ultimately, the vector dimensionality of the proposed method is 54 + 36 + 16 + 9 = 115 bins.Figure 5
Retrieval performance using edge amplitude quantization numbers with the Corel-10K dataset: (a) precision (%), (b) recall (%), and (c) F-measure (%).
(a)
(b)
(c)As shown in Table1, not using the colour information of the RGB colour space resulted in reductions in precision of at least 1% on the two datasets. If we use the lateral inhibition formula, the performance improved to a certain degree. Performance was better using the selected direction than using any single direction (0°, 45°, 90°, or 135°). Thus, using the selected direction and the lateral inhibition formula and adding the RGB colour space information, improved the performance.Table 1
Effectiveness of adding RGB colour space information and using the selected direction and lateral inhibition formula.
Dataset
Performance (%)
Default setting
Without RGB colour space information
Without lateral inhibition formula
Without selected direction
0°
45°
90°
135°
Corel-5K
Precision
66.91
65.50
66.50
66.39
66.08
66.54
66.09
Recall
8.03
7.86
7.98
7.97
7.93
7.98
7.93
F-measure
14.34
14.04
14.25
14.23
14.16
14.26
14.16
Corel-10K
Precision
56.88
55.66
56.43
56.41
56.03
56.43
56.04
Recall
6.83
6.68
6.77
6.77
6.72
6.77
6.72
F-measure
12.20
11.93
12.09
12.09
12.01
12.09
12.01In order to investigate the effectiveness of the proposed W1 distance method, we compared it with the L1 distance (Manhattan distance), the chi-square distance (χ2 statistics), the Canberra distance, the D1 distance (weighted L1 distance), and the proposed distance in CDH methods [22]. The performances of these distances or similarity metrics are listed in Table 2. It is clear that the W1 distance performed better than the other metrics on the Corel-5K and Corel-10K datasets. Moreover, the W1 distance metric does not require complex operations such as square or square root and can be regarded as an extended weighted L1 distance [22].Table 2
Performance comparison of the W1 distance and other metrics.
Dataset
Performance (%)
Distance or similarity metric
L1
Chi-square
Canberra
D1
CDH
STH
Proposed
Corel-5K
Precision
42.44
48.57
65.00
65.09
58.20
60.27
66.91
Recall
5.09
5.83
7.80
7.81
6.98
7.23
8.03
F-measure
9.09
10.41
13.93
13.95
12.47
12.91
14.34
Corel-10K
Precision
33.60
39.28
54.79
54.88
48.07
50.62
56.88
Recall
4.03
4.71
6.58
6.59
5.77
6.07
6.83
F-measure
7.20
8.42
11.74
11.76
10.30
10.85
12.20


### 4.5. Performance Comparisons

Here, we compare the proposed method with state-of-the-art methods such as SSH [1], SoC-GMM [55], LDP [56], CPV-THF [57], and STH [58]. The results are listed in Table 3. The precision of the proposed method on the Corel-5K dataset was 15.11%, 3.01%, and 6.63% higher than with SoC-GMM [55], CPV-THF [57], and STH [58], respectively. The precision of the proposed method on the Corel-10K dataset was 9.63%, 4.97%, 2%, 4.6%, and 8.85% higher than with SoC-GMM [55], LDP [56], SSH [1], CPV-THF [57], and STH [58], respectively.Table 3
Comparison of the proposed method with existing state-of-the-art methods.
Dataset
Performance (%)
Method
SoC-GMM
LDP
SSH
CPV-THF
STH
Proposed
Corel-5K
Precision
51.80
—
—
63.90
60.28
66.91
Recall
6.22
—
—
7.66
7.23
8.03
Corel-10K
Precision
47.25
51.91
54.88
52.28
48.03
56.88
Recall
5.67
6.23
6.58
6.27
5.76
6.83
Feature dimensions
79
134
130
242
172
115It has rich colours, complex textures, and various shapes within natural images. However, SoC-GMM only describes colour information. LDP, SSH, CPV-THF, and STH represent colour information in a colour space such as RGB, HSV, or Lab. However, both HSV and RGB colour spaces are considered in the proposed IVD method. It is worth mentioning that the IVD method does not increase the vector dimensionality. CPV-THF and STH can analyse texture features using texton templates. However, texton templates cannot adequately describe texture information because they only consider two pixels with the same intensity in2×2 square blocks. The bar-shaped structure of the SSH method only considers situations where three pixels are invariant in intensity [1]. The proposed IVD describes a total of 27 variations in the intensity of three pixels. The IVD method not only describes the intensity variation in a certain direction but can also combine the colour, edges, greyscale, and spatial information. Therefore, the proposed method contains richer information than the SoC-GMM, LDP, SSH, CPV-THF, and STH methods.Figures6 and 7 show two image retrieval examples from the Corel-5K and Corel-10K datasets. Each example has 12 images, with the top-left image being the query image. In Figure 6, the query is a gravel image, which has varied colours and complex textures. In Figure 7, the query is an eagle image, which has a background of blue sky and the shape of the eagle. It can be seen that all returned images were correctly ranked within the top 12 images. Furthermore, the colour, texture, and shape of the query image and all the returned images have certain similarities.Figure 6
Retrieval example of a gravel image from the Corel-5K dataset using the proposed IVD method.Figure 7
Retrieval example of an eagle image from the Corel-10K dataset using the proposed IVD method.Figures8 and 9 show two image retrieval examples from the Corel-5K and Corel-10K datasets using the SSH method [1], and the queries are also the gravel and an eagle. Seven returned images were incorrectly ranked within the top 12 images in Figure 8. Both the IVD and the SSH methods have discriminatory power for colour, texture, shape, and spatial features, but the IVD method has strong discriminatory power for texture and significantly outperforms better than that of the SSH method.Figure 8
Retrieval example of a gravel image from the Corel-5K dataset using the SSH method [1].Figure 9
Retrieval example of an eagle image from the Corel-10K dataset using the SSH method [1].


## 4.1. Datasets

Two datasets were used for CBIR. (1) One was the Corel-5K dataset, which contains 5000 images with diverse content such as bark, food textures, waves, microscopic objects, and trees. It contains 50 categories, each with 100 images sized 192 × 128 or 128 × 192 pixels in JPEG format. (2) The second was the Corel-10K dataset containing 10,000 images of the same size as those in the Corel-5K dataset. It has various contents such as flowers, horses, fish, beaches, and mountains. It contains 100 categories, with 100 images in each category. The Corel-5K dataset is a subset of the Corel-10K dataset.


## 4.2. Distance Metric

In the CBIR experiments, we propose an improved distance formula based on CDH [22], namely, the W1 distance, to implement image matching. It can be considered as an extended weighted L1 distance. Let T and Q be the feature vectors of template and query images, respectively. Both are K-dimensional feature vectors: T=T1,T2,…,TK and Q=Q1,Q2,…,QK. The W1 distance between the template and query images is simply calculated as follows:(13)DT,Q=∑i=1KTi−QiTi+Qi+w×uT,where uT=∑i=1KTi/K. The W1 distance has the best performance when w=0.01.


## 4.3. Performance Measures

In comparative CBIR experiments, deciding what kind of performance evaluation metric to use is important. In this paper, the precision, recall, andF-measure metrics were utilised to evaluate the effectiveness of the proposed method. They are defined as follows:(14)P=INN,R=INM,F=1+β2×R×Pβ2×P+R,where IN is the number of images retrieved in the top N positions that are similar to the query image, N is the total number of images retrieved, and M is the total number of images in the database that are similar to the query. Parameter β allows one to weight either precision or recall more heavily.In the Corel-5K and Corel-10K datasets, we setN=12, M=100, and β=1.


## 4.4. Retrieval Performance

The vector dimensionality has a very important impact on performance. In general, with the increase of dimensions, the retrieval performance of an algorithm will improve. However, exorbitant dimensions increase the computational burden. Therefore, choosing an appropriate vector dimensionality not only produces efficient results but also avoids excessive computational cost.In the CBIR experiments, hue (H) is uniformly quantified into 6, 8, and 12 bins, and Saturation (S) and Value (V) are both quantified as 3 bins in HSV colour space. Thus, the quantization numbers of colours are 6 × 3 × 3 = 54 bins, 8 × 3 × 3 = 72 bins, and 12 × 3 × 3 = 108 bins. At the same time, the quantization numbers of edge orientations are 9 bins, 18 bins, 36 bins, 45 bins, 60 bins, and 90 bins, and the quantization numbers of intensity are 16 bins, 32 bins, and 64 bins.As shown in Figure4, the precision did not change much as the quantization number of colours increased. Overall, precision declined with increases in the quantization number of intensity and increased with increases in the quantization number of edge orientation. When the quantization number of edge orientation increased from 18 to 36, the precision increased by more than 1%. When the quantization number of edge orientation was fixed to 36 bins, the quantization number of colour and greyscale were 54 bins and 16 bins, respectively, and the vector dimensionality was the smallest and the precision was the highest. Therefore, we set the quantization numbers of colours as 54 bins and 16 bins.Figure 4
Precision using various quantization numbers of colour, edge orientation, and intensity on the Corel-10K dataset: (a) the quantization number of greyscale is 16 bins, (b) the quantization number of greyscale is 32 bins, and (c) the quantization number of greyscale is 64 bins.
(a)
(b)
(c)We further evaluated the effects of the quantization numbers of edge amplitude. The quantization numbers of edge amplitude were 9 bins, 18 bins, 36 bins, 45 bins, 60 bins, and 90 bins. In Figure5, the precision, recall, and F-measures decrease with increases in the quantization number of edge amplitude. According to the results, we set the quantization number of edge amplitude to 9 bins. Ultimately, the vector dimensionality of the proposed method is 54 + 36 + 16 + 9 = 115 bins.Figure 5
Retrieval performance using edge amplitude quantization numbers with the Corel-10K dataset: (a) precision (%), (b) recall (%), and (c) F-measure (%).
(a)
(b)
(c)As shown in Table1, not using the colour information of the RGB colour space resulted in reductions in precision of at least 1% on the two datasets. If we use the lateral inhibition formula, the performance improved to a certain degree. Performance was better using the selected direction than using any single direction (0°, 45°, 90°, or 135°). Thus, using the selected direction and the lateral inhibition formula and adding the RGB colour space information, improved the performance.Table 1
Effectiveness of adding RGB colour space information and using the selected direction and lateral inhibition formula.
Dataset
Performance (%)
Default setting
Without RGB colour space information
Without lateral inhibition formula
Without selected direction
0°
45°
90°
135°
Corel-5K
Precision
66.91
65.50
66.50
66.39
66.08
66.54
66.09
Recall
8.03
7.86
7.98
7.97
7.93
7.98
7.93
F-measure
14.34
14.04
14.25
14.23
14.16
14.26
14.16
Corel-10K
Precision
56.88
55.66
56.43
56.41
56.03
56.43
56.04
Recall
6.83
6.68
6.77
6.77
6.72
6.77
6.72
F-measure
12.20
11.93
12.09
12.09
12.01
12.09
12.01In order to investigate the effectiveness of the proposed W1 distance method, we compared it with the L1 distance (Manhattan distance), the chi-square distance (χ2 statistics), the Canberra distance, the D1 distance (weighted L1 distance), and the proposed distance in CDH methods [22]. The performances of these distances or similarity metrics are listed in Table 2. It is clear that the W1 distance performed better than the other metrics on the Corel-5K and Corel-10K datasets. Moreover, the W1 distance metric does not require complex operations such as square or square root and can be regarded as an extended weighted L1 distance [22].Table 2
Performance comparison of the W1 distance and other metrics.
Dataset
Performance (%)
Distance or similarity metric
L1
Chi-square
Canberra
D1
CDH
STH
Proposed
Corel-5K
Precision
42.44
48.57
65.00
65.09
58.20
60.27
66.91
Recall
5.09
5.83
7.80
7.81
6.98
7.23
8.03
F-measure
9.09
10.41
13.93
13.95
12.47
12.91
14.34
Corel-10K
Precision
33.60
39.28
54.79
54.88
48.07
50.62
56.88
Recall
4.03
4.71
6.58
6.59
5.77
6.07
6.83
F-measure
7.20
8.42
11.74
11.76
10.30
10.85
12.20


## 4.5. Performance Comparisons

Here, we compare the proposed method with state-of-the-art methods such as SSH [1], SoC-GMM [55], LDP [56], CPV-THF [57], and STH [58]. The results are listed in Table 3. The precision of the proposed method on the Corel-5K dataset was 15.11%, 3.01%, and 6.63% higher than with SoC-GMM [55], CPV-THF [57], and STH [58], respectively. The precision of the proposed method on the Corel-10K dataset was 9.63%, 4.97%, 2%, 4.6%, and 8.85% higher than with SoC-GMM [55], LDP [56], SSH [1], CPV-THF [57], and STH [58], respectively.Table 3
Comparison of the proposed method with existing state-of-the-art methods.
Dataset
Performance (%)
Method
SoC-GMM
LDP
SSH
CPV-THF
STH
Proposed
Corel-5K
Precision
51.80
—
—
63.90
60.28
66.91
Recall
6.22
—
—
7.66
7.23
8.03
Corel-10K
Precision
47.25
51.91
54.88
52.28
48.03
56.88
Recall
5.67
6.23
6.58
6.27
5.76
6.83
Feature dimensions
79
134
130
242
172
115It has rich colours, complex textures, and various shapes within natural images. However, SoC-GMM only describes colour information. LDP, SSH, CPV-THF, and STH represent colour information in a colour space such as RGB, HSV, or Lab. However, both HSV and RGB colour spaces are considered in the proposed IVD method. It is worth mentioning that the IVD method does not increase the vector dimensionality. CPV-THF and STH can analyse texture features using texton templates. However, texton templates cannot adequately describe texture information because they only consider two pixels with the same intensity in2×2 square blocks. The bar-shaped structure of the SSH method only considers situations where three pixels are invariant in intensity [1]. The proposed IVD describes a total of 27 variations in the intensity of three pixels. The IVD method not only describes the intensity variation in a certain direction but can also combine the colour, edges, greyscale, and spatial information. Therefore, the proposed method contains richer information than the SoC-GMM, LDP, SSH, CPV-THF, and STH methods.Figures6 and 7 show two image retrieval examples from the Corel-5K and Corel-10K datasets. Each example has 12 images, with the top-left image being the query image. In Figure 6, the query is a gravel image, which has varied colours and complex textures. In Figure 7, the query is an eagle image, which has a background of blue sky and the shape of the eagle. It can be seen that all returned images were correctly ranked within the top 12 images. Furthermore, the colour, texture, and shape of the query image and all the returned images have certain similarities.Figure 6
Retrieval example of a gravel image from the Corel-5K dataset using the proposed IVD method.Figure 7
Retrieval example of an eagle image from the Corel-10K dataset using the proposed IVD method.Figures8 and 9 show two image retrieval examples from the Corel-5K and Corel-10K datasets using the SSH method [1], and the queries are also the gravel and an eagle. Seven returned images were incorrectly ranked within the top 12 images in Figure 8. Both the IVD and the SSH methods have discriminatory power for colour, texture, shape, and spatial features, but the IVD method has strong discriminatory power for texture and significantly outperforms better than that of the SSH method.Figure 8
Retrieval example of a gravel image from the Corel-5K dataset using the SSH method [1].Figure 9
Retrieval example of an eagle image from the Corel-10K dataset using the SSH method [1].


## 5. Conclusions

We have proposed a novel image representation, namely, the intensity variation descriptor, to represent image content, and applied it to image retrieval. The proposed descriptor not only extracts the richness of colour information by combining the HSV and RGB colour spaces but effectively describes texture features by extracting information on edges and intensity variations. It still considers the direction and spatial structure information of the textures by simulating the orientation-selection mechanism and the lateral inhibition mechanism.We have proposed an extended weighted L1 distance metric to improve the retrieval performance of the proposed method. Experimental performance comparisons with the Corel-5K and Corel-10K datasets demonstrate that our method outperforms some state-of-the-art methods and has the power to discriminate texture and spatial structure. There are some potential applications of the proposed IVD method, and it can be applicated to texture recognition, trademark image retrieval and palmprint image retrieval.



---

*Source: 6283987-2020-01-09.xml*