# Features Conduction Neural Response and Its Application in Content-Based Image Retrieval

Authors: Zhengfa Hu; Tian Yue; Haixia Xiao
Journal: Mathematical Problems in Engineering
 (2016)

---

## Abstract

A novel image representation is proposed for content-based image retrieval (CBIR). The core idea of the proposed method is to do deep learning for the local features of image and to melt semantic component into the representation through a hierarchical architecture which is built to simulate human visual perception system, and then a new image descriptor of features conduction neural response (FCNR) is constructed. Compared with the classical neural response (NR), FCNR has lower computational complexity and is more suitable for CBIR tasks. The results of experiments on a commonly used image database demonstrate that, compared with those of NR related methods or some other image descriptors that were originally developed for CBIR, the proposed method has wonderful performance on retrieval efficiency and effectiveness.


---

## Body

## 1. Introduction

Driven by the demand of search service market, the method of content-based image retrieval (CBIR) becomes a hot issue in the research field of pattern recognition and artificial intelligence for many years. The common ground for CBIR systems is to extract a signature for every image based on its pixel values and to define a rule for comparing images. The components of the signature are called features. An obvious advantage of a signature over the original pixel values is the significant compression of image representation. However, a more important reason for using the signature is to gain an improved correlation between image representation and image semantics. Actually, the main task of designing a signature is to bridge the gap between image semantics and the pixel representation, that is, to create a better correlation with image semantics [1].The researchers have tried to use machine learning techniques to derive the similarity measure of the high-level semantics of the image from the existing image representations [2] or to cluster the images by self-organizing maps firstly and then to do retrieval [3] with the former such as bandletized regions through support vector machines (BRSVM) learning and online multiple kernel similarity (OMKS) learning [4–6] and the latter such as tree structured self-organizing maps (TS-SOM) and growing hierarchical quadtree self-organizing map (GHSOQM) [7, 8]. These methods are often used in combination with relevance feedback technology, which can enhance the retrieval effectiveness to a certain extent [8, 9]. However, these methods are very technical and often need a lot of training time which makes them difficult to be applied in practice.On the other hand, the research of image representation for CBIR is constantly advancing, and many creative image representation methods are proposed. These methods can be broadly divided into two categories: the global feature based approach and the local feature based approach. For example, the edge histogram descriptor (EHD) [10], multiple texture histogram (MTH) [11], and color difference histogram (CDH) [12] are all based on the global characteristics of the algorithm. These algorithms to extract characteristics have good identification ability and robustness. However, we know that the overly complex feature representation is not always applicable to CBIR [1, 13]. At the same time, the local feature extraction method has also been a great concern [14, 15]. These methods focus on the feature representation of the image using the key points [16] or significant blocks in the image [17, 18]. How to determine the key points and the salient regions of the image are often dependent on the complex image segmentation technology. So far, however, the image segmentation technology is still one of the difficult problems in image processing and thus limits the application of these methods in CBIR.In recent years, the human visual cortex neural science and the related hierarchical learning methods provide a new direction for studying of this problem. Research has shown that the human visual perception system has very good abilities of learning and generalization through a few examples, and these abilities are given by the hierarchical structure of the visual cortex [19–21]. Based on the hierarchical structure of visual cortex, Smale et al. proposed the concept of derived kernel and the related theory of neural responses (NR) [22]. They established a mathematical model to simulate the process of hierarchical processing information of the human visual system. In the NR model, the inner product defined by the neural response led to a similarity measure between images which was called the derived kernel. Based on a hierarchical architecture, a recursive definition of the neural response and associated derived kernel was given. The derived kernel can be used in a variety of application domains such as classification of images, strings of text, and genomics data. Theoretical analysis and experimental results show that the NR model is an effective feature extraction method. It has the potential to be further improved and enhanced in many applications [17, 23–25]. Most important of all, the NR model has a key semantic component: a system of templates which can fuse the visual features and the semantic features of an image together and which is very important in CBIR.However, because of the underlying neural response using the pixel value of the bottom subblock of image and then being passed to the upper level of the subblock, this algorithm is not suitable for CBIR. Because, in the task of CBIR, the image databases are usually very large and the resolution of the image is usually very high, the exhaustion algorithm of pixel to pixel is difficult to bridge the “semantic gap” in complex scene images and a huge amount of computation also limits its application in practice. In order to capture the high-level semantic feature of the image and at the same time improve the efficiency of retrieval, we propose the concept and the corresponding algorithm of features conduction neural response (FCNR) on the basis of the related theory of NR.In the proposed method, we divide the spatial domain of an image in a simple way firstly and then obtain the local feature representation of the image by extracting the basic characteristics such as color, texture, and shape feature on the local area of the image. Next, we establish a hierarchical structure for the local feature representation of the image; at the same time, for each layer of the structure, a local feature template set is constructed. In the first layer of the hierarchical structure, local features are used to construct initial neural response and then these features are conducted to the senior subblocks layer by layer by the normalized inner product of the neural response. Finally, the image is expressed as a vector which is called FCNR, which can be used as an image representation for CBIR. The major advantages of the proposed method can be summarized as given here.(i)
The FCNR is derived from a local feature array rather than just using the pixel values, which overcome the drawback of overlearning problem in classical NR method.
(ii)
The high-level semantic component of the image is introduced into the feature representation by the interaction between the subblocks of the image and the templates in every layer of the constructed hierarchical structure.
(iii)
Without loss of the excellent identification and the invariance, the FCNR gets rid of the plight of the pixel to pixel exhaustion algorithm of the NR and reduces the computational complexity significantly, which is essential for the CBIR purpose.The rest of this paper is organized as follows. In Section2 the models of FCNR are constructed firstly. In Section 3 the image retrieval method based on FCNR is introduced. Then, in Section 4, we verify the effectiveness of the proposed method with extensive experiments on popular data sets and compare it with other CBIR methods. Finally, conclusions are drawn, and some future research issues are discussed in Section 5.


## 2. Feature Conduction Neural Response (FCNR)

The starting point of NR was to establish the mathematical model for visual mechanism of primate visual cortex [19, 26, 27]. In order to simulate the hierarchical information processing of visual cortex, Smale et al. [22] divided the image domain into some nested blocks as shown in Figure 1.Figure 1
Then layers of nested architectures.The NR of an image was defined in bottom-up fashion based on the hierarchical architecture. As a feature vector of an image, NR can be used to define the similarity between images. The theoretical analysis and experimental results show that the NR model has good performance on discrimination and it was robust to transformations, which suggested that the learning process of NR model possessed the characteristics of the human visual system in a certain degree.


### 2.1. Notation and Preliminaries

In this paper, we consider the case of a three-layer hierarchical architecture.As shown in Figure2, let regions u, v, and S
q in R
2  (u
⊂
v
⊂
S
q) be pieces of the domain on which the patches or subpatches of images are defined. In the vision interpretation, these regions can be considered as receptive fields with different sizes. When we are working with gray scale images, an image or an image patch can be seen as a discrete function of two variables which take the corresponding gray values as the functional values. That is to say, an image of size S
q can be seen as a function defined on the domain S
q. In this case, an image set consisting of the images defined on the domain S
q can be denoted by I
S
q. For description convenience, we denote the cardinality of a set A as A in the rest of this paper. Accordingly, the images in I
S
q can be denoted by f
1
,
f
2
,
…
,
f
I
S
q; that is, I
S
q
=
f
1
,
f
2
,
…
,
f
I
S
q. Similarly, the sets of image patches of size u and size v can be denoted by I
u and I
v with I
u
=
f
1
u
,
f
2
u
,
…
,
f
I
u
u and I
v
=
f
1
v
,
f
2
v
,
…
,
f
I
v
v, respectively.Figure 2
The three layers of nested architectures.As an example, Figure3 shows the nested architectures and the relationship of the image and the image patches. In Figure 3, f is a whole image of size 384
×
256, f
i
v
(
i
=
1,2
) are two examples of image patches of size 128
×
128 cut out from image f, and f
j
u
(
j
=
1,2
,
3,4
) are four examples of image patches of size 64
×
64 cut out from image f
i
v, respectively.Figure 3
The hierarchical relationship of image and image patches.The elementsh
i
v
(
i
=
1,2
) and h
j
u
(
j
=
1,2
,
3,4
) in Figure 3 which restrict an image or image patch to a specific subpatch are the other key ingredients in NR model: transformations associating two adjacent domains. Formally, the set H
u
=
h
1
u
,
h
2
u
,
…
,
h
H
u
u is called transformation set, in which the map h
u
:
u
→
v is a transformation from the smallest patch u to the next larger patch v. Similarly H
v
=
h
1
v
,
h
2
v
,
…
,
h
H
v
v with h
v
:
v
→
S
q can be defined. In this paper, the transformations are limited to translations and take the form h
(
x
)
=
x
+
a. Consequently, we can consider H
u as a set of translations corresponding to moving a sliding window of size u in patch v and similarly H
v as a set of translations corresponding to moving a sliding window of size v in patch S
q. For example, given an image of size M
×
N, if the step length equals one pixel, (
M
-
m
+
1
)
×
(
N
-
n
+
1
) image patches can be obtained by restricting the image on the given subpatch of size m
×
n.The following fundamental assumption related to image sets and transformation sets is supposed to be satisfied throughout this paper [22].Axiom 1.
Iff
v
∈
I
v and h
u
∈
H
u, then f
v
∘
h
u
∈
I
u, where f
v
∘
h
u denotes the restriction of image patch f
v on region u by transformation of h
u. Similarly, f
∘
h
v
∈
I
v if f
∈
I
S
q and h
v
∈
H
v.The last essential factor in NR model is series templates sets. The finite elementsT
u
∈
I
u are selected as the first-layer templates and the first-layer template set T
u
=
T
1
u
,
T
2
u
,
…
,
T
T
u
u is constituted. In the same way, the second-layer template set T
v
=
T
1
v
,
T
2
v
,
…
,
T
T
v
v can be obtained. Obviously, templates are some image patches which can be seen as image elements frequently encountered and serve as building blocks to represent other images. Those templates implicate abundant higher semantic information of the images which can be used to promote identification ability in image retrieval.


### 2.2. The Construction of FCNR

The first step of constructing FCNR is to segment the whole image in a simply way, which is different from other feature extraction methods based on region segmentation technology [8, 14], here just using the perpendicular line network to segment the image into some small rectangular area of the same size. Then in each small region we extract features such as color, texture, and shape, and all these characteristics are represented by a vector. So, an image can be represented as a three-dimension character array. On the basis of this three-dimension array, the local characteristics are conducted step by step to higher layer following the same mode of NR, and finally the FCNR can be obtained. Specific process is given below.For any imagef
⊂
I
S
q, we divide it into M
×
N rectangular blocks f
i
j
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N with the same size using perpendicular lines network; that is,(1)
f
=
f
11
f
12
⋯
f
1
N
⋮
⋮
⋮
⋮
f
M
1
f
M
2
⋯
f
M
N
.We extract some visual characteristics on each rectangular block in the same way. The details of features extraction methods will be presented in the third part of this paper. Normalizing the vectors with these characteristics as components and denoting them by w
i
j
(
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N
), we can get an array(2)
w
f
=
w
11
w
12
⋯
w
1
N
⋮
⋮
⋮
⋮
w
M
1
w
M
2
⋯
w
M
Nwhich is the local feature representation of the image f.It should be emphasized that thesew
i
j are all normalized vectors with the same dimension and each component of these vectors represents a feature of image block. An obvious advantage of normalization is said to be invariance to the brightness change of the image. If P characteristics are extracted from each rectangle image block f
i
j, then w
f is a three-dimensional array and can be simply represented as(3)
w
f
=
w
i
j
k
M
×
N
×
Pin which w
i
j
k
(
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N
;
k
=
1,2
,
…
,
P
) denotes the kth feature of the image block in ith row and jth column of the image f.In this circumstance, the related notations and their meanings introduced in the previous section should be adjusted accordingly. The set of local feature representations of the images in the setI
S
q is denoted by W
S
q
∗; that is,(4)
W
S
q
∗
=
w
f
1
,
w
f
2
,
…
,
w
f
I
S
q
,where S
q
∗ denotes the area of size M
×
N. Accordingly, we use W
u
=
w
1
u
,
w
2
u
,
…
,
w
W
u
u and W
v
=
w
1
v
,
w
2
v
,
…
,
w
W
v
v to denote the sets of patches of size u and size v, respectively. It is needed to emphasize that the elements of previous W
u and W
v are obtained by sampling from the rows and columns of the array w
f by moving windows rather than directly sampling from the image f.For example, assume thatf is an image of size 256
×
384. It is divided into 8
×
8 square subblocks and we extract 6 characteristics from each subblock. At this point, the local feature representation w
f is a three-dimensional array of size 32
×
48
×
6 and the size of S
q
∗ is 32
×
48. If u-size is 15
×
15 and v-size is 21
×
21, then w
u
∈
W
u and w
v
∈
W
v are three-dimensional arrays of size 15
×
15
×
6 and size 21
×
21
×
6, respectively.The notationsH
u and H
v still denote the transformation sets of the transformations from u to v and v to S
q
∗. The template sets T
u
⊂
W
u and T
v
⊂
W
v are obtained from w
f in a similar way by moving the window on v and S
q
∗, respectively. The elements in these template sets denoted by t
i
u
(
i
=
1,2
,
…
,
|
T
u
|
) and t
j
v
(
j
=
1,2
,
…
,
|
T
v
|
) are also some three-dimensional arrays.Now we can define the feature conduction neural response. Firstly, assumew
v
∈
W
v and, for any h
u
∈
H
u, we have w
v
∘
h
u
∈
W
u according to Axiom 1. Taking a template t
u
∈
T
u, we call(5)
N
v
w
v
t
u
=
max
h
u
∈
H
u
⁡
∑
i
,
j
,
k
w
v
∘
h
u
·
t
u
i
j
kneural responses of w
v to the template t
u, where (
w
v
∘
h
u
)
·
t
u denotes a three-dimensional array that is obtained by multiplying the corresponding elements of two three-dimensional arrays (
w
v
∘
h
u
) and t
u, and (
(
w
v
∘
h
u
)
·
t
u
)
i
j
k represents the element of (
w
v
∘
h
u
)
·
t
u in ith row and jth column of pth page. When t
u take over the template set T
u, we can get a |
T
u
| dimensional vector(6)
N
v
w
v
=
N
v
w
v
t
1
u
,
…
,
N
v
w
v
t
T
u
uwhich is called the first layer of neural response of w
v to the template set T
u. After normalization, it is denoted as N
^
v
(
w
v
); that is,(7)
N
^
v
w
v
=
N
v
w
v
N
v
w
v
,
N
v
w
v
,where ·
,
· is inner product of two vectors in the usual sense.Next, setw
f
∈
W
S
q
∗, and, according to Axiom 1, we know that w
f
∘
h
v
∈
W
v. For any template t
v
∈
T
v, we call(8)
N
S
q
∗
w
f
t
v
=
max
h
v
∈
H
v
⁡
N
^
v
w
f
∘
h
v
,
N
^
v
t
vneural responses of w
f to the template t
v. When t
v take over the template set T
v, we can get a |
T
v
| dimensional vector(9)
N
S
q
∗
w
f
=
N
S
q
∗
w
f
t
1
v
,
…
,
N
S
q
∗
w
f
t
T
v
vwhich is called the second layer of neural response of w
f to the template set T
v.Finally, for any imagef
∈
I
S
q, we define N
S
q
∗
(
w
f
) as features conduction neural response (FCNR) of the image f
∈
I
S
q and it is denoted by N
(
f
); that is,(10)
N
f
=
N
S
q
∗
w
f
=
N
S
q
∗
w
f
t
1
v
,
…
,
N
S
q
∗
w
f
t
T
v
v
.We add some remarks.(i)
The FCNR of an image is a vector whose dimension is equal to the number of templates in the second layer and has nothing to do with the dimension of the image itself. Therefore, in the process of image processing, we can transform all the images into vectors with the same dimension, regardless of the idea that the sizes of the images are the same or not.
(ii)
Due to the use of image low-level visual features in the underlying layer, FCNR model effectively overcomes the shortcomings of pixel to pixel exhaustion algorithm of the NR model. At the same time, the low-level visual features of image are conducted to the upper layer by the interaction between the subblocks of the image and the templates in every layer of the constructed hierarchical structure and make the FCNR contain high-level semantic elements of the image and this is very important in the task of CBIR.
(iii)
From the perspective of learning theory, the feature extraction method of the FCNR belongs to the category of unsupervised learning [2, 19, 22], and the hierarchical structure is introduced to do deep learning for the low-level visual features.


### 2.3. Computational Complexity Analysis

In image retrieval task, we often have the real-time requirements. As a result, the complexity of the algorithm is very important when constructing the feature representation for CBIR. Here we analyze the computational complexity of the proposed method in this paper.Consider the case of then layers hierarchical architecture as shown in Figure 1. We define a set of global transformations where the range is always the entire image domain S
q rather than the next larger patch recursively setting (11)
H
m
g
=
h
:
v
m
→
S
q
∣
h
=
h
′
∘
h
′
′
,
with
h
′
∈
H
m
+
1
g
,
h
′
′
∈
H
m
,for any 1
≤
m
≤
n
-
1, where H
n
g contains only the identity {
I
:
S
q
→
S
q
}. In the above formula, we denote the transformation set from patch of mth layer to the next larger patch by H
m.We denote the template in themth layer by T
m and ignoring the cost of normalization and of precomputing the neural responses of the templates, the number of required operations to export the NR is given by(12)
τ
=
∑
m
=
1
n
-
1
H
m
g
T
m
T
m
-
1
+
H
m
+
1
g
H
m
T
m
,where we denote for notational convenience the cost of computing the initial kernel by T
0 [22].Because the image is preprocessed,|
H
m
g
| in (12) in the calculation of the FCNR will be less than that in the calculation of the NR. This will eventually lead to the fact that τ of FCNR is far less than that of NR. In order to illustrate this point intuitively, we give a specific example.Supposef is an original image of size 256
×
384. In the calculation of the NR of f, we take the u-size as 112
×
112 pixels and the v-size as 172
×
172 pixels. On the other hand, the image f will be divided into the square subblock of size 16
×
16 and 14 features will be extracted from each block before the calculation of the FCNR of f. In this case, we take the u-size as 7
×
7 and the v-size as 11
×
11 which correspond to the 112
×
112 pixels and the 172
×
172 pixels in the original image. We also assume that the number of templates selected in each layer of the two methods is equal. Note that n
=
3 in this paper; we can calculate τ of NR and FCNR using (12), respectively. The values of the parameters in (12) and the results τ of NR and FCNR are listed in Table 1.Table 1
FCNR and NR are compared in terms of computational complexity.
H
1
g
H
2
g
H
3
g
H
1
H
2
T
0
T
1
T
2
τ
NR
39585
18318
1
3721
18318
19246
500
300
4.18
×
1
0
11
FCNR
180
84
1
25
84
686
500
300
7.54
×
1
0
7From Table1, we can see that the number of operations of FCNR is less than one five-thousandth of the number of operations of NR. This means that the computational complexity of NR is much higher than that of FCNR. In fact, for the image with high resolution, it is not practical to directly calculate the NR. Usually, we will do some simple preprocessing for image before calculating its NR. Therefore, the difference of computation is not so great in practice (see Section 4).


## 2.1. Notation and Preliminaries

In this paper, we consider the case of a three-layer hierarchical architecture.As shown in Figure2, let regions u, v, and S
q in R
2  (u
⊂
v
⊂
S
q) be pieces of the domain on which the patches or subpatches of images are defined. In the vision interpretation, these regions can be considered as receptive fields with different sizes. When we are working with gray scale images, an image or an image patch can be seen as a discrete function of two variables which take the corresponding gray values as the functional values. That is to say, an image of size S
q can be seen as a function defined on the domain S
q. In this case, an image set consisting of the images defined on the domain S
q can be denoted by I
S
q. For description convenience, we denote the cardinality of a set A as A in the rest of this paper. Accordingly, the images in I
S
q can be denoted by f
1
,
f
2
,
…
,
f
I
S
q; that is, I
S
q
=
f
1
,
f
2
,
…
,
f
I
S
q. Similarly, the sets of image patches of size u and size v can be denoted by I
u and I
v with I
u
=
f
1
u
,
f
2
u
,
…
,
f
I
u
u and I
v
=
f
1
v
,
f
2
v
,
…
,
f
I
v
v, respectively.Figure 2
The three layers of nested architectures.As an example, Figure3 shows the nested architectures and the relationship of the image and the image patches. In Figure 3, f is a whole image of size 384
×
256, f
i
v
(
i
=
1,2
) are two examples of image patches of size 128
×
128 cut out from image f, and f
j
u
(
j
=
1,2
,
3,4
) are four examples of image patches of size 64
×
64 cut out from image f
i
v, respectively.Figure 3
The hierarchical relationship of image and image patches.The elementsh
i
v
(
i
=
1,2
) and h
j
u
(
j
=
1,2
,
3,4
) in Figure 3 which restrict an image or image patch to a specific subpatch are the other key ingredients in NR model: transformations associating two adjacent domains. Formally, the set H
u
=
h
1
u
,
h
2
u
,
…
,
h
H
u
u is called transformation set, in which the map h
u
:
u
→
v is a transformation from the smallest patch u to the next larger patch v. Similarly H
v
=
h
1
v
,
h
2
v
,
…
,
h
H
v
v with h
v
:
v
→
S
q can be defined. In this paper, the transformations are limited to translations and take the form h
(
x
)
=
x
+
a. Consequently, we can consider H
u as a set of translations corresponding to moving a sliding window of size u in patch v and similarly H
v as a set of translations corresponding to moving a sliding window of size v in patch S
q. For example, given an image of size M
×
N, if the step length equals one pixel, (
M
-
m
+
1
)
×
(
N
-
n
+
1
) image patches can be obtained by restricting the image on the given subpatch of size m
×
n.The following fundamental assumption related to image sets and transformation sets is supposed to be satisfied throughout this paper [22].Axiom 1.
Iff
v
∈
I
v and h
u
∈
H
u, then f
v
∘
h
u
∈
I
u, where f
v
∘
h
u denotes the restriction of image patch f
v on region u by transformation of h
u. Similarly, f
∘
h
v
∈
I
v if f
∈
I
S
q and h
v
∈
H
v.The last essential factor in NR model is series templates sets. The finite elementsT
u
∈
I
u are selected as the first-layer templates and the first-layer template set T
u
=
T
1
u
,
T
2
u
,
…
,
T
T
u
u is constituted. In the same way, the second-layer template set T
v
=
T
1
v
,
T
2
v
,
…
,
T
T
v
v can be obtained. Obviously, templates are some image patches which can be seen as image elements frequently encountered and serve as building blocks to represent other images. Those templates implicate abundant higher semantic information of the images which can be used to promote identification ability in image retrieval.


## 2.2. The Construction of FCNR

The first step of constructing FCNR is to segment the whole image in a simply way, which is different from other feature extraction methods based on region segmentation technology [8, 14], here just using the perpendicular line network to segment the image into some small rectangular area of the same size. Then in each small region we extract features such as color, texture, and shape, and all these characteristics are represented by a vector. So, an image can be represented as a three-dimension character array. On the basis of this three-dimension array, the local characteristics are conducted step by step to higher layer following the same mode of NR, and finally the FCNR can be obtained. Specific process is given below.For any imagef
⊂
I
S
q, we divide it into M
×
N rectangular blocks f
i
j
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N with the same size using perpendicular lines network; that is,(1)
f
=
f
11
f
12
⋯
f
1
N
⋮
⋮
⋮
⋮
f
M
1
f
M
2
⋯
f
M
N
.We extract some visual characteristics on each rectangular block in the same way. The details of features extraction methods will be presented in the third part of this paper. Normalizing the vectors with these characteristics as components and denoting them by w
i
j
(
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N
), we can get an array(2)
w
f
=
w
11
w
12
⋯
w
1
N
⋮
⋮
⋮
⋮
w
M
1
w
M
2
⋯
w
M
Nwhich is the local feature representation of the image f.It should be emphasized that thesew
i
j are all normalized vectors with the same dimension and each component of these vectors represents a feature of image block. An obvious advantage of normalization is said to be invariance to the brightness change of the image. If P characteristics are extracted from each rectangle image block f
i
j, then w
f is a three-dimensional array and can be simply represented as(3)
w
f
=
w
i
j
k
M
×
N
×
Pin which w
i
j
k
(
i
=
1,2
,
…
,
M
;
j
=
1,2
,
…
,
N
;
k
=
1,2
,
…
,
P
) denotes the kth feature of the image block in ith row and jth column of the image f.In this circumstance, the related notations and their meanings introduced in the previous section should be adjusted accordingly. The set of local feature representations of the images in the setI
S
q is denoted by W
S
q
∗; that is,(4)
W
S
q
∗
=
w
f
1
,
w
f
2
,
…
,
w
f
I
S
q
,where S
q
∗ denotes the area of size M
×
N. Accordingly, we use W
u
=
w
1
u
,
w
2
u
,
…
,
w
W
u
u and W
v
=
w
1
v
,
w
2
v
,
…
,
w
W
v
v to denote the sets of patches of size u and size v, respectively. It is needed to emphasize that the elements of previous W
u and W
v are obtained by sampling from the rows and columns of the array w
f by moving windows rather than directly sampling from the image f.For example, assume thatf is an image of size 256
×
384. It is divided into 8
×
8 square subblocks and we extract 6 characteristics from each subblock. At this point, the local feature representation w
f is a three-dimensional array of size 32
×
48
×
6 and the size of S
q
∗ is 32
×
48. If u-size is 15
×
15 and v-size is 21
×
21, then w
u
∈
W
u and w
v
∈
W
v are three-dimensional arrays of size 15
×
15
×
6 and size 21
×
21
×
6, respectively.The notationsH
u and H
v still denote the transformation sets of the transformations from u to v and v to S
q
∗. The template sets T
u
⊂
W
u and T
v
⊂
W
v are obtained from w
f in a similar way by moving the window on v and S
q
∗, respectively. The elements in these template sets denoted by t
i
u
(
i
=
1,2
,
…
,
|
T
u
|
) and t
j
v
(
j
=
1,2
,
…
,
|
T
v
|
) are also some three-dimensional arrays.Now we can define the feature conduction neural response. Firstly, assumew
v
∈
W
v and, for any h
u
∈
H
u, we have w
v
∘
h
u
∈
W
u according to Axiom 1. Taking a template t
u
∈
T
u, we call(5)
N
v
w
v
t
u
=
max
h
u
∈
H
u
⁡
∑
i
,
j
,
k
w
v
∘
h
u
·
t
u
i
j
kneural responses of w
v to the template t
u, where (
w
v
∘
h
u
)
·
t
u denotes a three-dimensional array that is obtained by multiplying the corresponding elements of two three-dimensional arrays (
w
v
∘
h
u
) and t
u, and (
(
w
v
∘
h
u
)
·
t
u
)
i
j
k represents the element of (
w
v
∘
h
u
)
·
t
u in ith row and jth column of pth page. When t
u take over the template set T
u, we can get a |
T
u
| dimensional vector(6)
N
v
w
v
=
N
v
w
v
t
1
u
,
…
,
N
v
w
v
t
T
u
uwhich is called the first layer of neural response of w
v to the template set T
u. After normalization, it is denoted as N
^
v
(
w
v
); that is,(7)
N
^
v
w
v
=
N
v
w
v
N
v
w
v
,
N
v
w
v
,where ·
,
· is inner product of two vectors in the usual sense.Next, setw
f
∈
W
S
q
∗, and, according to Axiom 1, we know that w
f
∘
h
v
∈
W
v. For any template t
v
∈
T
v, we call(8)
N
S
q
∗
w
f
t
v
=
max
h
v
∈
H
v
⁡
N
^
v
w
f
∘
h
v
,
N
^
v
t
vneural responses of w
f to the template t
v. When t
v take over the template set T
v, we can get a |
T
v
| dimensional vector(9)
N
S
q
∗
w
f
=
N
S
q
∗
w
f
t
1
v
,
…
,
N
S
q
∗
w
f
t
T
v
vwhich is called the second layer of neural response of w
f to the template set T
v.Finally, for any imagef
∈
I
S
q, we define N
S
q
∗
(
w
f
) as features conduction neural response (FCNR) of the image f
∈
I
S
q and it is denoted by N
(
f
); that is,(10)
N
f
=
N
S
q
∗
w
f
=
N
S
q
∗
w
f
t
1
v
,
…
,
N
S
q
∗
w
f
t
T
v
v
.We add some remarks.(i)
The FCNR of an image is a vector whose dimension is equal to the number of templates in the second layer and has nothing to do with the dimension of the image itself. Therefore, in the process of image processing, we can transform all the images into vectors with the same dimension, regardless of the idea that the sizes of the images are the same or not.
(ii)
Due to the use of image low-level visual features in the underlying layer, FCNR model effectively overcomes the shortcomings of pixel to pixel exhaustion algorithm of the NR model. At the same time, the low-level visual features of image are conducted to the upper layer by the interaction between the subblocks of the image and the templates in every layer of the constructed hierarchical structure and make the FCNR contain high-level semantic elements of the image and this is very important in the task of CBIR.
(iii)
From the perspective of learning theory, the feature extraction method of the FCNR belongs to the category of unsupervised learning [2, 19, 22], and the hierarchical structure is introduced to do deep learning for the low-level visual features.


## 2.3. Computational Complexity Analysis

In image retrieval task, we often have the real-time requirements. As a result, the complexity of the algorithm is very important when constructing the feature representation for CBIR. Here we analyze the computational complexity of the proposed method in this paper.Consider the case of then layers hierarchical architecture as shown in Figure 1. We define a set of global transformations where the range is always the entire image domain S
q rather than the next larger patch recursively setting (11)
H
m
g
=
h
:
v
m
→
S
q
∣
h
=
h
′
∘
h
′
′
,
with
h
′
∈
H
m
+
1
g
,
h
′
′
∈
H
m
,for any 1
≤
m
≤
n
-
1, where H
n
g contains only the identity {
I
:
S
q
→
S
q
}. In the above formula, we denote the transformation set from patch of mth layer to the next larger patch by H
m.We denote the template in themth layer by T
m and ignoring the cost of normalization and of precomputing the neural responses of the templates, the number of required operations to export the NR is given by(12)
τ
=
∑
m
=
1
n
-
1
H
m
g
T
m
T
m
-
1
+
H
m
+
1
g
H
m
T
m
,where we denote for notational convenience the cost of computing the initial kernel by T
0 [22].Because the image is preprocessed,|
H
m
g
| in (12) in the calculation of the FCNR will be less than that in the calculation of the NR. This will eventually lead to the fact that τ of FCNR is far less than that of NR. In order to illustrate this point intuitively, we give a specific example.Supposef is an original image of size 256
×
384. In the calculation of the NR of f, we take the u-size as 112
×
112 pixels and the v-size as 172
×
172 pixels. On the other hand, the image f will be divided into the square subblock of size 16
×
16 and 14 features will be extracted from each block before the calculation of the FCNR of f. In this case, we take the u-size as 7
×
7 and the v-size as 11
×
11 which correspond to the 112
×
112 pixels and the 172
×
172 pixels in the original image. We also assume that the number of templates selected in each layer of the two methods is equal. Note that n
=
3 in this paper; we can calculate τ of NR and FCNR using (12), respectively. The values of the parameters in (12) and the results τ of NR and FCNR are listed in Table 1.Table 1
FCNR and NR are compared in terms of computational complexity.
H
1
g
H
2
g
H
3
g
H
1
H
2
T
0
T
1
T
2
τ
NR
39585
18318
1
3721
18318
19246
500
300
4.18
×
1
0
11
FCNR
180
84
1
25
84
686
500
300
7.54
×
1
0
7From Table1, we can see that the number of operations of FCNR is less than one five-thousandth of the number of operations of NR. This means that the computational complexity of NR is much higher than that of FCNR. In fact, for the image with high resolution, it is not practical to directly calculate the NR. Usually, we will do some simple preprocessing for image before calculating its NR. Therefore, the difference of computation is not so great in practice (see Section 4).


## 3. CBIR System Based on FCNR

For a given image library, we divided all the images in the library into rectangular blocks with appropriate size using mutually perpendicular lines. In each rectangular block, the low-level features are extracted in the same way and thus the local feature representation of an original image is obtained. The local features representations of all images in the library will constitute a local characteristic database. So, we can construct a hierarchical architecture for the local feature representation and the template sets of every layer of the architecture can be obtained using the local characteristic database. On this basis and using the algorithm as mentioned in Section2.2 to compute FCNR for all images in the library, we can establish a FCNR library associated with original image library. If a proper similarity measure is defined on the feature space of FCNR, then the image retrieval can be carried out.When the user enters a query image for the relevant images retrieval, first of all, the user calculates the FCNR of the query image according to the above-mentioned steps and then calculates the similarity between the query image and all images in the image database according to the defined similarity measure. Finally, the user sorts the image in the library in decreasing order according to the similarity, and some numbers of images which are arranged in the top are output to the user. The flow diagram of CBIR based on FCNR is shown in Figure4.Figure 4
The flow diagram of CBIR based on FCNR.


### 3.1. Local Low-Level Feature Extraction

In this paper, some simple and robust methods are used to extract fourteen basic low-level features, including color feature, texture feature, and shape feature, for the image block.Similar to some CBIR related literatures, we use the well-knownYCbCr color space in the extraction of color features [1, 8]. In this color space, the luminance information is stored with a single component Y, and the color information is stored with two color difference components C
b and C
r. We calculate the mean and standard deviation of C
r, C
b, and Y for each subblock, among which the mean values are denoted as g
1, g
2, and g
3 and the standard deviations are denoted as g
4, g
5, and g
6. In this way we can get six color features (for monochrome images, only two brightness features can be extracted).Next, we will use Haar wavelet transform to extract texture features from theY component of the rectangular image block. First of all, we will take Haar wavelet transform on each 4
×
4 subblock in the rectangular image block and four 2
×
2 matrixes can be obtained, which include a sampling approximation and three detail matrixes in three directions (horizontal, vertical, and diagonal). Set the three detail matrixes to be (13)
a
11
a
12
a
21
a
22
,
b
11
b
12
b
21
b
22
,
c
11
c
12
c
21
c
22
,respectively, and let(14)
a
=
∑
i
=
1
2
∑
j
=
1
2
a
i
j
2
4
,
b
=
∑
i
=
1
2
∑
i
=
1
2
b
i
j
2
4
,
c
=
∑
i
=
1
2
∑
j
=
1
2
c
i
j
2
4
.After wavelet transformation, we just assign the three variables to each pixel of the rectangular image block. Then, we can compute the averages and standard deviations of the three variables a, b, and c for each rectangular image block and denote the averages as g
7
,
g
8, and g
9 and the standard deviations as g
10, g
11, and g
12, respectively.Note that the standard deviationg
4 of the Y component of the rectangular image block has been obtained; we take the thirteenth feature as(15)
g
13
=
1
-
1
1
+
g
4
2
,which is the smoothness of the image block and it reflects the relative smooth degree of brightness in the corresponding region. The last feature g
14 is the entropy of the Y component of the rectangular image block; that is,(16)
g
14
=
-
∑
i
=
1
L
-
1
p
z
i
log
2
⁡
p
z
i
,where p
(
z
) is the gray level histogram of the Y component of the rectangular image block and L is the number of possible gray series. Entropy is a measure of the randomness of the image elements [13].In this way, the 14 features mentioned above are combined together; we can get low-level visual features representation of the rectangular image block, and we denote it asg; that is,(17)
g
=
g
1
,
g
2
,
…
,
g
14
.After obtaining the low-level visual features representation of all rectangle blocks of an image, we can get the local feature representation of the whole image as shown in (2).


### 3.2. The Similarity Measure

Retrieval accuracy is not only dependent on a robust feature representation, but also dependent on a good similarity measure. In order to highlight the advantages of FCNR in the image feature representation, we adopt a very basic and very natural way in this paper; that is, the similarity between two images is defined as the normalized inner product of their FCNRs. Specifically, for any off
∈
I
S
q, its FCNR is a vector of N
(
f
)
∈
R
T
v, where T
v represents the number of the templates in the second layer. To normalize N
(
f
), we can obtain(18)
N
^
f
=
N
f
N
f
L
2
R
T
vand the similarity of two images f
,
f
∗
∈
I
S
q can be defined as(19)
S
f
,
f
∗
=
N
^
f
,
N
^
f
∗
L
2
R
T
v
.It is not difficult to see that the mode of definition of image similarity comes down in one continuous line of the definition of similarity of image patches at all layers in the process of construction of FCNR.Thus, when the user inputs the query imageq, the system firstly computes N
(
q
) according to (10) and N
^
(
q
) according to (18) and then calculates the similarities S
(
q
,
f
i
)
i
=
1,2
,
…
,
I
S
q of the query image and all images in the database according to (18) and (19). Finally, descending sorts the images f
i based on the similarity S
(
q
,
f
i
) and outputs the top k images to the user as the query result, where the parameter k is specified by the user according to the query requirements.


## 3.1. Local Low-Level Feature Extraction

In this paper, some simple and robust methods are used to extract fourteen basic low-level features, including color feature, texture feature, and shape feature, for the image block.Similar to some CBIR related literatures, we use the well-knownYCbCr color space in the extraction of color features [1, 8]. In this color space, the luminance information is stored with a single component Y, and the color information is stored with two color difference components C
b and C
r. We calculate the mean and standard deviation of C
r, C
b, and Y for each subblock, among which the mean values are denoted as g
1, g
2, and g
3 and the standard deviations are denoted as g
4, g
5, and g
6. In this way we can get six color features (for monochrome images, only two brightness features can be extracted).Next, we will use Haar wavelet transform to extract texture features from theY component of the rectangular image block. First of all, we will take Haar wavelet transform on each 4
×
4 subblock in the rectangular image block and four 2
×
2 matrixes can be obtained, which include a sampling approximation and three detail matrixes in three directions (horizontal, vertical, and diagonal). Set the three detail matrixes to be (13)
a
11
a
12
a
21
a
22
,
b
11
b
12
b
21
b
22
,
c
11
c
12
c
21
c
22
,respectively, and let(14)
a
=
∑
i
=
1
2
∑
j
=
1
2
a
i
j
2
4
,
b
=
∑
i
=
1
2
∑
i
=
1
2
b
i
j
2
4
,
c
=
∑
i
=
1
2
∑
j
=
1
2
c
i
j
2
4
.After wavelet transformation, we just assign the three variables to each pixel of the rectangular image block. Then, we can compute the averages and standard deviations of the three variables a, b, and c for each rectangular image block and denote the averages as g
7
,
g
8, and g
9 and the standard deviations as g
10, g
11, and g
12, respectively.Note that the standard deviationg
4 of the Y component of the rectangular image block has been obtained; we take the thirteenth feature as(15)
g
13
=
1
-
1
1
+
g
4
2
,which is the smoothness of the image block and it reflects the relative smooth degree of brightness in the corresponding region. The last feature g
14 is the entropy of the Y component of the rectangular image block; that is,(16)
g
14
=
-
∑
i
=
1
L
-
1
p
z
i
log
2
⁡
p
z
i
,where p
(
z
) is the gray level histogram of the Y component of the rectangular image block and L is the number of possible gray series. Entropy is a measure of the randomness of the image elements [13].In this way, the 14 features mentioned above are combined together; we can get low-level visual features representation of the rectangular image block, and we denote it asg; that is,(17)
g
=
g
1
,
g
2
,
…
,
g
14
.After obtaining the low-level visual features representation of all rectangle blocks of an image, we can get the local feature representation of the whole image as shown in (2).


## 3.2. The Similarity Measure

Retrieval accuracy is not only dependent on a robust feature representation, but also dependent on a good similarity measure. In order to highlight the advantages of FCNR in the image feature representation, we adopt a very basic and very natural way in this paper; that is, the similarity between two images is defined as the normalized inner product of their FCNRs. Specifically, for any off
∈
I
S
q, its FCNR is a vector of N
(
f
)
∈
R
T
v, where T
v represents the number of the templates in the second layer. To normalize N
(
f
), we can obtain(18)
N
^
f
=
N
f
N
f
L
2
R
T
vand the similarity of two images f
,
f
∗
∈
I
S
q can be defined as(19)
S
f
,
f
∗
=
N
^
f
,
N
^
f
∗
L
2
R
T
v
.It is not difficult to see that the mode of definition of image similarity comes down in one continuous line of the definition of similarity of image patches at all layers in the process of construction of FCNR.Thus, when the user inputs the query imageq, the system firstly computes N
(
q
) according to (10) and N
^
(
q
) according to (18) and then calculates the similarities S
(
q
,
f
i
)
i
=
1,2
,
…
,
I
S
q of the query image and all images in the database according to (18) and (19). Finally, descending sorts the images f
i based on the similarity S
(
q
,
f
i
) and outputs the top k images to the user as the query result, where the parameter k is specified by the user according to the query requirements.


## 4. Experiments

In this section, we will discuss simulation experiments to demonstrate the performance of the proposed method in image retrieval. Firstly, the evaluation standards of the performance of CBIR system are given and the appropriate parameters of FCNR method are selected. Then, we will compare the performance of the FCNR with the classical NR and the local neural responses (LNR) [17] in image retrieval. Finally, we also compare the proposed method with several feature extraction methods which were originally designed for image retrieval, including the benchmark method and some relatively new methods.The image library used in the experiments contains 1000 images with size256
×
384 or 384
×
256 selected from COREL database which is a general-purpose image database including about 60,000 pictures [1]. These selected images have ten classes, each of which has a semantic name and contains 100 pictures. For the sake of clarity, these 1000 images are numbered from 0 to 999. The semantic name and the corresponding number range of each class are listed in Table 2 [8]. We randomly selected four pictures from each class and show them in Figure 5.Table 2
Ten classes of 1000 experimental images.
Classes
Semantic name
Number range
1
African
0~99
2
Beach
100~199
3
Building
200~299
4
Buses
300~399
5
Dinosaurs
400~499
6
Elephants
500~599
7
Flowers
600~699
8
Horses
700~799
9
Glaciers
800~899
10
Food
900~999Figure 5
Example of images in COREL database.It is necessary to emphasize that the templates used in the experiment for calculating FCNR are randomly intercepted from the local feature arrays of the image (and not from the original image) by moving or rotating window with specific size. The experiments were conducted on a computer with 4 GB random access memory and 2.60 GHz Intel(R) Core (TM) i5-3230 M processor, and the code was implemented in MATLAB in which the image processing toolbox functions are called [13].


### 4.1. Evaluation Standards and Parameter Determinations

There are a variety of ways for evaluation of the performance of retrieval. In this paper, we mainly use the recall-precision graph which is the most commonly used in community of image retrieval to evaluate the performance of FCNR. PrecisionP is defined as(20)
P
k
=
n
k
k
,where k is the number of retrieved images and n
k is the number of relevant images in the retrieved images. Recall R is defined as(21)
R
k
=
n
k
N
,where N is the number of all relevant images in the library. An optimal recall-precision graph would have a straight line; that is, precision is always at 1. Typically, when recall increases, precision decreases accordingly.However, the results of one or two times of retrieval can not fully exhibit the advantage and disadvantage of an algorithm, and it is not convenient to compare with other methods. Therefore, we randomly selected 50 images from the image database to form a set of query images. For fixed recall, averaging the precision of the 50 time queries, we can obtain the recall-average precision graph which is a relatively reliable evaluation standard. In general, the high average precision and high recall mean that the algorithm has good performance. This means that the algorithm whose recall-average precision graph is over the right upper is better. In addition, due to the requirements of real-time in CBIR task, the shorter the query time the better the performance of the algorithm.In our experiments, the images of size384
×
256 are transformed into images of size 256
×
384 firstly through the rotation, and then all the images are divided into the square subblock size of 16
×
16, which is a total of 16
×
24 blocks for each image. For each image block, we extract local features by the methods described in Section 3.1 and we can get a three-dimensional array of 16
×
24
×
14 for every image. The templates sets are constructed by randomly extracting 500 patches of u-size and 300 patches of v-size, respectively, from the local feature arrays of some 10 images per class. In the process of constructing FCNR, two very important parameters are the size of u and v. In order to select the proper sizes, we have carried out a series of experiments for different sizes of u and v.Figure6 shows four recall-precision graphs corresponding to four different patch sizes. In these experiments, the number of retrieved images k is taken as 30. It is not difficult to see from Figure 6 that the sizes of u, and v are too small or too large to get good results. In contrast, when the u-size is 7
×
7 pixels and the v-size is 11
×
11 pixels, the retrieval results are the best. Therefore, we use these sizes in the experiments shown in Figure 6.Figure 6
Recall-precision graphs of different patch sizes.Figure7 shows the top 20 images of two queries. The image in the front of the list is the query image and the figure at the bottom of each image is the number of the image in the library. As seen from Figure 7, the proposed CBIR system based on FCNR was done efficiently on the COREL image library. For the query semantics of “flower,” the outputs of the top 20 images are all the theme of flowers, and these flowers take different color, size, background, and forms. This suggests that the high-level semantics of “flower” can be correctly identified by the system. It is worth mentioning that the two images of number 674 and number 677 are rotated before the local feature extraction and still can be retrieved, indicating that the FCNR algorithm preserves the rotation invariance of NR [24, 27, 28]. For the query semantics as “the elephant,” the top 13 images of output are relevant to the query image, and among the top twenty images, only four images are inconsistent with the query image (the added border images in Figure 7).Figure 7
Two examples of FCNR-based query on COREL-1000 database.


### 4.2. Comparison with NR-Based and LNR-Based Methods

Next, we will compare the performance of FCNR with the classic NR method and the LNR method in CBIR [17]. Local neural response is an improved version of the neural response, which uses sparse techniques in the representation of image and its subblocks. Before calculating NR and LNR, it is necessary to do preprocessing to the images as mentioned in Section 2.3. In order to be relatively fair, we adopt the approach which makes the algorithm perform best as reported in the relevant literature: we convert images into 60
×
90 gray images and the u-size is 15
×
15 pixels and the v-size is 21
×
21. We use a similar manner to select the template in the three methods, that is, to intercept 500 templates of u-size and 300 templates of v-size randomly from the gray images or the local feature matrix of images.Table3 shows the time consumption of the three different methods at different stages, and the retrieval performance is shown in Figure 8. In these experiments, the number of retrieved images k is still taken as 30.Table 3
Time consumption in three different methods.
Method
Local feature
Hierarchical feature
Query
Total
NR-based
0 s
1236.6 s
5.1 s
1241.7 s
LNR-based
0 s
1882.4 s
6.4 s
1888.8 s
FCNR-based
607.3 s
117.5 s
2.6 s
727.4 sFigure 8
PR curves for three different algorithms.It can be seen from Table3 that both the hierarchical feature learning time and the query time of FCNR-based methods are significantly shorter than the other two methods. This is mainly because the latter two methods use exhaustion algorithm of translations of pixel to pixel. In particular, the LNR method, which introduces the solution of the quadratic optimization problems, is the most time-consuming method. Therefore, although the retrieval method based on FCNR can take some time on the extraction of local features, the time of learning the FCNR and the query time can be greatly reduced. This point is very important for the image retrieval task, because the real-time performance is the basic requirement in image retrieval [29].Besides that, it is not hard to see from Figure8 that the retrieval precision of the method based on FCNR is better than those methods based on NR or LNR. The main reason is that the FCNR-based method effectively overcomes the shortcomings of comparison of pixel values in the underlying image blocks just as in the NR and LNR methods. At the same time, the loss of color information also affected the performance of LNR and NR to a certain extent. By the way, the results based on LNR are better than that based on the NR. This is mainly because the localization and the sparse encoding in the LNR method make the image on the target location have a high value of neural response.


### 4.3. Comparison with Some Other Methods Proposed for Image Retrieval

Finally, we will compare FCNR with some other methods originally proposed for image retrieval, which include the edge histogram descriptor (EHD) method [10], the color difference histogram descriptor (CDH) [12], and the latest methods such as the error diffusion block truncation coding (EDBTC) [18] and the bandletized regions through support vector machines (BRSVM) [6].As a benchmark, the EHD was initially used for texture image retrieval. In order to be fair, we extract features from the three components of R, G, and B of the image, and the edge intensity of the image block over 11 is used for the calculation of the histogram. Each component corresponds to an 80-dimensional feature vector, so that the resulting EHD feature vector is 240 dimensions. In the CDH representation we useYCbCr color space and the color and the orientation parameters are taken as 90 and 18, which is the relatively good performance parameter configuration as reported in the literature [12]. Thus the image of the CDH is expressed as a 108-dimension vector. The EDBTC produces two color quantizers and a bitmap image, which are further processed using vector quantization to generate the image feature descriptor. There are two features that are introduced in EDBTC, namely, color histogram feature and bit pattern histogram feature, to measure the similarity between a query image and the target image in database. The BRSVM method was prosed to overcome drawbacks and limitations of this traditional image segmentation technology. In BRSVM method, a bandelets transform based image representation technique is presented, which reliably returns the information about the major objects found in an image, and support vector machine is applied for image retrieval purposes.Similarly, we randomly select 50 images from the image database to form a query image setQ, and the output image number is set to 10, 20, and 50, respectively. Table 4 lists the average precision and recall in the three different cases corresponding to the five methods, and Figure 9 shows the PR curve when the number of output images is set to 50.Table 4
Average precision and recall of four different methods in different situations.
Output
Criterion
EHD
CDH
EDBTC
BRSVM
FCNR
10
Precision
48.47
62.44
63.87
67.13
68.63
Recall
4.85
6.24
6.58
6.73
6.86
20
Precision
38.15
53.72
52.98
54.24
55.24
Recall
7.63
10.74
10.64
10.87
11.05
50
Precision
29.76
42.87
44.23
45.79
46.98
Recall
14.89
21.44
21.98
22.15
23.50Figure 9
The PR curves of four different algorithms on COREL-1000 database.It is not difficult to see from Table4 and Figure 9 in the COREL-1000 image database that the performance of proposed method not only is significantly better than the MPEG-7 standard feature extraction method such as EHD but also has a stroke above those latest methods, such as EDBTC and BRSVM. From our point of view, this is mainly due to two reasons: the first reason is that, based on the hierarchical structure, FCNR is the result of deep learning on the low-level features of the image and the second reason is that the high-level semantic elements of images are integrated into the feature representation of FCNR by using the templates sets.


## 4.1. Evaluation Standards and Parameter Determinations

There are a variety of ways for evaluation of the performance of retrieval. In this paper, we mainly use the recall-precision graph which is the most commonly used in community of image retrieval to evaluate the performance of FCNR. PrecisionP is defined as(20)
P
k
=
n
k
k
,where k is the number of retrieved images and n
k is the number of relevant images in the retrieved images. Recall R is defined as(21)
R
k
=
n
k
N
,where N is the number of all relevant images in the library. An optimal recall-precision graph would have a straight line; that is, precision is always at 1. Typically, when recall increases, precision decreases accordingly.However, the results of one or two times of retrieval can not fully exhibit the advantage and disadvantage of an algorithm, and it is not convenient to compare with other methods. Therefore, we randomly selected 50 images from the image database to form a set of query images. For fixed recall, averaging the precision of the 50 time queries, we can obtain the recall-average precision graph which is a relatively reliable evaluation standard. In general, the high average precision and high recall mean that the algorithm has good performance. This means that the algorithm whose recall-average precision graph is over the right upper is better. In addition, due to the requirements of real-time in CBIR task, the shorter the query time the better the performance of the algorithm.In our experiments, the images of size384
×
256 are transformed into images of size 256
×
384 firstly through the rotation, and then all the images are divided into the square subblock size of 16
×
16, which is a total of 16
×
24 blocks for each image. For each image block, we extract local features by the methods described in Section 3.1 and we can get a three-dimensional array of 16
×
24
×
14 for every image. The templates sets are constructed by randomly extracting 500 patches of u-size and 300 patches of v-size, respectively, from the local feature arrays of some 10 images per class. In the process of constructing FCNR, two very important parameters are the size of u and v. In order to select the proper sizes, we have carried out a series of experiments for different sizes of u and v.Figure6 shows four recall-precision graphs corresponding to four different patch sizes. In these experiments, the number of retrieved images k is taken as 30. It is not difficult to see from Figure 6 that the sizes of u, and v are too small or too large to get good results. In contrast, when the u-size is 7
×
7 pixels and the v-size is 11
×
11 pixels, the retrieval results are the best. Therefore, we use these sizes in the experiments shown in Figure 6.Figure 6
Recall-precision graphs of different patch sizes.Figure7 shows the top 20 images of two queries. The image in the front of the list is the query image and the figure at the bottom of each image is the number of the image in the library. As seen from Figure 7, the proposed CBIR system based on FCNR was done efficiently on the COREL image library. For the query semantics of “flower,” the outputs of the top 20 images are all the theme of flowers, and these flowers take different color, size, background, and forms. This suggests that the high-level semantics of “flower” can be correctly identified by the system. It is worth mentioning that the two images of number 674 and number 677 are rotated before the local feature extraction and still can be retrieved, indicating that the FCNR algorithm preserves the rotation invariance of NR [24, 27, 28]. For the query semantics as “the elephant,” the top 13 images of output are relevant to the query image, and among the top twenty images, only four images are inconsistent with the query image (the added border images in Figure 7).Figure 7
Two examples of FCNR-based query on COREL-1000 database.


## 4.2. Comparison with NR-Based and LNR-Based Methods

Next, we will compare the performance of FCNR with the classic NR method and the LNR method in CBIR [17]. Local neural response is an improved version of the neural response, which uses sparse techniques in the representation of image and its subblocks. Before calculating NR and LNR, it is necessary to do preprocessing to the images as mentioned in Section 2.3. In order to be relatively fair, we adopt the approach which makes the algorithm perform best as reported in the relevant literature: we convert images into 60
×
90 gray images and the u-size is 15
×
15 pixels and the v-size is 21
×
21. We use a similar manner to select the template in the three methods, that is, to intercept 500 templates of u-size and 300 templates of v-size randomly from the gray images or the local feature matrix of images.Table3 shows the time consumption of the three different methods at different stages, and the retrieval performance is shown in Figure 8. In these experiments, the number of retrieved images k is still taken as 30.Table 3
Time consumption in three different methods.
Method
Local feature
Hierarchical feature
Query
Total
NR-based
0 s
1236.6 s
5.1 s
1241.7 s
LNR-based
0 s
1882.4 s
6.4 s
1888.8 s
FCNR-based
607.3 s
117.5 s
2.6 s
727.4 sFigure 8
PR curves for three different algorithms.It can be seen from Table3 that both the hierarchical feature learning time and the query time of FCNR-based methods are significantly shorter than the other two methods. This is mainly because the latter two methods use exhaustion algorithm of translations of pixel to pixel. In particular, the LNR method, which introduces the solution of the quadratic optimization problems, is the most time-consuming method. Therefore, although the retrieval method based on FCNR can take some time on the extraction of local features, the time of learning the FCNR and the query time can be greatly reduced. This point is very important for the image retrieval task, because the real-time performance is the basic requirement in image retrieval [29].Besides that, it is not hard to see from Figure8 that the retrieval precision of the method based on FCNR is better than those methods based on NR or LNR. The main reason is that the FCNR-based method effectively overcomes the shortcomings of comparison of pixel values in the underlying image blocks just as in the NR and LNR methods. At the same time, the loss of color information also affected the performance of LNR and NR to a certain extent. By the way, the results based on LNR are better than that based on the NR. This is mainly because the localization and the sparse encoding in the LNR method make the image on the target location have a high value of neural response.


## 4.3. Comparison with Some Other Methods Proposed for Image Retrieval

Finally, we will compare FCNR with some other methods originally proposed for image retrieval, which include the edge histogram descriptor (EHD) method [10], the color difference histogram descriptor (CDH) [12], and the latest methods such as the error diffusion block truncation coding (EDBTC) [18] and the bandletized regions through support vector machines (BRSVM) [6].As a benchmark, the EHD was initially used for texture image retrieval. In order to be fair, we extract features from the three components of R, G, and B of the image, and the edge intensity of the image block over 11 is used for the calculation of the histogram. Each component corresponds to an 80-dimensional feature vector, so that the resulting EHD feature vector is 240 dimensions. In the CDH representation we useYCbCr color space and the color and the orientation parameters are taken as 90 and 18, which is the relatively good performance parameter configuration as reported in the literature [12]. Thus the image of the CDH is expressed as a 108-dimension vector. The EDBTC produces two color quantizers and a bitmap image, which are further processed using vector quantization to generate the image feature descriptor. There are two features that are introduced in EDBTC, namely, color histogram feature and bit pattern histogram feature, to measure the similarity between a query image and the target image in database. The BRSVM method was prosed to overcome drawbacks and limitations of this traditional image segmentation technology. In BRSVM method, a bandelets transform based image representation technique is presented, which reliably returns the information about the major objects found in an image, and support vector machine is applied for image retrieval purposes.Similarly, we randomly select 50 images from the image database to form a query image setQ, and the output image number is set to 10, 20, and 50, respectively. Table 4 lists the average precision and recall in the three different cases corresponding to the five methods, and Figure 9 shows the PR curve when the number of output images is set to 50.Table 4
Average precision and recall of four different methods in different situations.
Output
Criterion
EHD
CDH
EDBTC
BRSVM
FCNR
10
Precision
48.47
62.44
63.87
67.13
68.63
Recall
4.85
6.24
6.58
6.73
6.86
20
Precision
38.15
53.72
52.98
54.24
55.24
Recall
7.63
10.74
10.64
10.87
11.05
50
Precision
29.76
42.87
44.23
45.79
46.98
Recall
14.89
21.44
21.98
22.15
23.50Figure 9
The PR curves of four different algorithms on COREL-1000 database.It is not difficult to see from Table4 and Figure 9 in the COREL-1000 image database that the performance of proposed method not only is significantly better than the MPEG-7 standard feature extraction method such as EHD but also has a stroke above those latest methods, such as EDBTC and BRSVM. From our point of view, this is mainly due to two reasons: the first reason is that, based on the hierarchical structure, FCNR is the result of deep learning on the low-level features of the image and the second reason is that the high-level semantic elements of images are integrated into the feature representation of FCNR by using the templates sets.


## 5. Conclusions

This research has been devoted to construct a new image feature representation of FCNR to be used for CBIR. It preserves the excellent characteristics of NR and LNR, such as being invariant to translation and illumination and robust to local distortion and clutter. More importantly, it also takes both visual feature and semantic feature into account in image recognition. The experimental results on the COREL-1000 image database have shown that, compared with NR and LNR, FCNR is more suitable for image retrieval tasks. In addition, the proposed method achieves a higher retrieval accuracy compared with other methods originally proposed for image retrieval in the COREL database. We attribute the effectiveness of the proposed method to both the local feature extraction and the hierarchical architecture which is used in deep learning of low-level visual features and take the high-level semantics of the image into its feature representation.Although both theoretical analysis and experimental results show that FCNR is an applicable representation of image for CBIR, there are still some problems to be further studied.(i)
The template sets and the number of selected templates in this paper are determined according to experience; we do not pay any attention to qualitative and quantitative analysis. How to select the more representative templates and what are the optimal numbers of templates which are needed in the construction of FCNR are still important issues for future work [30].
(ii)
In this paper, we used the simple inner product kernels as the similarity measure. In order to obtain a better retrieval performance, how to combine FCNR and the mainstream of similarity learning technology in recent years is worth studying problem.
(iii)
As is well known, the relevance feedback technique plays an important role in image retrieval [9], and how to introduce relevance feedback into the algorithm proposed in this paper is another interesting subject.



---

*Source: 3908056-2016-09-29.xml*