# Retrieval Architecture with Classified Query for Content Based Image Recognition

Authors: Rik Das; Sudeep Thepade; Subhajit Bhattacharya; Saurav Ghosh
Journal: Applied Computational Intelligence and Soft Computing
 (2016)

---

## Abstract

The consumer behavior has been observed to be largely influenced by image data with increasing familiarity of smart phones and World Wide Web. Traditional technique of browsing through product varieties in the Internet with text keywords has been gradually replaced by the easy accessible image data. The importance of image data has portrayed a steady growth in application orientation for business domain with the advent of different image capturing devices and social media. The paper has described a methodology of feature extraction by image binarization technique for enhancing identification and retrieval of information using content based image recognition. The proposed algorithm was tested on two public datasets, namely, Wang dataset and Oliva and Torralba (OT-Scene) dataset with 3688 images on the whole. It has outclassed the state-of-the-art techniques in performance measure and has shown statistical significance.


---

## Body

## 1. Introduction

Image data has strong impact on consumer attention that has kindled the buying intention of consumers [1, 2]. Advent of globalization has rapidly influenced the customer preferences and demands [3]. Consumer satisfaction has led to strong and positive behavioral outcomes connected to sustainable purchase [4]. The revenue of digital business process has been adversely affected by predominant dissatisfaction of consumers due to large amount of irrelevant results generation from text based queries [5]. The customers were deprived of taking pleasure in a perishable searching environment. This has led to the popularity of image data which has principally reinstated the text based keyword searching. The progression of multimedia technology has encouraged the use of multimedia in business practice and changed the way we use computers [6, 7]. Image data has revealed increasing importance in the field of contemporary business environment. The authors have proposed a novel methodology for retrieving image data with query classification which has stimulated the performance compared to state-of-the-art techniques for information identification. Statistical measures have been adopted to validate user responses for the fruitfulness of the proposed technique and to establish the significance of the findings.


## 2. Related Work

Accessibility of diverse online and offline information for products and services has radically altered the customer preferences [8]. Traditional means to locate the product of interest by the customers were based on text queries. However, the method has huge amount of irrelevant results as output. One of the driving factors for inappropriate output has been due to reprehensible selection of keywords as query. Recent approaches of searching have emphasized on the content of the searched object rather than its name as a keyword [9–12]. The content based searching process has been facilitated by the product image which can provide the necessary knowledge for the required product based on its image contents and has been anticipated to filter out the unwanted results with higher probability. Various methods have been carried out for feature extraction that has applied image binarization as a tool to signify the object of interest and its background, respectively [13–15]. Threshold selection has been vital to enable binarization of image to differentiate the object and its background. It has been stated that uneven illumination and inconsistent gray levels within the image and its backgrounds have adversely affected threshold selection for binarization [16–18]. Threshold selection can be divided into three different categories, namely, mean threshold selection, local threshold selection, and global threshold selection. Feature extraction with mean threshold for binarization has been discussed in [19, 20] and with bit plane slicing in [21]. The problem of uneven illumination in images was efficiently addressed by local threshold techniques [9, 22–26]. The literatures have used measures of dispersion like standard deviation and variance to calculate the threshold. El Alami, 2011 [27], has worked with color and texture based features by 3D color histogram and Gabor filters. Hiremath and Pujari, 2007 [28], have calculated local descriptors of color and texture from the color moments and moments on Gabor filter responses by dividing the image into nonoverlapping blocks. Imagery significant point features were chosen by Banerjee et al., 2009 [29], for retrieval process of images. Jalab, 2011 [30], has fused color layout descriptor and Gabor texture for better detection of images. Shen and Wu, 2013 [31], have extracted signatures from images by dealing with color, texture, and spatial structure descriptors. Irtaza et al., 2014 [32], have explored wavelet packets and eigen values of Gabor filters to generate feature vectors from images. Rahimi and Moghaddam, 2015 [33], have referred to the intraclass and interclass features for effective extraction of image features of interest. The authors have proposed a novel technique for content based product recognition in an Internet based business model and has compared the same with the existing techniques of object recognition discussed in the literature. The performance of the proposed technique has outclassed the existing methods and has shown statistically significant improvement to foster generation of relevant results for a given content based product query.


## 3. Our Approach

The proposed methodology has considered different image categories which have signified diversified nature of product varieties offered by the firms. Uneven illumination of images can be a factor for degradation of product recognition process. It has been addressed by the authors by selection of local threshold for binarization. At the outset, the binarization was carried out using Niblack’s method of local threshold selection [9, 18]. Each color component of an image was considered for derivation of pixel-wise threshold values by sliding a rectangular window over the component. The localmean
m
(
i
,
j
) andstandard deviation
σ
(
i
,
j
) were calculated primarily with a window size of (25
×
25). The threshold calculation was done as T
(
i
,
j
)
=
m
(
i
,
j
)
+
k
·
σ
(
i
,
j
). The value of k was a constant in between 0 and 1 and was considered to be 0.6. The quality of binarization was dependent on the size of the sliding window and the value of k.


## 4. Feature Vector Generation

The pixels were locally divided into two different intensity values, namely, the higher intensity values and the lower intensity values by comparing with the corresponding threshold values. The mean and the standard deviation of the higher intensity value pixels and the lower intensity value pixels were considered to derive the higher intensity feature vector and the lower intensity feature vector, respectively. The process of feature extraction has been graphically illustrated in Figure1. Initially, the original image displaying the model with a cell phone has been considered for feature extraction as shown in Figure 1(a). The image was divided into red (R), green (G), and blue (B) color component at the beginning in Figures 1(b)–1(d). The process was followed by binarization of each of the color components by Niblack threshold selection method shown in Figures 1(e)–1(g). The binarized image of individual color components has different shades of black and white as clearly visible. Two feature vectors of higher and lower intensity values for each color component were computed from the binarized image and were stored for image recognition as shown in Figure 1(h).Figure 1
(a) Original image. (b) Red component. (c) Green component. (d) Blue component. (e) Binarization of red component. (f) Binarization of green component. (g) Binarization of blue component. (h) Extracted feature vectors.The algorithm has been given in Algorithm1.Algorithm 1.
1234
Begin
(1)
Input an image I with three different color components R, G, and B, respectively, of sizem
∗
n each.
(2)
Calculate the local threshold valueT
x for each pixel in each of the color components R, G, and B using Niblack’s method:(1)
T
x
=
1
m
∗
n
∗
∑
i
=
1
m
∑
j
=
1
n
x
i
,
j
+
k
∗
σ
1
m
∗
n
∗
1
m
∗
n
∗
∑
i
=
1
m
∑
j
=
1
n
x
i
,
j
-
x
i
,
j
2
,
wherek
=
0.6.
/x
∗ = R, G and B /
∗.
(3)
Compute binary image maps for each pixel for the given image:(2)
B
i
t
M
a
p
x
i
,
j
=
1
i
f
x
i
,
j
>
T
x
0
i
f
x
i
,
j
<
T
x
/x
∗ = R, G and B /
∗.
(4)
Generate image features for the given image for each color component:(3)
X
u
p
m
e
a
n
=
1
∑
i
=
1
m
∑
j
=
1
n
B
i
t
M
a
p
x
i
,
j
∗
∑
i
=
1
m
∑
j
=
1
n
B
i
t
M
a
p
x
i
,
j
∗
X
i
,
j
X
l
o
m
e
a
n
=
1
m
∗
n
-
∑
i
=
1
m
∑
j
=
1
n
B
i
t
M
a
p
x
i
,
j
∗
∑
i
=
1
m
∑
j
=
1
n
1
-
B
i
t
M
a
p
x
i
,
j
∗
X
i
,
j
/x
∗ = R, G and B /
∗.


## 5. Complexity Analysis

The proposed technique has parted the images into three color components, namely, red, green, and blue. If the total number of gray levels for each component was assumed to beG then linear time O
(
G
) was consumed for selection of threshold for all the gray values in each component. Hence, the number of iterations for three color components was O
(
3
N
). Consequently, it was inferred that the time complexity for the feature selection process was linear. Conventional feature extraction techniques have same feature dimension to that of the image from which the feature was extracted. Hence, for an image of size S
∗
S the feature size was S
2. The proposed method has radically reduced the feature size to 12 irrespective of the image dimension. Thus, the space complexity was efficiently addressed with small feature size.


## 6. Retrieval Architecture

The process of retrieval was carried out by means of classified query as in Figure2. Conventional retrieval process comprised searching the entire dataset with a generic user query. On the contrary, retrieval with classified query initially classifies the query image into the nearest category of images. The classification process was followed by retrieval of images only from the class of interest. The rest of the image categories were pruned down for a classified query as they do not belong to the native class of the query. Thus, the process has evidently improved the recognition performance compared to state-of-the-art techniques. However, a wrong query classification resulting from bleak feature extraction process would result in zero image retrieval of relevant class as all images will be retrieved from the misclassified category. In case of generic query the scenario is different and in most of the cases a minimum number of image retrieval can be expected from the class of interest. Higher degree of misclassification of retrieval query would have adverse effect on retrieval performance which may be considered as a disadvantage of the proposed technique with respect to existing methods. Nevertheless, it can be avoided by designing effective techniques for robust feature extraction.Figure 2
Block diagram of feature extraction and recognition process.


## 7. Experimental Verification

Two different datasets, namely, Wang dataset (10 categories, 1000 images) and Oliva and Torralba (OT-Scene) dataset (8 categories, 2688 images) were considered for the evaluation purpose. Figures3 and 4 have illustrated the sample of the datasets used for the experimentation process. The validation process was carried out with 10-fold cross validation in which 9 subsets were considered as training set and 1 subset was considered as the testing set. 10 trials were conducted for the performance assessment of the classifiers. The final decision was made by combining the 10 results thus obtained after evaluating 10-fold cross validation. The evaluation process was performed using three different classifiers, namely, K Nearest Neighbor (KNN), Support Vector Machine (SVM), and Artificial Neural Network (ANN) [36]. KNN was considered as an instance based classifier which has acted based on similarity functions of two different instances. SVM has followed the learning process of Self-Organizing Map (SOM) which has presumed that only nearby nodes have affected the behavior of each other. Finally, ANN was based on a feed-forward architecture known as multilayer perceptron (MLP). The retrieval process was conducted using city block distance as a measure to match the query image with the database image. The equation for city block distance has been given in (4)
D
=
∑
i
-
1
n
Q
i
-
T
i
,where D is distance, Q
i is query image, and T
i is database image.Figure 3
Sample image for Wang dataset.Figure 4
Sample image for OT-Scene dataset.Two different metrics were considered for evaluation purpose, namely, precision and recall. Precision was defined as the probability that an object is classified correctly as per the actual value and recall was considered as the probability of a classifier to produce true positive results.Table1 has given the precision and recall rate for two different datasets, namely, Wang dataset and OT-Scene dataset under three different classifier environments. It was observed that in case of Wang dataset the highest precision was of 0.838 and the highest recall was of 0.837 using ANN classifier. The highest precision rate for OT-Scene dataset was noted to be of 0.753 with a recall rate of 0.754.Table 1
Precision and recall for two datasets under different classifier environment.
Wang dataset
OT-Scene dataset
KNN
SVM
ANN
KNN
SVM
ANN
Precision
0.776
0.807
0.838
0.63
0.753
0.665
Recall
0.763
0.804
0.837
0.619
0.754
0.663The precision and recall rate for classification with proposed technique of feature extraction were further compared with state-of-the-art techniques as shown in Figure5.Figure 5
Graphical illustration for precision and recall comparison.It was clearly observed that classification with proposed technique of feature extraction has outperformed the existing techniques.Hypothesis 1.
Classification with the proposed method of feature extraction has outclassed the existing techniques.Table2 has shown the significance of p values obtained from precision comparison and hence the null hypothesis of equal precision rate for the proposed algorithm and existing algorithms was rejected. Hence it was inferred that the proposed method has been capable of improving the classification performance radically. Further, the proposed technique of feature extraction was tested for retrieval performance. The performance was assessed by comparing retrieval with classified query and retrieval with generic query. Table 3 has shown the category wise comparison of precision for retrieval for both the retrieval techniques. The comparison in Table 3 has revealed that retrieval with classified query has improved the identification for each category of images in the Wang dataset. Hence the overall performance of retrieval has increased considerably. The average precision for retrieval with classified query has also outperformed the conventional retrieval technique with generic query.Table 2
Pairedt-test for significance testing of precision results for classification.
Comparison
t-calc
p value
Significance of difference in value
Feature extraction by binarization using bit plane slicing with Niblack’s local threshold method (Thepade et al., 2014 [9])
3.1626
0.0133
Significant
Feature extraction by binarization with multilevel mean threshold (Kekre et al., 2013 [19])
3.1626
0.0133
Significant
Feature extraction by  binarization using Bit Plane Slicing with mean threshold (Kekre et al., 2013 [19])
2.9059
0.0197
Significant
Feature extraction by binarization of original + even image with mean threshold (Thepade et al., 2013 [20])
3.4116
0.0092
Significant
Traditional feature extraction by binarization with Bernsen’s local threshold method (Yanli and Zhenxing, 2012 [26])
3.2593
0.0115
Significant
Traditional feature extraction by binarization with Sauvola’s local threshold method (Ramírez-Ortegón and Rojas, 2010 [23])
3.0157
0.0167
Significant
Traditional feature extraction by binarization with Niblack’s local threshold method (Liu, 2013 [22])
3.9038
0.0045
Significant
Traditional feature extraction by binarization with Otsu’s global threshold method (Shaikh et al., 2013 [24])
3.661
0.0064
SignificantTable 3
Category wise comparison for retrieval with classified query and generic query.
Categories
Retrieval with classified query
Retrieval with generic query
Tribals
80
54
Sea beach
60
48
Gothic structure
60
39
Bus
80
61
Dinosaur
100
100
Elephant
80
65
Roses
80
70
Horses
100
100
Mountains
80
56
Food
60
54
Average precision
78
64.7Hypothesis 2.
Retrieval with classified query cannot give higher precision results compared to retrieval with generic query.p value in Table 4 has shown significance and hence Hypothesis 2 was rejected. Henceforth, the precision results of proposed technique of feature extraction for retrieval with classified query were compared to state-of-the-art techniques of retrieval as in Table 5.Table 4
Pairedt-test for significance of retrieval results with classified query.
Comparison
t-calc
p value
Significance of difference in value
Retrieval with generic query and retrieval with classified query
4.4888
0.0015
SignificantTable 5
Category wise comparison of precision results of the proposed technique with the state-of-the art techniques.
Categories
Proposed
Subrahmanyam et al., 2012 [34]
Jalab, 2011 [30]
Walia and Pal, 2014 [25]
Hiremath and Pujari, 2007 [28]
Yue et al., 2011 [35]
Rahimi and Moghaddam, 2015 [33]
Tribals
80
54.1
32.3
54
48
58.75
41.8
Sea beach
60
48.3
61.2
51
34
41.19
44.6
Gothic structure
60
40.2
39.2
38
36
42.35
49.2
Bus
80
89.5
39.5
46
61
71.69
30.5
Dinosaur
100
97.7
99.6
100
95
74.53
55.4
Elephant
80
39.1
55.7
63
48
65.08
69.9
Roses
80
85.9
89.3
90
61
83.24
61.1
Horses
100
81.4
65.2
93
74
69.3
57.2
Mountains
80
32
56.8
48
42
44.86
43.7
Food
60
41
44.1
39
50
44.54
43.5
Average precision
78
60.9
58.29
62.2
54.9
59.55
49.69Hypothesis 3.
Retrieval with classified query with proposed feature extraction technique has higher precision compared to the existing techniques for product recognition for diversified product categories.It was evidently revealed in Table5 that the proposed method has higher performance compared to the existing techniques and has outclassed the category wise precision results for each of the existing techniques. The statistical significance of retrieval with the proposed technique was established by a paired t-test and the result has been given in Table 6. The test was carried out to validate that the difference in precision results was not generated from a population with zero mean.Table 6
Pairedt-test for significance of precision results for retrieval.
Comparison
t-calc
p value
Significance of difference in value
Subrahmanyam et al., 2012 [34]
2.9118
0.0173
Significant
Jalab, 2011 [30]
3.3348
0.0087
Significant
Walia and Pal, 2014 [25]
3.5295
0.0064
Significant
Hiremath and Pujari, 2007 [28]
7.1826
0.0001
Significant
Yue et al., 2011 [35]
5.3223
0.0005
Significant
Rahimi and Moghaddam, 2015 [33]
5.8345
0.0002
SignificantTable6 has shown significant p values for the comparison of precision for retrieval with proposed feature extraction technique with respect to the existing techniques. Hence Hypothesis 3 was accepted and the supremacy of the proposed method for content based image retrieval was established.Hypothesis 4.
Image data based query and text data based query in digital marketplace have similar impact on consumer satisfaction.Analysis in Table7 has shown significant association in between consumer satisfaction and image data based query (likelihood ratio = 79.270; Phi = 0.734; Cramer’s V
=
0.424; and p
<
0.01) related to product recognition in diversified product categories compared to text data based query (likelihood ratio = 18.577; Phi = 0.296; Cramer’s V
=
0.210; and p
<
0.05). It was observed that image data based query was having greater association with consumer satisfaction related to product recognition with respect to text data based query. Henceforth, the relation between consumer satisfaction and product query based on image data has been analyzed in Table 8.Table 7
Chi-square test for consumer satisfaction analysis related to product recognition in diversified product categories in digital environment.
Chi-square tests
⁢Raw variable → consumer satisfaction related to product recognition in diversified product categories
Column variables↓
Likelihood ratio
Sig. (2-sided)
Phi
Cramer’sV
Image data based query
79.270
.000
.734
.424
Text data based query
18.577
.039
.296
.210
Number of samples 185.Table 8
Nonparametric correlation test for consumer satisfaction analysis related to product recognition in diversified product categories in online environment.
Kendall’s tau: nonparametric correlation coefficient
Consumer satisfaction related to product recognition in diversified product categories  in online environment
Image data based query
Correlation coefficient
.498
Sig. (2-tailed)
.000
Text data based  query
Correlation coefficient
−0.32
Sig. (2-tailed)
0.643From the above results shown in Table8, significant correlation was observed between consumer satisfaction and query based on image data related to product recognition in diversified product categories. On the other hand it was detected that significant negative correlation lies in between consumer satisfaction and query based on text data related to product recognition in diversified product categories.


## 8. Conclusion

The model of digital marketplace has enriched the firms with competitive strategy to gain technical payback from their competitors. The authors have investigated the application orientation of innovation information identification by image data. The study has addressed the challenge of relevant result generation for customer query and has efficiently suggested a content based search method for relevant output generation. The novel technique of product identification based on query with image data has envisioned a new direction for innovation and value addition for using information technology in organizational development. The technique has outclassed all the state-of-the-art techniques and has increased customer satisfaction in product recognition from multiple product categories. It has efficiently pruned the inappropriate outcomes for customer queries by replacing text based search process with image query formulation. The work can be extended towards unstructured data analysis and e-commerce and for other analytical activities related to consumer orientated revenue generation for modern business process.



---

*Source: 1861247-2016-02-29.xml*