論文 / 著書情報
Article / Book Information
Title

Finding Image Semantics from a Hierarchical Image Database basedon Adaptively Combined Visual Features

AuthorPritee Khanna, Shreelekha Pandey, Haruo Yokota

Journal/Book name

Lecture notes in computer science, LNCS, 8055, , pp. 103-117

発行日 / Issue date2013, 8

DOI

http://dx.doi.org/10.1007/978-3-642-40285-2̲11

権利情報 / Copyright

The original publication is available at www.springerlink.com.

Note

このファイルは著者（最終）版です。
This file is author (final) version.

Powered by T2R2 (Science Tokyo Research Repository)

Finding Image Semantics from a Hierarchical Image
Database based on Adaptively Combined Visual Features
Pritee Khanna1, Shreelekha Pandey1, Haruo Yokota2
1

PDPM Indian Institute of Information Technology, Design and Manufacturing Jabalpur,
Dumna Airport Road, Jabalpur 482-005 M.P. India{pkhanna, shreelekha}@iiitdmj.ac.in
2
Graduate School of Information Science and Engineering, Tokyo Institute of Technology,
2-12-1 Ookayama, Meguro-kuTokyo, 152-8552 Japan
yokota@cs.titech.ac.jp

Abstract. Correlating image semantics with its low level features is achallenging task. Although, humans are adept in distinguishing object
categories, both in visual as well as in semantic space, but to accomplish thiscomputationally is yet to be fully explored. The learning based techniques do
minimize the semantic gap, but unlimited possible categorization of objects inreal world is a major challenge to these techniques. This work analyzes andutilizes the strength of a semantically categorized image database to assign
semantics to query images. Semantics based categorization of images would
result in image hierarchy. The algorithms proposed in this work exploit visual
image descriptors and similarity measures in the context of a semantically
categorized image database. A novel ‘Branch Selection Algorithm’ isdeveloped for a highly categorized and dense image database, which drasticallyreduces the search space. The search space so obtained is further reduced byapplying any one of the four proposed ‘Pruning Algorithms’. Pruningalgorithms maintain accuracy while reducing the search space. Thesealgorithms use an adaptive combination of multiple visual features of an image
database to find semantics of query images. Branch Selection Algorithm tested
on a subset of ‘ImageNet’ database reduces search space by 75%. The best
pruning algorithm further reduces this search space by 26% while maintaining
95% accuracy.

1 Introduction
Cognitive psychology defines categories by grouping “similar objects” and supercategories by grouping “similar categories”. Semantic categories form clusters invisual space, and visual similarity is correlated to semantic similarity [1]. Humans caneasily correlate these similarities which gives them enormous power to distinguish alarge number of objects. Content Based Image Retrieval (CBIR) systems use only
visual similarity obtained in terms of low level image features to interpret images [24]. The lack of coincidence between the high level semantic and the low-levelfeatures of an image is known as semantic gap [5]. In an attempt to reduce semantic
gap, proposed work aims to correlate visual similarity and semantics of images in asemantically categorized large image database. Semantic based categorization of animage database would result in categories and subcategories of images. Visualfeatures of images in such a database forms a semantics based hierarchical search
space. This tree is searched to assign semantics to query images. For efficient search,
it is not advisable to traverse the entire tree or even an entire branch. A novel ‘Branch
Selection Algorithm’ effectively traverses this hierarchical search space and selects a
few subtrees to search. Pruning Algorithm further reduces this search space, while
maintaining the accuracy. An adaptive combination of multiple visual features andsimilarity measures are used to design branch selection and pruning algorithms. To
ensure the applicability of the proposed algorithms, their performance has been tested
on a subset of ImageNet database.
The paper is organized as follows. A review of the related research is given in
Section 2. Section 3 emphasizes on correlating visual and semantic similarity. Section
4 gives an insight of related databases. Proposed system is explained in Section 5.
Experimental setup is given in Section 6. Section 7 summarizes results and discussion
on related issues. Finally, Section 8 concludes the work.

2 Related Work
Computer Vision and Machine Learning approaches use learning based systems to
reduce semantic gap [6-7]. It has already been recognized that learning accompanied
by object extraction produces good results [8]. In [9], semantic templates areautomatically generated during the process of relevance feedback. WordNet is used toconstruct a network of such semantic templates, which helps in retrieving images
based on semantic. The system works on 500 images from categories like human,
animal, car, etc. A statistical modeling approach for automatic linguistic indexing of
pictures is introduced in [6]. Each of the 600 concepts is represented by a twodimensional multi-resolution hidden Markov model and is trained using categorized
images. A likelihood function measures the extent of the association between an
image and the textual description of a concept. The model given in [10] learns visual
recognition from semantic segmentation of photographs. For efficient labeling ofobject classes, a combination of integral image processing and feature sharing is
employed. The developed classifier reports 70.5% region-based recognition accuracy
on a 21-class database. The work presented in [7] focuses on using a few training
images for quick learning. Generative probabilistic models of object categories arelearned using a Bayesian incremental algorithm. The system quoted a feasible realtime learning rate for 101 object categories. A region-based image retrieval systemwith high-level semantic learning is given in [11]. The system uses a decision tree
based image semantic learning algorithm but learns natural scenery image semantics
only.
Besides in literature, one can find a few more learning based techniques tominimize the semantic gap [5]. Unlimited number of concepts in the real world is amajor hindrance for learning based approaches. Most of the works have considered
non-hierarchical image database with thousands of images. The proposed work uses ahierarchical image database to correlate visual similarity with semantic similarity.

Such a correlation would be an asset to people working in image processing andcomputer vision. The main aim of this study is to efficiently assign semantics toimages through such correlations. Instead of using any of the available learningtechniques, this work exploits the inherited features of a hierarchical image database.

3

Correlating Visual Similarity with Semantics of Images

Humans have natural instinct in distinguishing object categories, both in visual aswell as in semantic space, but to accomplish this computationally is yet to be fully
explored. The semantic based categorization of images would give a hierarchical tree
structure having images of different categories at various levels. The focus of thiswork is to explore whether semantic categories (e.g. dog, flower, mountains etc.) can
also be visually segregated.
A semantically categorized database may contain images belonging to a domain or
spread over multiple domains. It becomes difficult for a common user to search suchdatabase if the nature of its classification or the exact semantics required for the
search is unknown. For example, medical terminology is an obvious choice forcategorizing medical images but it is very difficult for a common user to understandsemantics of these categories and hence finding proper keywords to search the
database. In another scenario, a categorized database may have ten categories
corresponding to dog based on their breeds, tail, coat etc. Looking at the image of awhite dog with black spots, a user may not exactly know its breed name i.e.
‘Dalmatian’. The user can derive such knowledge (semantics) by our approach.
Proposed approach utilizes visual features of a categorized image database of any
depth and height to determine semantics of images. Huge size of the search spacedemands algorithms which keep only the desired categories/subcategories inconsideration during search. A novel ‘Branch Selection Algorithm’ has been designed
and tested on a large hierarchical image database.

4

Related Databases and Database used for Experimentation

The nature and scope of image data influences the performance of retrieval
algorithms. For decades, in the absence of standard test data, researchers used selfcollected images to show their results. Many domain specific and uncategorized
databases came into existence lately for example, WANG, UW, IRMA 10000,
ZuBuD, and UCID [3]. Some more challenging datasets are Caltech 101/256 [7],Coral Image, Tiny Image, ESP, LabelMe, Lotus Hill, and ImageNet [12].
A publicly available, densely populated, and semantically organized hierarchicalimage database covering a wide range of domains was required for experimentation.
With large number of images for nearly all object classes, ImageNet serves the
purpose. Built upon the backbone of the WordNet structure, a subset of ImageNet
2011 Winter Release given in Table 1 is used for experimentation. A category in
ImageNet corresponds to a synonym set (synset) in WordNet. Fig. 1 shows some
representative images of ImageNet.

Fig. 1. A snapshot of Flower and Tree subtrees of ImageNet 2011 Winter Release.
Table 1. Subset of ImageNet database used for experimentation.
Subtree
Width
Depth
Animal99Appliance
44
Fabric2
5Flower
9
3
Fruit
6
5
Geological Formation
55Person12
4Sport, Athletic
54Structure
6
6
Tree
7
6
Vegetable
6
5
Total (on an average 910 images per synset)

5# of Synsets
32
29
12
24
42
50
34
23
36
42
41
365

# of Images (K)
38
32
11.5
26
30.5
55
16.5
30.5
33
24
35
332 K

Methodology

The work visualizes categories/subcategories of a semantically categorized image
database as nodes in the image tree. The flow of execution shown in Fig. 2 starts with
an offline extraction of visual features of images. Visual features of images belonging
to a node form visual signatures of that node. On the basis of the distance between
query image and visual signatures of nodes, the Branch Selection Algorithm selects
some subtrees to search. This search space is further reduced by pruning algorithms.
Retrieval module assigns semantics of the nodes at lower distances to the query
image. The proposed system supports both types of searches, i.e. aimed search to geta specific semantic; and category search to find a group of similar semantics.

Fig. 2. Work flow of the proposed system.

5.1

Feature Extraction Techniques

Conventionally, color, texture, and shape features are used to measure visual
similarity of images. The combination of these features gives better results [13].
Color Features. Color is one of the most widely used low-level visual features. It is
invariant to size and orientation of image [2]. It shows the strongest similarity tohuman eye [14]. Color histogram is the most commonly used representation. Various
versions of histogram e.g. cumulative histograms, quantized color space histograms
have been proposed [3, 15-16].
This work uses a histogram with perceptually smooth color transition in HSV color
space [17]. When applied on an image as a whole, the Global Color Histogram (GCH)feature is obtained. Five color histograms corresponding to five regions (central
ellipsoidal region and four surrounding regions) are concatenated to form a Local
Color Histogram (LCH) feature. In general, GCH and LCH are represented as (1).
 = ℎ , ℎ , … , ℎ  ,, … ,  ; ℎ,  ℎ     .

 = ℎ , ℎ , … , ℎ ,  ,  , … , 
, … ℎ , ℎ , … , ℎ  ,,(1)

 , … ,   .

Another popular color feature is a statistical model of color representation [18-19].
Color distribution of each channel of an image is uniquely characterized by its three
central moments i.e. average (Ei), variance (σi) and skewness (si) as given in (2).






)

!
!
*
 = ∑!
#$  # , % = & ∑#$ # −   (   = & ∑#$ # −   ( .
!

!

!

(2)

pij = value of ith color channel at jth image pixel and N = number of image pixels.
Images are compared by taking a weighted sum of differences of corresponding color
moments. Similarity between two images with r color channels and color moments(Ei1, σi1, si1) and (Ei2, σi2, si2) is given in (3).

+,+ = ∑0$ -  |  −  | + - |%  − % | + -  |  −  | .

(3)

where, w56 ≥ 0 is speci=ied by the user [19].

Similar to histogram, Global Color Moment (GCM) and Local Color Moment(LCM) features of an image in HSV color space are obtained as shown in (4).
H =  , % ,  ,  , % ,  ,  , % ,  ; H =  , … ,  , … ,  , … ,   .

(4)

Texture Features. Texture captures the information of patterns lying in an image. An
image may contain textures of different degrees of detail. Grey level co-occurrence
matrices (GLCM) and Tamura Features arepopular single scale texture features.
Multi-resolution texture features include Pyramidal Wavelet Transform (PWT), TreeStructured Wavelet Transform (TSWT), Discrete Cosine Transform (DCT), Gaborfilters, and ICA Filters [14]. The most frequently used Gabor filter is given by (5).
IJ,  = K



LMN MO RSTS

P J Q− K S + SPU + 2WXYJ .
MO

MO

(5)

I+Z J,  =  [+ IJ \ ,  \ ; ],  = , ] = 0,1, … , ^ − 1,

J \ =  [+ J _`θ + y sinθ,  \ = [+ −J  θ + y cosθ .

Suitable dilations and rotations of the Gabor function g(x,y) through thegenerating function gmn give a self-similar filter dictionary. Here θ = nπ/K, K = total
number of orientations, S = number of scale, a = (Uh/Ul)-1/(S-1). Uh and Ul are upperand lower centre frequencies of interest [20]. This work uses Gabor filter with fourscales and six orientations. For retrieval purposes the most commonly used measures
are mean µ mn and standard deviation σmn of the magnitude of the wavelet transform
coefficients. The resulting Gabor Texture (GT) feature vector is given in (6).
d = e , % ,

= 1,2,3, … ,24(6)

Shape Features. Shape features are powerful descriptors in image retrieval. Generic
Fourier Descriptors, Zernike and Pseudo Zernike Moments, and Wavelet Descriptors
are some popular representations [14]. Recent researches focus on computationallyefficient local image descriptors. Scale Invariant Feature Transform (SIFT) extract
large number of keypoints from image that leads to robustness in extracting small
objects among clutter [8, 21]. This work uses SIFT with 4 octaves and 5 levels. Kmeans clustering forms 32 clusters per image [3]. For each cluster, count, mean andvariance form a SIFT Shape (SS) feature vector given in (7).
hh = ij , ij , … , ij , klj, , lj, , … lj ,  m, jj, , jj, , … jj ,   .

(7)

5.2

Construction of Visual Signature of a Node/Category

To correlate low-level visual features and high level semantics of images belonging toa node, a visual signature is attached to each node. Feature vector of an image is acombination of GCH, LCH, GCM, LCM, GT and SS. Mean feature vectors of all theimages in a node, GCHmean, LCHmean, GCMmean, and LCMmean, GTmean, andSSmean form its visual signature. To get the semantics of an image, Branch Selection
and Pruning algorithms make use of the similarity measures summarized in Table 2.
Table 2. Visual signatures and similarity measures.
Visual Signature
GCHmean, LCHmean
GCMmean, LCMmean
GTmeanSSmean5.3

Similarity Measure
Vector Cosine DistanceCity Block Distance
Euclidean Distance
Earth Mover’s DistanceBranch Selection Algorithm

The work proposes a novel Branch Selection Algorithm given in Fig. 3.
Steps to find the subtrees, semantically similar to query image, at each level are as follows:
Step 1: Calculate feature vector of the query image.
Step 2: Let there are N nodes at this level. Calculate the distance of query image with N
nodes. For each feature, select n subtrees (n ≤ N) having minimum distance fromthe query image. This results in three lists, one corresponding to each feature,containing n entries. It gives rise to any of the three possibilities:
a. If subtree X is 1st choice in all the three lists, then select only this subtree forsearch. As root of this subtree X has the closest distance with query image withrespect to all the feature vectors considered. Go to Step 3.
b. If subtree X is 1st choice for any two lists, then select this subtree X for search.
In addition,i. Select (n-1) more subtrees having maximum frequency of appearance in the
two lists where X is 1st choice, and go to Step 3. In case, subtrees have same
frequency then go to Step (ii).
ii. Select one/more subtrees which have minimum sum of distances based on
all 3 features.
iii. Go to Step 3, if (n-1) subtrees are selected by now, otherwise go to Step (i).
c. If 1st choice of subtrees for all 3 lists is different, then select top n subtrees
based on the maximum frequency of their appearance in these 3 lists. In case ofa tie, select one/more subtrees which have minimum sum of distances based on
all 3 features.
Step 3: Repeat Step 2 for subtrees at every level.
Fig. 3. Branch Selection Algorithm

Branch Selection Algorithm selects a few subtrees (n) out of ‘N’ available at the firstlevel of the image tree. Only limited nodes that belong to these n subtrees aresearched to find semantics of the query image. The algorithm aims to reduce thesearch space as much as possible, without compromising the accuracy of the system.
The sum of distances based on GCH, LCH, GCM, and LCM is color distance.
Distance based on GT is texture distance, and sum of distances based on SIFT Mean
and Variance is shape distance. The algorithm prepares three lists corresponding tothese distances and ‘adaptively’ selects a branch.
Performance of the system greatly depends on the value of n chosen. Experimental
results for n=N/4, allows 75% pruning of the actual search space in terms of subtrees.
Initial pruning for more than this results in rejection of the target subtree most of thetime and therefore it is not fruitful to generate further results on its output.
An output of this algorithm for n=3 is shown in Fig. 4, where query image“n00450866_898” has been taken from “pony-trekking” synset. At the first level 11
subtrees are used for experimentation. The algorithm selects 3 subtrees (concepts) i.e.
Geological Formation, Tree, and Sport, Athletic. At the subsequent levels, synsets ofthese three high level semantics are chosen to get the complete search space for thisquery image. In this case, algorithm selects only 51 synsets out of the total 365
synsets in the image tree. Thus search space is reduced by 86% w.r.t. number ofsynsets to be searched, still keeping the desired subtree in consideration.

Fig. 4. Output of the Branch Selection Algorithm (n=3) for query image “n00450866_898”.

5.4

Pruning Algorithms

Branch Selection Algorithm applied on an image database results in any number ofnodes depending on the height and width of the n subtrees chosen in its step 2.

Pruning of this search space would further improve the performance. This pruning
helps in retaining good nodes while discarding the bad nodes of a selected subtree. A
good node is the one that lies on the path leading to the node containing images
semantically similar to the query image, while a bad node leads to either a different
path in the same subtree or a different subtree. Ideally, bad nodes and their subtrees
are to be pruned.
Goodness of a node is tested in terms of distances explained later in this section. In“strict pruning”, the whole subtree is pruned if its root fails to prove itself good.
While developing pruning approaches, it is observed that often a particular node on
the path does not fulfills the criteria of being good one but the query image belongs tosome lower level node of that path. Based on this observation, a “soft pruning” isproposed, which removes only the so-called bad node from the path and not the entire
subtree following it. The children of this bad node become the children of its parent.
Fig. 5 explains these approaches with the same query image n00450866_898. Strictpruning shown in Fig. 5(a) loses the target synset “pony-trekking” because its parent“riding” fails to prove itself good. A less restricted soft pruning approach shown in
Fig. 5(b) preserves the target synset even if its parent is being neglected. This lessrestrictive approach for pruning is followed in this work.

Fig. 5. (a) Strict Pruning. (b) Soft Pruning Approaches (gray nodes are the pruned ones).

Fig. 6 shows proposed pruning algorithms working on the distances corresponding
to the dominant visual feature. Dominant feature of a subtree is the feature which
gives top rank to this subtree. Dqsi is the distance (already calculated) between query
image and ith node of the subtree (having Ns nodes) w.r.t. dominant feature. Dmean
and Dmed are mean and median of Dqs. If a subtree having 10 nodes is given top
ranking by texture feature, then mean and median of the GT based distances between
query image and each of these 10 nodes are calculated. Additionally, extended mean
distance (Dmeanx) and extended median distance (Dmedx) are calculated as shown in(8). Dmeanx is the sum of Average Absolute Deviation (AAD) in Dmean. Dmedx isthe sum of Median Absolute Deviation (MAD) in Dmed.AAD and MAD are lessaffected by extreme observations than are variance and standard deviation [22].
!

s|no − n]|⁄pq  .
n]J = n] + ∑ $n]J = n] + ] |no − n]|

(8)

Fig. 6. Pruning Algorithms.

In the quest of good pruning algorithms, the four possible combinations shown in
Fig. 6 are exhaustively tested. The performance of pruning algorithms is judged on
two parameters: The number of nodes retained to be searched in retrieval module andthe ability to preserve the nodes appearing in the path ending at the target synset. The
nodes retained by pruning algorithm are used to assign semantics to the query image.

6

Experimental Setup

The most common computing facility consisting of a PC with Intel Core 2 Quad
processor, 8GB RAM and 500GB hard disk is used to get a fair idea about theperformance of proposed algorithms. All experiments are performed on a subset of
ImageNet shown in the Table 1. A set of query images is formed by automatically andrandomly selecting 5% of images from each synset. Query images are taken from
ImageNet database only because the attached semantic hierarchy with images helps toautomate performance analysis of the branch selection and pruning algorithms. No
manual intervention is required as human subjectivity may affect the understanding ofcorrelation between visual and semantic similarity.
Feature vector of images and visual signatures of nodes in the database are
generated through an offline procedure. During experimentation new images are notinserted in the database. In real life scenario, database may be kept in the updated
mode. This insertion intensiveness can be easily handled in the online version.
Insertion of an image requires its feature extraction and re-computation of the visualsignature of the node to which this image is added.

7

Results and Discussion

The following discussion establishes a correspondence between visual similarities andsemantic similarity in a semantically categorized hierarchical image database.

7.1

Performance of Branch Selection Algorithm

Branch Selection Algorithm prunes the search space but the precise selection of target
subtree based on the query image is to be ensured. This selection is expressed in termsof ‘Precision’ that denotes the selection of target subtree in terms of percentage. The
graph in Fig. 7(a) shows the performance of Branch Selection Algorithm on 11
subtrees of ImageNet (Table 1) for n=3. This prunes the search space by 75%. Out of
11 hierarchies tested, 9 give more than 50% precision, while precision of 70% ormore is achieved for 6 hierarchies. The algorithm out-performs if a query image isfrom ‘Appliance’ (94%), but opposite is the case if it is ‘Fabric’ or ‘Sports, Athletic’(30%). This happens due to the nature of the images that constitutes these categories.
Fig. 8 gives a glimpse of some of the images at the top level categories for these
synsets. Appliance synset contains visually as well as semantically closer images,
while ‘Fabric’ or ‘Sports, Athletic’ consist of images poorly related on the semantics.
Visual signatures of nodes having dissimilar images do not represent these nodes
well. This greatly affects the performance of the algorithm. Average precision @ 3
over all 11 branches is 65.36%, but on removing the two outliers i.e. ‘Fabric’ and‘Sports, Athletic’, it becomes 73.22%, which is fairly acceptable. The target subtree isselected for approximately 75%, while pruning the search space by the same amount.

Fig. 7. Performance of (a) Branch Selection Algorithm. (b) Pruning Algorithm.

Fig. 8. Some representative images of three categories

The execution time of the algorithm is significantly affected by the width anddepth of the subtrees selected at level 1. If the width of subtrees is more, then it would
lead to selection of more subtrees and searching of; while higher depth means moreiteration. On an average, execution time of the algorithm is 70 sec. For an online
search this time is high but considering the computational facility used and theabsence of an appropriate indexing of image features with this size of database,results are encouraging. In the real time environment algorithm would be executed at
the server end with indexed feature vectors, which will significantly reduce the time.
7.2 Performance of Pruning Algorithms
It is desirable to have pruning algorithms with high pruning percentage and highprecision. It is difficult to achieve high precision with high pruning percentage as they
are inversely proportional to each other. The best algorithm would be the one thatgives the highest pruning percentage with the desired precision. Performance of only
three pruning algorithms is shown in Fig. 7(b) as they maintain good precision. AND
operator is more restrictive and reduces search space significantly but results in poor
precision. OR operator is less restrictive but improves precision. Ext AND algorithm
seems to be the best with 95% precision and 26% pruning. The time required forPruning Algorithms depend totally on the output of the Branch Selection Algorithm.
7.3 Semantics Assigned to Query Images
The query image is given the semantic of the nodes that are closer to it. Table 3 shows
the output of branch selection and Ext AND pruning algorithmon query images.
Table 3. Images along with the semantics assigned to them by proposed approach.
ImageNet
Semantic
AnimalAppliance

FabricPerson

Query
Image

Proposed Semantics
GeneralSpecific
AnimalLive stock,
Vegetable
Ravine,Geological Insectivore,
Draw
Formation
Clothes
dryer,
Appliance
Refrigerator,
Structure
Coffee maker,
PersonElectric range
Hand
towel,
FabricViscos rayon,
Fruit
Towel,
Structure
Honeydew
Optimist,
PersonPersonification,
Appliance
Neutral,Vegetable
Refrigerator

Query
Image

Proposed Semantics
GeneralSpecific
AnimalTreeSportsUngulate,
Pachyderm,
Animal, Ming
tree

Deep freeze,
Appliance Clothes dryer,
Oven, Wringer
FabricRayon, Fabric,
Appliance Towel, Ponytrekking
Sports
PersonFruitSportsNeutral, Masterof ceremonies,
Entertainer,
PersonThe output shows the top four semantics assigned to the query image. A query image
from any category; say Person, retrieves not only a general semantic ‘Person’ but also
a number of specific semantics like ‘Optimist, Personification, Neutral’, etc. Presenceof misclassified images in the database adversely affects the performance of proposed
algorithms.
Table 4 lists some conflicting images in ImageNet. For example, the first image in
the Table 4 belongs to ‘animal’ category while visually it seems to be a ‘structure’.
Proposed approach keeps it closer to the ‘structure’ semantics. The proposed
approach also helps to identify such cases and reclassification of these images willfurther improve the performance.
Table 4. Some misclassified images and their correct classification by the proposed approach.

Image

Proposed Semantics

ImageNet
Semantic

GeneralSpecific

AnimalStructure, Animal, Vegetable

Parapet, Otter shrew, Support,
Elephant

FruitTree, Sports, Flower

Gum tree, Gymnospermous,
Conifer, Eucalyptus

Fruit

Tree

Gymnospermous, Gum tree,
Rose gum, Tree

FlowerTree, Geological Formation,
Vegetable

Ravine, Forest red gum, Rosegum, Eucalyptus

7.4 Other Issues
Size of the Visual Signature of a node. The size of the visual signature of a node
although large for an online application, but the algorithms assign efficient semantics
to the images. In future, efforts would be made to obtain compact visual signatures.
Lack of Comparative Evaluations. As most of the available image databases are flatin nature, the performance of proposed algorithms cannot be compared. Due to lack ofhierarchy, subtrees selected by the Branch Selection Algorithm contain only a single
node, which serves as both the root and the leaf. Pruning algorithms are also
insignificant for flat structures. Further, most of the work done in this field is based on
the personal databases and thus, it is not possible to get the results of the proposed
algorithms on those databases.
In the present work, for the purpose of comparison, WANG database is categorizedat the top level. Table 5 shows the performance of the proposed Branch Selection
Algorithm on WANG and compares it with other related work. It gives an overall
precision of 94.2% with 75% reduction in the search space. As a result the retrieval
process is much faster in comparison to other approaches.

Table 5. A comparison on WANG database using average precision values.
Category
Database to
be searched
AfricaBeach
Bus
Dinosaur
Elephant
FlowerFood
HorseMonument
Mountain
Average8

Proposed
Approach

F. Malik et al.
[23]

R. Gali et al.
[24]

P. Kinnaree etal. [25]

25%

100%

100%

100%

0.93
0.9
0.96
1
0.96
0.97
0.9
0.95
0.9
0.95
0.942

1
0.58
0.61
0.71
0.49
0.58
0.48
0.72
0.57
0.47
0.621

0.76
0.587
0.963
1
0.741
0.945
0.733
0.941
0.714
0.457
0.7841

1
1
1
1
1
1
1
1
1
1
1

Conclusion and Future Scope

The paper discusses an open ended problem of semantic gap and proposes somealgorithms to correlate visual and semantic similarity. The algorithms are developed
for semantically categorized image database. The experiments show that visualfeatures based on the adaptive combination of multiple low level features of image
may serve well for a semantically categorized large image database. It shows that if
categorized properly, low level features of the images can be combined with theirsemantics. The selection of good nodes by proposed algorithms ensures better
performance of the system. Derived semantics can be used for effective image
retrieval as a future research. Proper indexing of visual signatures can significantly
reduce the time required for Branch Selection Algorithm. Inclusion of user feedback
will also enhance the performance of retrieval system.
Acknowledgements. The authors acknowledge the support provided by JSPS to carryout this work under JSPS Invitation Fellowship for Research in Japan (Long-Term).

References
1. Sternberg, R. J.: Cognitive Psychology. 5th edn. Wadsworth Cencage Learning, Belmont,
Calif. (2008)
2. Datta, R., Joshi, D., Li, J., and Wang, J. Z.: Image Retrieval: Ideas, Influences, and Trendsof the New Age. ACM Comput. Surv. 40(2) (May 2008) 5-60
3. Deselaers, T., Keysers, D., and Ney, H.: Features for Image Retrieval: An Experimental
Comparison. Inf. Retr. 11(2) (April 2008) 77-107
4. Liu, Y., Zhang, D., Lu, G, Ma W. Y.: A survey of content-based image retrieval with highlevel semantics. Pattern Recogn. 40(1) (Jan. 2007) 262– 282

5. Wang, H. H., Mohamad, D. and Ismail, N. A.: Semantic Gap in CBIR: Automatic Objects
Spatial Relationships Semantic Extraction and Representation. IJIP International Journal of
Image Processing 4(3) (July 2010) 192-204
6. Li, J., and Wang, J. Z.: Automatic Linguistic Indexing of Pictures by a Statistical ModelingApproach. IEEE Trans. Pattern Anal. Mach. Intell. 25(9) (Sept. 2003) 1075-1088
7. Fei-Fei, L., Fergus, R., and Perona, P.: Learning generative visual models from few training
examples: An incremental Bayesian approach tested on 101 object categories. Comput. Vis.
Image Underst. 106(1) (April 2007) 59-70
8. Lowe, D. G.: Object recognition from local scale-invariant features. In: International
Conference on Computer Vision ICCV 1999, Vol. 2 (1999) 1150–1157
9. Zhuang, Y., Liu, X., and Pan, Y. Apply Semantic Template to Support Content-based
Image Retrieval. In Proceeding of IS&T and SPIE Storage and Retrieval for Media
Databases. San Jose, California, USA. 23-28 January, 2000, 442-449.
10. Shotton, J., Winn, J., Rother, C., and Criminisi, A. Textonboost: Joint appearance, shapeand context modeling for multi-class object recognition and segmentation. In: European
Conference on Computer Vision, 3951 LNCS (ECCV06). 2006, 1-15.
11. Liu, Y., Zhang, D., Lu, G.: Region-based image retrieval with high-level semantics using
decision tree learning. Pattern Recogn.41(8) (Aug. 2008) 2554-2570
12. Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., and Fei-Fei, L.: ImageNet: A Large-Scale
Hierarchical Image Database. In: IEEE Conference on Computer Vision and Pattern
Recognition. (June 2009) 248-255
13. Wang, X. Y., Yu, Y. J., Yang, H. Y.: An effective image retrieval scheme using color,
texture and shape features. Computer Standards & Interfaces33(1) (Jan. 2011) 59–68
14. Vassilieva, N. S.: Content-based Image Retrieval Methods. Program.Comput.Softw.35(3)(May 2009) 158–180
15. Liu, G. H., Yang, J. Y.: Content-based image retrieval using color difference histogram.
Pattern Recogn. 46(1) (Jan. 2013) 188-198
16. Pandey, K. K., Mishra, N., and Sharma, H. K.: Enhanced of color matching algorithm forimage retrieval. International Journal of Computer Science Issues 8(3) (May 2011) 529-532
17. Sural, S., Qian, G., and Pramanik, S.: A Histogram with Perceptually Smooth Color
Transition for Image Retrieval. In: 4th International Conference on Computer Vision,
Pattern Recognition and Image Processing. Durham, North Carolina (March 2002) 664-667
18. Shih, J. L., and Chen, L. H.: Colour image retrieval based on primitives of colour moments.
IEEE Proceedings on Vision, Image and Signal Processing.149 (6) (Dec. 2002) 370-376
19. Stricker, M., and Orengo, M.: Similarity of Color Images. In: SPIE Conference on Storageand Retrieval for Image and Video Databases III, Vol. 2420. San Jose, CA, USA (1995)
381–392
20. Manjunath, B. S., and Ma, W. Y.: Texture Features for Browsing and Retrieval of Image
Data. IEEE Trans. Pattern Anal. Mach. Intell. 18(8) (Aug. 1996) 837–842
21. Lowe, D. G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput.
Vision 60(2) (Nov. 2004) 91-110
22. NIST/SEMATECH:
e-handbookofstatisticalmethods
http://www.itl.nist.gov/div898/handbook/eda/section3/eda356.htm (2012)
23. Malik, F., and Baharudin, B. Quantized histogram color features analysis for image
retrieval based on median and Laplacian filters in DCT domain. In: International
Conference on Innovation Management and Technology Research (ICIMTR). Malacca,
Malaysia. 21-22 May 2012, 624-629.
24. Gali, R., Dewal, M. L., and Anand, R. S. Genetic Algorithm for Content Based Image
Retrieval. In: International Conference on Computational Intelligence, Communication
Systems and Networks (CICSyN). Phuket, Thailand. 24-26 July 2012, 243-247.
25. Kinnaree, P., Pattanasethanon, S., Thanaputtiwirot, S., and Boontho, S. RGB Color
Correlation Index for Image Retrieval. In Procedia Engineering (8) (2011), 36-41.