Takahiko Furuya and Ryutarou Ohbuchi, Visual Saliency Weighting and Cross-Domain Manifold Ranking for Sketch-based Image Retrieval, accepted as regular paper, Proc. Multi-Media
Modeling (MMM) 2014, Dublin, Ireland, January 8-10, 2014.

Visual Saliency Weighting and Cross-Domain Manifold
Ranking for Sketch-based Image Retrieval
Takahiko Furuya1, Ryutarou Ohbuchi1
1

Graduate School of Medicine and Engineering, University of Yamanashi, Japan{g13dm003, ohbuchi}@yamanashi.ac.jp

Abstract. A Sketch-Based Image Retrieval (SBIR) algorithm compares a linedrawing sketch with images. The comparison is made difficult by image background clutter. A query sketch includes an object of interest only, while database images would also contain background clutters. In addition, variability ofhand-drawn sketches, due to “stroke noise” such as disconnected and/or wobbly
lines, also makes the comparison difficult. Our proposed SBIR algorithm compares edges detected in an image with lines in a sketch. To emphasize presumed
object of interest and disregard backgrounds, we employ Visual Saliency
Weighting (VSW) of edges in the database image. To effectively compare thesketch containing stroke noise with database images, we employ Cross-Domain
Manifold Ranking (CDMR), a manifold-based distance metric learning algorithm. Our experimental evaluation using two SBIR benchmarks showed thatthe combination of VSW and CDMR significantly improves retrieval accuracy.
Keywords: sketch-based image retrieval, visual saliency detection, crossdomain matching, manifold ranking

1

Introduction

Querying modality is an issue central to image retrieval. Query-by-keywords may
be the easiest among query modalities for users to perform, but its retrieval accuracy
is not satisfactory. Images taken by phones and cameras generally do not have keyword tags, and if they do, specifying image content by keywords alone can be difficult. An alternative is content-based image retrieval, which uses an example image ora user-drawn sketch of some kind to query images. The first option, querying by image example, is not practical as a user often does not have an image appropriate forquery. The second option, Sketch-Based Image Retrieval (SBIR), has become a popular modality of late. Earlier SBIR systems, mostly likely using a mouse as an input
device, have drawn sketches by using simple geometrical primitives, such as straightlines, rectangles or circles for drawing, possibly combined with color specification.
Recently, with prevalence of touch- and pen-based devices, line-drawing sketch has
become the most popular querying modality for content-based image retrieval.
The issue in SBIR is an effective comparison between a line drawing sketch and a
2D image. The comparison should be robust against background clutter in the 2D
adfa, p. 1, 2011.
© Springer-Verlag Berlin Heidelberg 2011

image. It should also be robust against “stroke noise” of the line drawing sketch, thatis, wobbling and/or disconnected lines and difference in drawing styles.
A SBIR algorithm uses a common ground representation, either line/edge image[1-5] or gradient field image [6, 7], for sketch-to-image comparison. Then, one ormore image features are extracted from the common ground representation for comparison. Saavedra et al. [3] uses a set of local histograms of Canny edge orientations
as an image feature, while Eitz et al. [2] combines several orientation sensitive image
features including a variant of HoG [14]. The BF-GFHoG by Hu et al. [6][7] employs
gradient field image as the common ground and extracts HoG descriptors. Hundredsof HoG descriptors extracted from a gradient field image are integrated into a feature
vector per image by using the Bag-of-Features (BF) approach. To our knowledge, BFGFHoG is one of the best performing SBIR algorithms.
These algorithms perform well if database images do not have any backgroundclutter and if all the strokes in sketch images are stable and well-connected. However,images contain background clutters, and sketch strokes are disconnected or wobbly.
For example, images shown in Figure 1a contain leaves and clouds in their backgrounds, producing noisy Canny edge images in Figure 1c. A casual photograph taken
by using phones would contain even more clutters. Also, hand-drawn sketches containstroke noise. Comparison of a sketch having stroke-noise with an image having background-clutter would lead to low retrieval accuracy.
To suppress background clutter, visual saliency map is often employed. A visualsaliency map, which approximates visual attention of human, is used for image segmentation [16], object detection [17], and other applications. Recently, Yang et al. [8]proposed a graph-based saliency detection method called Manifold Ranking-based
Saliency Detection (MRSD). The MRSD yields better saliency maps than existing
methods. Our proposed algorithm uses the MRSD algorithm to emphasize foregroundobject of an image for background-clutter-free comparison of a sketch with an image.
To perform robust sketch-to-image comparison under the presence of sketch strokenoise, distance metric learning has been employed [9][12][15]. For example, Weinberger et al. [15] compares handwritten digits in a subspace where distances among
feature vectors are robust against stroke noise. Recently, we proposed Cross-Domain
Manifold Ranking (CDMR) algorithm for the task of sketch-to-3D model “crossdomain” comparison [12]. The CDMR is based on Manifold Ranking proposed by
Zhou et al. [9]. It learns distributions, or manifolds, of sketch features and 3D modelfeatures to improve distance (or similarity) computation among them. Our proposed
algorithm adopts the CDMR algorithm for improved sketch-to-image comparison.
In this paper, we aim for a SBIR algorithm that is robust against background clutters in images and robust against stroke noise in sketches. Figure 2 illustrates theoverall processing pipeline of the proposed algorithm.
To gain robustness against background clutters, the algorithm tries to emphasize aregion presumed to be foreground, i.e., the object sought for by the sketch query. The
emphasis, called Visual Saliency Weighting, is done by multiplying, pixel-by-pixel, a
Canny edge image with the visual saliency map computed by using the MRSD algorithm [8]. Figure 1d shows the examples of visual-saliency-weighted edge images.
Background clutter, i.e., edges of leaves and clouds, are effectively suppressed.

To gain robustness against stroke noise in sketch, the algorithm employs dataadaptive distance metric learning. Relevance values from a sketch to images in the
database are computed by using the CDMR algorithm [12]. The CDMR uses a Cross
Domain Manifold (CDM) constructed by using similarity values due to multiple,
heterogeneous features, each of which is may be optimized for a given comparisontask. A manifold of database images is computed (without edge detection) by using
the densely sampled SIFT [11] features. A manifold of sketches is computed by using
the BF-fGALIF [12] feature. Then these two manifolds in two different domains arecoupled into a CDM by using the BF-fGALIF feature. Once the CDM is constructed,
relevance values from a sketch to images in a database are computed as diffusion
distances over the CDM. The CDMR may be used in either semi-supervised, supervised or unsupervised mode. In this paper, we use the CDMR in unsupervised mode.
Note that the CDMR algorithm has a built-in ability to perform automatic query
expansion if an unlabeled (or labeled) corpus of sketches is available. Relevance diffused from a sketch query turns sketches similar to it into secondary sources of relevance diffusion, creating an expanded set of queries. Such a corpus of sketches may
be collected beforehand, or collected online over time as sketch queries are made.
We experimentally evaluated the proposed algorithm by using two sketch-based
image retrieval benchmarks by Hu et al., that are, the Flickr160 [6] and the Flickr15k[7]. Small but consistent improvement in retrieval accuracy is observed for bothbenchmarks when the VSW is applied. The CDMR improved retrieval accuracy verysignificantly. For example, for the Flickr160 benchmark, the combination of the VSW
and the CDMR produced MAP score of 72.3 %, which is about 18 % higher
than 54.0 % of the BF-GFHoG reported in [6].
Contribution of this paper can be summarized as follows; Proposal of a novel sketch-based image retrieval algorithm. It employs Visual Saliency Weighting (VSW) to suppress background clutter in images. The features extracted from edge images processed by VSW are compared against the feature of asketch query by using the Cross-Domain Manifold Ranking (CDMR), a distancemetric learning algorithm adept at comparing heterogeneous feature domains.
 Experimental evaluation of the proposed algorithm using multiple benchmarks,
which showed effectiveness of the proposed algorithm.

(a) input image(b) saliency map(c) edge image

Fig. 1. Examples of visual saliency weighting.

(d) saliency-weighted
edge image

Visual Saliency Weighting of edge image

Ranking of sketch and image features
sketchsaliency
detection(MRSD)

saliencymap

sketchdomaincross-domain
manifoldQuery
WSS
sketchsketch
similarity

2D imageedge imageedge
detection

saliencyweighted
edge image

weighting

2D imagedomainWSI

WII

sketch-2D
similarity
2D-2D
similarity

Fig. 2. Outline of the proposed method that employs Visual Saliency Weighting (VSW) of
Canny edge images and Cross-Domain Manifold Ranking (CDMR) of image features.

2

Proposed Method

2.1

Visual Saliency Weighting of edge image

Our algorithm converts (2D) images in a database into saliency-weighted edge images for comparison with sketches. The algorithm first computes Canny edge image
from the database image. We used the parameters found in [2] for Canny edge detection; low threshold and high threshold are set to 0.05 and 0.2, respectively. Then,edges due to background clutters are suppressed by using Visual Saliency Weighting(VSW), which multiplies, pixel-by-pixel, a visual saliency map with the Canny edge
image. Visual saliency map is computed by using the MRSD algorithm by Yang et al.
[8]. The MRSD algorithm computes visual saliency in two steps, assuming peripheryof an image as background.
In the first step, “background-ness” is propagated from image periphery at foursides of the image toward the center. The propagation is performed by using Manifold
Ranking (MR) [9] algorithm over a graph that connects, conceptually, all the pixels of
the image. However, MR on such a large graph would be too expensive. Thus, thepropagation is done on a simplified graph connecting superpixels, a cluster of neighboring pixels having similar color property. An edge of the graph connecting a pair ofsuperpixels is weighted by the similarity of the pair of superpixels. The similarity iscomputed from CIE LAB color features of each superpixel. We set the number ofsuperpixels per image to 200, the number used in [8]. After the first step, the nodes
having “background-ness” value lower than a threshold becomes foreground.
In the second step, “foreground-ness” is propagated from the foreground regions
over the graph of superpixels. After the second step, “foreground-ness” of eachsuperpixel becomes saliency value for pixels belonging to the superpixel. We blur thesaliency map before multiplying it with the Canny edge image.

Figure 1d shows examples of saliency-weighted edge images. Edges on background (e.g., edges on leaves and edges on clouds) are effectively suppressed.
2.2

Cross-Domain Manifold Ranking for sketch-based image retrieval

Cross-Domain Manifold Ranking (CDMR)
The CDMR [12] consists of two stages; Cross-Domain Manifold (CDM) generation stage and relevance diffusion stage.
In the CDM generation stage, a CDM matrix W is generated. W is a graph whose
vertices are the features from a sketch domain and an image domain. Given the number of sketches NS and the number of images NI, the matrix W has the size (NS + NI) ×(NS + NI).
 WSS
W  
 WIS

WSI 

WII 

(1)

The submatrix WSS having size NS ×NS is the manifold graph of sketch featuresgenerated by linking features of sketches produced by the BF-fGALIF [12] algorithm.
We will describe the BF-fGALIF in the following section. An edge of the graph WSS
connecting vertices i and j is undirected and has a weight, which is a similarity w(i, j)among the vertices. The similarity w(i, j) is computed by using the equation (2) after
normalizing the distance d(i, j) of features i and j to range [0,1].
 exp  d  i, j   
w (i , j )  
0


if

i j

(2)

otherwize

The submatrix WII having size NI ×NI is a manifold graph of image features. It iscreated in a similar manner as WSS. Features for image-to-image comparison arecomputed by using BF-DSIFT [13] algorithm.
The submatrix WSI of size NS ×NI couples two submanifolds WSS and WII that lie
in different domains, that are, sketch feature domain and image feature domain. To
compute a feature similarity w(i, j) between a sketch i and an image j, BF-fGALIF
features are extracted from a sketch image i and a saliency-weighted edge image generated from an image j. Finally, feature similarity w(i, j) is computed by using equation (2).
Submatrix WIS of size NI ×NS is a zero matrix as we assume no diffusion of relevance occurs from image features to sketch features.
In the relevance diffusion stage, ranking of images to a query sketch is done by diffusing relevance value from the query to the images over the CDM by using MR [9].
We normalize W for S by using the following equation;
S  D  1 2 WD  1 2(3)

where D is a diagonal matrix whose diagonal element is Dii   Wij . We use thejfollowing equation to find rank values in F given initial value, or “source” matrix Y;

F   I  S  Y
1(4)

Y is a diagonal matrix of size (NS + NI) × (NS + NI) that defines source(s) of relevance value diffusion. If a vertex i is the source of diffusion Yii  1 , and, if not,Yii  0 . In our case, the vertex corresponding to the query sketch becomes the sourceof diffusion. Fij is the relevance value of the image j given the sketch i. The higher the
relevance value Fij, the higher the rank of the image j in the retrieval result.
The ranking is very robust, as diffusion from the query sketch to the images occurs
via multiple paths. For example, the relevance value may first diffuse quickly throughsketches similar to the query before reaching to the images. In such a case, the CDMR
embodies a form of query expansion.
The parameter σ in equation (2) controls diffusion of relevance value across the
CDM. We use different values σSS，σII，and σSI for each of the submatrices WSS,
WII, and WSI as optimal value of σ depends on each submatrix. The parameter  [0,1) in equation (4) controls regularization.
Computing feature similarities for Cross-Domain Manifold.
Generation of the CDM W requires computation of similarities in the submatrices
WSS, WII, and WSI. In this section, we describe features used for computing eachsubmatrix.
Computing similarities for WSI .
Figure 3a shows a feature comparison pipeline for WSI. Given an image in a database, it is resized so that the shorter edge of the image becomes 256 pixels. A saliency-weighted edge image is generated from the resized image by the VSW described in
Section 2.1.
From each of the saliency-weighted edge images and the sketch images, we extract
BF-fGALIF feature, which is shown to be among the best methods for sketch-based
3D shape retrieval [12]. GALIF features [10] are extracted densely at regular grid
points on the image. GALIF feature captures orientation of lines and intensity gradient in the image by using Gabor filters. Parameters for GALIF we used in this paper
are the same as those in [12]. 1,000 to 1,500 GALIF features are extracted per image.
The set of GALIF features extracted from an image is integrated into a feature vector per image by using a standard bag-of-features (BF) approach. This integration
reduces cost of image-to-image matching significantly compared to directly comparing a set of features to another set of features. We used vocabulary size k=3,500 for
the experiments. We used k-means clustering to learn the vocabulary, and used kdtree to accelerate vector quantization of GALIF features into words of vocabulary.
A BF-fGALIF feature of a sketch image is compared against a BF-fGALIF featureof a saliency-weighted edge image by using Cosine distance.
Computing similarities for WSS.
Similarities for the submatrix WSS is computed, again, by the BF-fGALIF algorithm (Figure 3b). For each sketch image, BF-fGALIF feature is extracted after thesketch image is resized down to 256256 pixels. A distance between BF-fGALIF
features is computed using Cosine distance. All the parameters for the BF-fGALIF(i.e., parameters for Gabor filter, number of GALIF features per image, and vocabulary size) are the same as those used to compute WSI.
Computing similarities for WII.
Similarities for the submatrix WII is computed by the BF-DSIFT algorithm [13](Figure 3c). BF-DSIFT is extracted from grayscale images without edge detection.
Given an image, it is resized so that the shorter edge of the image becomes 256 pixels
as with computing WSI.
From each resized image, about 3,000 SIFT [11] features are extracted at densely
and randomly placed feature points on the image. SIFT has invariance against scaling,rotation, illumination changes and minor changes in viewing direction. The set ofabout 3,000 SIFT features are integrated into a feature vector per image by using a BF
approach. We used vocabulary size k=3,500 for the experiments. A distance between
BF-DSIFT features is computed using symmetric version of Kullback-Leibler Divergence.
compute VSW

extract BF-fGALIF

sketch2D imagecompute distance
 2
 
 4
1
 

weighted edge image

 0
 
 4
 3
 

sketch-2D image
similarity
BF-fGALIF
feature vector(a) Computing a sketch-2D image similarity for WSI.
extract BF-fGALIF

compute distancesketch

1
 
 5
1
 

sketch

1
 
 2
 4
 

Sketch-sketchfeature similarity
BF-fGALIF
feature vector(b) Computing a sketch-sketch similarity for WSS.
extract BF-DSIFT

compute distance2D image

 3
 
 3
1
 

2D image

 4
 
 3
0
 

2D image-2D image
feature similarity
BF-DSIFT
feature vector(c) Computing a 2D image-2D image similarity for WII.

Fig. 3. Feature comparison methods for generating CDM.

3

Experiments and Results

We experimentally evaluated effectiveness of weighting by the VSW of Cannyedge images and ranking by the CDMR for sketch-based image retrieval. We used
two sketch-based image retrieval benchmark databases; the Flickr160 [6] and the
Flickr15k [7], both by Hu et al. Figure 4 shows examples of sketch queries and retrieval target images for the two benchmarks.
The Flickr160 consists of a set of 25 sketch queries and a set of 160 retrieval target
images. Each of the set of sketch queries and the set of target images is partitioned
into 5 categories. The Flickr15k, which is a larger-scale version of the Flickr160,
consists of a set of 330 sketch queries and a set of 14,660 retrieval target images. The
set of sketch queries is partitioned into 33 shape categories (e.g., “round”, “heartshape”, etc.). The set of retrieval target images is partitioned into 60 semantic categories (e.g., “pyramid”, “bicycle”, etc.). A query in the Flickr15k could belong to multiple semantic categories, for example, a round-shaped sketch query belongs to suchsemantic categories as “moon”, “fire_balloon”, and “london_eye”.
We used our own implementations for the CDMR and the BF-fGALIF. The MRSD
[7] and BF-GFHoG [8] are computed by using original codes downloaded from respective authors’ websites.
Parameters σSS，σII，σSI and α for the CDMR are determined through a set of preliminary experiments so that the retrieval accuracy is the highest among the combinations of parameters we tried. Table 1 summarizes parameters for the CDMR used in
the experiments below.
We used Mean Average Precision (MAP) [%] and Recall-Precision plot for quantitative evaluation of retrieval accuracy.
Table 1. Parameters for the CDMR.
algorithms
BF-GFHoG
BF-GALIF
BF-GALIF(w)σSS
0.00750.00750.0075Flickr160
σII
σSI
0.02
0.020
0.02
0,075
0.02
0.075

α
0.95
0.95
0.95

σSS
0.04
0.04
0.04

Flickr15k
σII
σSI0.00750.05
0.00750.05
0.00750.05(a) Flickr160.

(b) Flickr15k.

Fig. 4. Examples of sketch queries and retrieval target images.

α
0.7
0.7
0.7

3.1

Effectiveness of edge weighting by VSW

Figure 5 shows the relationship between vocabulary size k and retrieval accuracy
for the Flickr160 and the Flickr15k benchmarks. In the figure, “BF-fGALIF” means
BF-fGALIF is extracted directly from Canny edge images, while “BF-fGALIF(w)”with “(w)” means that the feature is computed from saliency-weighted edge images
generated by the VSW. For both benchmarks, at almost all the vocabulary size, theBF-fGALIF(w) with saliency weighting produced MAP scores about 2 % higher thanthe BF-fGALIF without saliency weighting.
It can be concluded that edges due to background clutter are suppressed to certain
degree by the VSW, resulting in small but consistent improvements in retrieval accuracy.
BF-fGALIF(w)BF-fGALIFBF-GFHoG

60

50

BF-fGALIF(w)BF-fGALIFBF-GFHoG

20
MAP [%]

MAP [%]

55

25

15
10

45

5

40
0

1000 2000 3000 4000 5000
vocabulary size k(a) Flickr160

01000 2000 3000 4000 5000
vocabulary size k(b) Flickr15k

Fig. 5. Vocabulary size and retrieval accuracy. (Please note the difference in MAP scales.).

3.2

Effectiveness of ranking by CDMR

Table 2 shows ranking performance of the CDMR for the Flickr160 and the
Flickr15k benchmarks. In this experiment, the sketch-to-image comparison algorithm
for the submatrix WSI of the CDMR is selected from the following three; the BFGFHoG, the BF-fGALIF, and the BF-fGALIF(w). We fixed the sketch-to-sketch
comparison algorithm for the submatrix WSS to the BF-fGALIF. We also fixed theimage-to-image comparison algorithm for the submatrix WII to the BF-DSIFT.
For all the three sketch-to-image comparison algorithms we compared, that are, theBF-GFHoG, the BF-fGALIF, and the BF-fGALIF(w), the CDMR significantly improved their retrieval accuracy for the two benchmarks. In case of the Flickr160, theBF-fGALIF(w) feature using the CDMR produced the highest MAP score of 72.3 %.
It is about 19 % better than the BF-fGALIF(w) feature without using the CDMR. In
case of the Flickr15k, the BF-fGALIF(w) with the CDMR yielded MAP score of
22.4 %. This score is 6 % higher than the BF-fGALIF(w) without the CDMR.
Overall, gain in retrieval accuracy due to the CDMR is quite significant. Diffusionof relevance via multiple paths through the sketches and images of the CDM makes
the ranking robust against stroke noise and other variations in sketches. Also, diffu-

sion of relevance from the query sketch via sketches similar to the query toward target
images pushes retrieval accuracy up, as it works like an automatic query expansion.
Table 2. Feature selection and ranking accuracy by the CDMR.
algorithms
BF-GFHoG
BF-fGALIFBF-fGALIF(w)3.3

Flickr160
without CDMR
with CDMR
44.8
64.1
51.6
69.4
53.7
72.3

Flickr15k
without CDMR
with CDMR
9.5
15.4
14.3
20.6
16.5
22.5

Comparison with other algorithms

Table 3 shows comparison of retrieval accuracy using the Flickr160 and the
Flickr15k benchmarks. The table also lists MAP scores for algorithms reported in Hu
et al. [6][7] for comparison. Figure 7 shows recall-precision plots for the 6 algorithms
we compared. Every algorithm listed in Table 2 employs a set of local features tocompare images with sketches. Note that, in Table 2, MAP scores of the BF-GFHoG
are different among our own experiments and theirs found in [6][7]. This discrepancy
is due probably to the difference in distance metric; we used Cosine distance for feature comparison, while Hu et al. used Histogram Intersection.
Our proposed CDMR-BF-fGALIF(w), which employs edge weighting by the VSW
and ranking by the CDMR, did the best among 12 methods listed in Table 3, for bothbenchmarks. Our best performer, the CDMR-BF-fGALIF(w), produced MAP=72.3%
for the Flickr160 benchmark. In comparison, the MAP score of the BF-GFHoG is
54% [6]. When compared using the Flickr15k benchmark, the CDMR-BF-fGALIF(w)produced MAP score of 22.5 %. This score is about 10 % higher than that of BFGFHoG, 12.2%, reported in [7]. Most of the gain in retrieval accuracy comes from the
CDMR ranking, but contribution due to visual saliency weighting is consistent.
Recall-precision curves of Figure 6 also shows the advantages in retrieval accuracyof the CDMR-BF-fGALIF(w) over the other algorithm.
Table 3. Comparison of MAP scores [%] among several algorithms.
algorithms
BF-GFHoG
BF-fGALIFBF-fGALIF(w)CDMR-BF-GFHoG
CDMR-BF-fGALIF
CDMR-BF-fGALIF(w)
BF-GFHoG [6][7]BF-HoG [6][7]BF-SIFT [6][7]BF-SelfSimilarity [6][7]BF-ShapeContext [7]BF-StructureTensor [7]Flickr160
44.8
51.6
53.7
64.1
69.4
72.3
54.0
42.0
41.0
42.0

Flickr15k
9.5
14.3
16.5
15.4
20.6
22.5
12.2
10.9
9.1
9.6
8.1
8.0

CDMR-BF-fGALIF
BF-fGALIF

1

1

0.8

0.8
Precision

Precision

CDMR-BF-fGALIF(w)BF-fGALIF(w)

0.6
0.4
0.2

CDMR-BF-GFHoG
BF-GFHOG

0.6
0.4
0.2

0

0
0

0.2

0.4
0.6
Recall0.810(a) Flickr160

0.2

0.4
0.6
Recall0.81(b) Flickr15k

Fig. 6. Recall-Precision plots.

4

Conclusion and Future WorkIn this paper, we proposed an algorithm for Sketch-Based Image Retrieval (SBIR).
A challenge in SBIR is effective comparison of a line drawing sketch and a 2D image.
It should be robust against background clutter in the image. It should also be robust
against stroke noise in the line drawing sketch. Previous algorithms for SBIR convert
a 2D image into an edge image to compare it against a query line drawing sketch.
However, unnecessary edges due to background clutter in an image interfere with thefeature comparison between a sketch and the 2D image. Stroke noise such as disconnected or wobbly lines also makes the comparison difficult.
Our proposed algorithm first converts an image into edge image, and then performs
Visual Saliency Weighting (VSW) to suppress edges due to background clutters. To
effectively compare a sketch containing stroke noise with images, we employ a distance metric learning algorithm called Cross-Domain Manifold Ranking (CDMR)[12]. Our experimental evaluation using two SBIR benchmarks showed that the combination of the VSW and the CDMR significantly improves retrieval accuracy.
We are currently looking into the improvement of computational efficiency of the
CDMR algorithm, as the CDMR is expensive to compute for a large database.

References
1. A. Chalechale, G. Naghdy, A. Mertins, Sketch-based image matching using angular partitioning, IEEE Transactions on Systems, Man and Cybernetics, Part A, 35(1), pp. 28-41,
2005, (2005).
2. M. Eitz, K. Hildebrand, T. Boubekeur M. Alexa, Sketch-Based Image Retrieval: Benchmark and Bag-of-Features Descriptors, Visualization and Computer Graphics 2011,17(11), pp.1624-1636, (2011).
3. J. Saavedra, B. Bustos, An improved histogram of edge local orientations for sketch-based
image retrieval. LNCS 6376, Springer, pp. 432-441, (2010).

4. Y. Cao, C. Wang, L. Zhang, L. Zhang, Edgel index for large-scale sketch-based image
search, CVPR 2011, (2011).
5. K. Bozas, E. Izquierdo. Large scale sketch based image retrieval using patch hashing. In
Advances in Visual Computing 2012, pp 210-219, (2012).
6. R. Hu, M. Barnard, J. Collomosse, Gradient Field Descriptor for Sketch based Retrieval
and Localization, ICIP 2010, (2010).
7. R. Hu, J. Collomosse, A Performance Evaluation of Gradient Field HOG Descriptor for
Sketch Based Image Retrieval, CVIU 2013, (2013).
8. C. Yang, L. Zhang, H. Lu, X. Ruan, M. H. Yang, Saliency Detection via Graph-based
Manifold Ranking, CVPR 2013, (2013).
9. D. Zhou, O. Bousquet, T.N. Lal, J. Weston, B. Schölkopf, Learning with Local and Global
Consistency, NIPS 2003, (2003).
10. M. Eitz, R. Richter, T. Boubekeur, K. Hildebrand, M. Alexa, Sketch-Based Shape Retrieval, ACM TOG, 31(4) pp.1-10, (2012).
11. D. G. Lowe, Distinctive Image Features from Scale-Invariant Keypoints, IJCV, 60(2), November 2004.
12. T. Furuya, R. Ohbuchi, Ranking on cross-domain manifold for sketch-based 3D model retrieval, accepted as regular paper, Cyberworlds 2013, (2013).
13. T. Furuya, R. Ohbuchi, Dense sampling and fast encoding for 3D model retrieval using
bag-of-visual features, ACM CIVR 2009, Article No. 26, (2009).
14. N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, CVPR 2005,(2005).
15. K. Q. Weinberger, L. K. Saul, Distance Metric Learning for Large Margin Nearet Neighbor Classification, Journal of Machine Learning Research (JMLR) 2009, (2009).
16. R. Achanta, F. Estrada, P. Wils, S. Susstrunk, Salient region detection and segmentation.
International Conference on Computer Vision Systems 2008, (2008).
17. X. Hou, L. Zhang, Saliency detection: A spectral residual approach. CVPR 2007, (2007).
BF-GFHoG [6][7]




Query
(flower,sunflower)









Query
(Big Ben)









BF-fGALIF(w) 

CDMR-BF-fGALIF(w) 

BF-GFHoG [6][7]BF-fGALIF(w)CDMR-BF-fGALIF(w)
















 



Fig. 7. Retrieval examples for the Flickr15k.

