Journal of Robotics, Networking and Artificial Life
Vol. 7(1); June (2020), pp. 16–21
DOI: https://doi.org/10.2991/jrnal.k.200512.004; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Evaluation of the Relationships between Saliency
Maps and Keypoints
Ryuugo Mochizuki*, Kazuo Ishii
Center for Socio-Robotic Synthesis, Kyushu Institute of Technology, 2-4, Hibikino Wakamatsuku, Kitakyushu 808-0196, Japan

ARTICLE INFOABSTRACT

Article History

The saliency map is proposed by Itti et al., to represent the conspicuity or saliency in the visual field and to guide the selectionof attended locations based on the spatial distribution of saliency, which works as the trigger of bottom-up attention. If a certain
location in the visual field is sufficiently different from its surrounding, we naturally pay attention to the characteristic of visual scene.
In the research of computer vision, image feature extraction methods such as Scale-Invariant Feature Transform (SIFT), Speed-Up
Robust Features (SURF), Binary Robust Invariant Scalable Keypoint (BRISK) etc., have been proposed to extract keypoints robust
to size change or rotation of target objects. These feature extraction methods are inevitable techniques for image mosaicking and
Visual SLAM (Simultaneous Localization and Mapping), on the other hand, have big influence to photographing condition changeof luminance, defocusing and so on. However, the relation between human attention model, Saliency map, and feature extraction
methods in computer vision is not well discussed. In this paper, we propose a new saliency map and discuss the stability of keypoints
extraction and their locations using BRISK by comparing other saliency maps.

Received 26 November 2019
Accepted 19 February 2020
Keywords
Saliency map
binary robust invariant scalable
keypoint
keypoint stability© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
In recent years, many attempts have been done such as the selection of desired information in input information [1,2]. If attention
models can be constructed to select information, the intelligence
and awareness of humans can be implemented in computers.
According to Itti et al., saliency is defined as the property of images,
which triggers bottom-up attentions. Saliency occurs by the local
conspicuity over the entire visual scene [3]. In this model (Figure 1),
input image is decomposed into luminance, color, and orientation
components, then, each component is processed individually with
Gaussian filter.
Considering that the saliency map is applied to environment recognition by mobile robots, various changes in photographing condition are expected to affect the input image. The change affects
spatial frequency components of the image. If the spatial frequencychanges, the response of Gaussian filter also changes, then, theeffect reflects saliency map. Considering that the saliency map isused to select the keypoints of the image, changes in the saliency
map affect the results of feature selection, then, input data of detectors vary. Thus, recognition results are influenced according to thechange in photographing conditions.
For keypoint extraction, small influence is desirable in spite of thevariety of object size, angle and luminance. In case of the keypoint
application for object detection, repetitively extracted keypoints are
ideal to select.
Corresponding author. Email: rmochizuki@lsse.kyutech.ac.jp*

In our research, we propose a method for generating saliency maps,
which can absorb the effect of spatial frequency changes. If theparameters of the filters can be determined automatically, the effectof the spatial frequency change can be diminished in saliency maps
(Figure 2 Bottom). We evaluated the relationship between saliency
and keypoints.

2. RELATED WORK
2.1. Saliency Map
Itti et al. [3] simulated human eye movement, and expressed the result
as saliency maps (Figure 1). In the process of saliency map creation
input image is reduced by 1/2n and nine resolutions of the images are
obtained. The Center and the Surround can be obtained through thesmoothing operation by a common Gaussian filter. This signal process is similar to the different responses from fovea and its neighborin retina for the common stimuli. All the reduced images are enlarged
to the same size, and the across scale difference image of the two components is normalized and added to obtain a map for each component (i.e. Luminance, Color, Orientation). Saliency map is obtained
through the addition of all the maps of the three components.
According to Frintrop et al. [4], saliency map changes if the parameter of the Gaussian filters are changed. The ratio of filter parametersc/ss is crucial for the determination of saliency. Arbitral selection of
sc/ss enabled high granularity in saliency map. However, in Itti et al.
[3] and Frintrop et al. [4], the parameters cannot be adjusted depending on the variety of spatial frequency. As the result, saliency map canbe affected in the event of spatial frequency change (Figure 2 Top).

R. Mochizuki and K. Ishii / Journal of Robotics, Networking and Artificial Life 7(1) 16–21

17

Figure 3 | BRISK Keypoint Extraction Process.

Figure 1 | Itti’s saliency map.

Figure 2 | Adjustment of smoothing filters under saliency map generation
against changing spatial frequency.

2.2. Keypoint Extraction

Figure 4 | Overview of proposed saliency map method.

a

b

c

d

Keypoint extraction is often applied for object recognition tasks[5], image stitching tasks [6], etc. by robot vision. A keypoint hasa co-ordinate, a descriptor which explains brightness gradient in
the neighborhood. In the object recognition task, the database
image and the newly observed image are searched. Recently, scale-­invariant keypoint extraction methods have been proposed, such as
SIFT [7], and Binary Robust Invariant Scalable Keypoints (BRISK)[8] (Figure 3). As the result, the stability of object detection has been
improved. However, if photographing conditions (brightness ofthe environment, size of the observed object, focusing conditions,camera internal parameters, etc.) change, the number of extracted
keypoints changes significantly. Stably extracted keypoints aredesirable for the use of object detection tasks by robot vision.

color, and orientation components in advance. For each component, the Center and the Surround are generated by the combination of integral image and box filters. The parameter of the filters
are automatically adjusted so that the pixel values of the across scaledifference are maximized. The across scale differences of all threecomponents are merged to form saliency map.

3. PROPOSAL OF SALIENCY MAP

3.2. Decomposition of Input

3.1. Outline

We utilize CIE-Lab color system to simplify the difference of complimentary color channel. IL, Ia and Ib indicates luminance, color(Red-Green), color (Blue-Yellow) component, each other. For theobtainment of orientation component, Haar-Like Filters (Figure 5[9,10]) are convoluted on IL. The operations are expressed as
Equation (1).

In this research, we developed the theory of Frintrop et al. [4] tomitigate the effect of spatial frequency variation. The strategy is
automatic adjustments of sc and ss. In the saliency map generation
process (Figure 4), the input image is decomposed into luminance,

Figure 5 | Haar-like Filters. (a) 0°, (b) 90°, (c) 90°, (d) 135°.

R. Mochizuki and K. Ishii / Journal of Robotics, Networking and Artificial Life 7(1) 16–21

18

11	    Iq ( p p ) = I1 ( p p ) − I 2 ( p p ) − I 3 ( p p ) 
2
2

(1)

3.3. The Center and Surround4.1. Outline of the Experiment

We align two box filters FBs, FBc centered with point pρ as shownin Figure 6. The filters are used for convolution to generate theCenter and Surround. The filter widths WBs, WBc can be variable
up to Wpmax and fulfills WBs > WBc. This arrangement is same as
Mochizuki et al. [11].

3.4. Filter Adjustment
To obtain across scale difference of luminance, color components,
we maximize the pixel value of the difference Ics(pp) as in Mochizuki
et al. [11] by changing WBs(pp), WBc(pp) according to Equation (2)and Figure 6.
	  I( p p ) = max WBc ( p p ),WBs ( p p ) I cs ( p p )cs= max WBc ( p p ),WBs ( p p ) I c ( p p ) − I s ( p p )

4.2. Evaluation Function

On the other hand, for orientation component, to obtain across
scale differences the sizes of the Haar-like Filters are set to W,
Bs ( p p ), then, the filters are convoluted with IL. The responses ofW
Bc ( p p )the Center and the Surround are denoted as Iq_c (pp), Iq_s (pp). The
across scale differences of all directions are obtained and merged
to map MO_q (pp).

3.5. Saliency Map Generation
Map MC (for Color), and MO (for Orientation) are obtained by
Equations (4) and (5). Saliency map MSal is formed through themerge of MI, MC, MO with Eq. (6). The functions fmix, gmix, hmix for
merging maps can be selected arbitrarily.

MO = g (MO _ 0 , MO _ 45 , MO _ 90 , MO _ 135) 

M Sal = hmix (M I , MC , MO ) 

Figure 6 | Alignments of box filters.

In this experiment, we assume that the selected image keypoints
are used for object detection. Thus, we evaluate the relationshipbetween saliency MSal and feature stability FStb. Suppose the numberof small regions is Nq, FStb and MSal are expressed in line vector of
Nq dimensions. However, we treat MSal and FStb as two dimensions(Figure 7). Then, we calculate the relationship fi by obtaining innerproduct FStb · MSal. The saliency maps were generated by conventional methods (i.e. Itti method, VOCUS2) and our proposal tocompare fi. The source codes for the experiment are Simpsal [12]by Caltech for Itti method, and [13] for VOCUS2. We chose BRISK[8] as keypoint extraction method because descriptor is expressedin binary system. Such system is reported to require shorter time
for matching than SIFT [7]. Furthermore, the descriptor has properties of rotation and scale invariance.

(2)


.
Here, WBs (pp), WBc (pp) satisfies W
Bs ( p p ),WBc ( p p )MC = f (MCa , MCb ) 

4. EVALUATION OF THE RELATIONSHIPBETWEEN SALIENCY AND KEYPOINT
EXTRACTION

We consider two conditions of keypoints which have high stability
under photographing condition variety. First, the keypoints must
extracted at the same location. We define the property as repeatability.
Second, the descriptors must remain the same, that is, the similarity.
To evaluate keypoint stability, keypoint displacement have to beconsidered because of image flicker, resize of observed object size.
For example, the combination of the same keypoints is consideredas (I) or (II) in Figure 8 in different photographing condition.
We define a small region of Wq × Hq [Pixels] to search identical
keypoints.
Suppose Nkp,nq, i keypoints are extracted at nq-th small region
under i-th photographing condition, the variance skp, nq of

(4)
(5)
(6)

Figure 7 | Relationship between keypoint stability FStb and saliency MSal, i.

Figure 8 | Ambiguity in keypoint identification under changing
photographing condition.

R. Mochizuki and K. Ishii / Journal of Robotics, Networking and Artificial Life 7(1) 16–21

keypoint number is obtained by Equation (7). The average N kp ,nq(for N variations of a parameter) of extracted keypoints is obtainedby Equation (8).
		s kp ,nq =		 Nkp , nq=

1 N∑(N − N kp,nq )2 
N i =1 kp ,nq ,i

(7)

1 N
∑N 
N i =1 kp ,nq ,i(8)

rftr, nq is obtained by the normalization of skp, nq rftr, nq should be larger
if skp, nq is smaller as shown in Equation (9).
		 r

ftr , nq=1−

s kp ,nq
max nq s kp ,nq(9)

To obtain similarity, we select two keypoints from the same small
region (as seen in Figure 8) and different photographing conditions, then calculate Hamming distance between the two descriptors. To obtain average Hamming distance of all combinations of
the keypoint pairs, we use Equation (10). The similarity sDsc,nq is
calculated with normalization by Equation (11) so that the range
satisfies [0,1], and rDerr, nq is smaller as the distance is larger.
N −1    rDerr ,nq =N

min K ∑ l =1 ∑ m = l +1dH (dnq ,l , kl , dnq ,m, km )
 N
c 
 2

*

1

LD

19

4.3. Method
Figure 9 shows the experimental images (Lenna, Flower, Tree,Things). These images were selected in the database of Caltech[14] and Standard Image Data BAse (SIDBA) [15]. The spatial frequency spectrums of the images are shown in Figure 10. Lenna iswell known for test image to be used image analysis. Flower has
wider spectrum than Lenna with higher frequency component. As
well as the comparison of Things and Tree, Things has higher frequency component than Tree.
The photographing condition to adjust to vary extracted keypoint number is IMax,i/IMax,1 for luminance, WObj,i/WObj,1 for object
size, each other, whose range is from 0.5 to 1.0 with the step 0.1
of increase.
For changing WObj,i, we selected images of no white background,(i.e. Tree and Flower). We selected TFAST = 20 (TFAST : Threshold of
FAST Score [8]) and IMax,1 = 255 during the adjustment of IMax,i and(10)s.t. K = [k1 , k2 , k3 ,…, kN ], l , m = 1, 2,…, N i , l ≠ mi		 sDsc ,nq = 1 −rDerr ,nq
max nq rDerr ,nq



(11)

Keypoint stability of FStb,nq is calculated by the weighting of rDerr,nq
and sDsc,nq as shown in Equation (12).
		 FStb ,nq = wrftr ,nq + (1 − w )sDsc ,nq Figure 9 | Experimental images (*Cited from [14,15]). (a) Lenna.
(b) Flower. (c) Tree. (d) Things.

(12)

For the saliency MSal,i,nq, The maximum response of MSal is searchedwithin each small region. Maximum saliency and feature stability
are expressed as Nq dimensions of line vectors (denoted as MSal,i,FStb, respectively). fi is calculated as the angle between the two vectors [Equation (13)]. To be noted that rftr, sDsc are calculated only forthe regions where keypoints are extracted more than twice during
N variations of photographing conditions.
 M Sal ,i ⋅ FStb 
fi = cos −1 
 M Sal ,i FStb 
FStb = [FStb ,1 , FStb ,2 ,…, FStb ,nq ,…, FStb , N q ]



(13)

M Sal , i = [M Sal ,i ,1 , M Sal ,i ,2 ,…, M Sal ,i ,nq ,…, M Sal ,i , Nq ]The average f is obtained according to Equation (14).
		    f =

N

1
∑fi 
N i =1(14)

Figure 10 | Spectrum of spatial frequency which FFT algorithm returns
as spatial frequency components. u = v = 256 for DC component. The
brightness means the intensity of the component.

R. Mochizuki and K. Ishii / Journal of Robotics, Networking and Artificial Life 7(1) 16–21

20

WObj,i. As the setting of the proposal, for Setting 1, Wpmax = WIM /4
and for Setting 2, Wpmax = WIM / 2. WIM indicates the image width.
The resolution of the image is WIM × HIM = 512 × 512 [Pixel].

4.4. Results and Discussion
Tables 1 and 2 shows the relationship of MSal, r ftr and MSal, sDsc undervariable IMax, i, and WObj,i, each other. Flower has higher frequency
component than Lenna, and Things has higher frequency component than Tree.
We discuss the comparison of f under variable IMax, i. Referring toTables 1 and 2 for VOCUS2 and Itti, f was large for high spatial
frequency. While, for proposal, f is less influenced by spatial frequency change compared with conventional method.
Figures 11 (for Lenna) and 12 (for Flower) show the location of
keypoints on saliency map (Left), the histogram which indicatesthe response of saliency at the locations respectively. There is difference in frequency component, however, for the case of the proposal, the location of the peak in the histogram is higher saliency
than other saliency map. Thus the inner product in Equation (13)becomes larger. The change of WObj,i means the image reduction,then high frequency component increases. The proposal recorded
smaller f than others. As the results, the MSal of our proposal
turned out to have larger correlation in feature stability and saliency,
which means our proposal is more suitable for keypoint selection.

Figure 11 | Keypoint location (Lenna). (a) Itti. (b) Proposed method(setting 2).

Table 1 | Comparison of f (variable IMax,i)(a) w = 1 (FStb = rftr)
Image
Lenna
FlowerTreeThings

VOCUS2
1/10

5/10

18.13
29.00
21.85
31.24

20.63
29.64
24.34
29.77

Itti
31.61
41.39
33.97
28.36

Proposal
Set 1

Set 2

19.94
19.72
20.76
19.60

18.04
18.98
20.10
18.37

20.05
19.66
19.24
15.21

17.65
18.82
17.27
13.11

(b) w = 0 (FStb = sDsc)
Lenna
FlowerTreeThings

17.20
28.85
18.97
28.58

18.45
29.32
21.53
27.40

33.15
42.18
33.90
26.67

Bold type indicates the best value of f among the all settings mentioned in the table.

Table 2 | Comparison of f (variable WObj,i)Figure 12 | Keypoint location (Flower). (a) Itti. (b) Proposed method(setting 2).

5. CONCLUSION
In this research, we proposed method of saliency map generation which
consists of adaptive filter adjustment to spatial frequency to aim at preventing fluctuation of saliency caused by different input image andphotographing condition change. Our saliency map method turned
to be suitable for selecting keypoints less affectable by photographing
condition change compared with other conventional methods.

(a) w = 1 (FStb = rftr)
Image
Tree
Things

VOCUS2
1/10

5/10

22.62
28.74

23.22
25.83

Itti
34.06
29.56

Proposal
Set 1

Set 2

21.87
18.96

22.16
17.68

23.24
21.52

23.88
20.09

(b) w = 0 (FStb = sdsc)
Tree
Things

24.57
28.30

25.37
27.06

35.30
29.74

Bold type indicates the best value of f among the all settings mentioned in the table.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

REFERENCES[1] A.A. Dalve, S. Shiravale, Real time traffic signboard detection
and recognition from street level imagery for smart vehicle, Int. J.
Comput. Appl. 135 (2016), 18–22.

R. Mochizuki and K. Ishii / Journal of Robotics, Networking and Artificial Life 7(1) 16–21[2] H. Wang, X. Dong, J. Shen, X. Wu, Z. Chen, Saliency-based adaptive
object extraction for color underwater images, Proceedings of the
2nd International Conference on Computer Science and Electronics
Engineering, Atlantis Press, Paris, France, 2013, pp. 2651–2655.
[3] L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach.
Intell. 20 (1998), 1254–1259.
[4] S. Frintrop, T. Werner, G.M. García, Traditional saliency
reloaded: a good old model in new shape, 2015 IEEE Conference
on Computer Vision and Pattern Recognition, IEEE, Boston,
MA, USA, 2015, pp. 82–90.
[5] M. Aly, M. Munich, P. Perona, Bag of words for large scale object
recognition properties and benchmark, Proceedings of the Sixth
International Conference on Computer Vision Theory andApplications (VISAPP), 2011, pp. 299–306.
[6] M. Brown, D.G. Lowe, Automatic panoramic image stitching
using invariant features, Int. J. Comput. Vis. 74 (2007), 59–73.
[7] D.G. Lowe, Distinctive image features from scale-invariant keypoints, Int. J. Comput. Vis. 60 (2004), 91–110.
[8] S. Leutenegger, M. Chili, R.Y. Siegwart, BRISK: binary robust
invariant scalable keypoints, 2011 International Conference on
Computer Vision, IEEE, Barcelona, Spain, 2011, pp. 2548–2555.

21[9] P. Viola, M. Jones, Rapid object detection using a boosted cascadeof simple features, Proceedings of 2001 IEEE Computer Society
Conference on Computer Vision and Pattern Recognition(CVPR), IEEE, Kauai, HI, USA, 2001, pp. 511–518.
[10] R. Lienhart, A. Kuranov, V. Pisarevsky, Empirical analysis
of detection cascades of boosted classifiers for rapid object
detection, Joint Pattern Recognition Symposium, DAGM
2003: Pattern Recognition, Springer, Berlin, Heidelberg, 2003,
pp. 297–304.
[11] R. Mochizuki, S. Yasukawa, K. Ishii, Proposition of saliency
map based on the maximization of center-surround difference,
Proceedings of the International Conference on Artificial Lifeand Robotics, vols. 24, 2019, pp. 487–492.
[12] MATLAB Saliency –Caltech Vision- HomePage. Available from:http://www.vision.caltech.edu/~harel/share/gbvs.php.
[13] Saliency System VOCUS2 Universität Bonn Home Page.
Available from: http://pages.iai.uni-bonn.de/frintrop_simone/
vocus2.html.
[14] Caltech Database Homepage. Available from: http://www.vision.
caltech.edu/Image_Datasets.
[15] Kanagawa Institute of Technology. Available from: http://www.
ess.ic.kanagawa-it.ac.jp/app_images_j.html.

AUTHORS INTRODUCTION
Dr. Ryuugo Mochizuki

Dr. Kazuo Ishii

He received his master of engineering
degree at Kyushu Institute of Technology
in 2008. Then he has been involved in
spec test and design of Integrated Circuit
as a worker in Shikino High-tech CO.,LTD. up to 2013. His research topic
during the doctor course was the relationship between scale-invariant keypoint
extraction and saliency in the domain ofimage processing. He finished PhD degree in September 2019.

He is a Professor in the Kyushu Instituteof Technology, where he has been since1996. He received his PhD degree in engineering from University of Tokyo, Tokyo,
Japan, in 1996. His research interests
span both ship marine engineering andIntelligent Mechanics. He holds five patents derived from his research. His lab
got “Robo Cup 2011 Middle Size League Technical Challenge
1st Place” in 2011. He is a member of the Institute of Electricaland Electronics Engineers, the Japan Society of MechanicalEngineers, Robotics Society of Japan, the Society of Instrument
and Control Engineers and so on.