Journal of Robotics, Networking and Artificial Life, Vol. 1, No. 2 (September 2014), 160-163

Multiple-window Bag of Features for Road Environment Recognition
Shou Morita, Joo Kooi Tan, Hyoungseop Kim and Seiji Ishikawa

Department of Mechanical &Control Engineering, Kyushu Institute of Technology
Sensuicho 1-1, Tobata, Kitakyushu, 804-8550, Janan
E-mail: motira@ss10.cntl.kyutech.ac.jp, {etheltan, kim, ishikawa}@cntl.kyutech.ac.jp

Abstract
The idea of Bag of Features (BoF) is recently often employed for general object recognition. But, as it does not takepositional relations of detected features into account, the recognition rate is still not very high for practical use. This
paper proposes a method of describing the feature of an object by the BoF representation which considers positional
information of the features. Although the original BoF representation is applied to an entire image, the proposed
method employs multiple windows on an image. The BoF representation is applied to each of the windows torepresent an object in the image interested for recognition. The performance of the proposed method is shown
experimentally.
Keywords: Object recognition, bag of features, multiple-windows, VLAD, computer vision.

1. Introduction
The future of a mankind will be more and more
complicated and will definitely need the help of an
intelligent robot. Then the robot must be equipped witha strong ability of object recognition. On the other hand,a hand-held camera and a wearable computer system
which can recognize every object around a blind person
may help him/her a lot in living a daily life safely aswell as conveniently. Such a system again needs to havea strong ability of object recognition. Varioustechniques of object recognition have been developed todate. But such techniques normally employ the features
depending solely on the objects interested. The featurescommon to every object should be considered todevelop a general objects recognition method.
General objects recognition has been paid muchattention among computer vision researchers recently. A
well-known general object recognition technique is theidea of Bag of Features (BoF) [1]. It is a point-based
feature description method and describes every object
using a visual word dictionary. But it describes anobject as a set of feature points without considering
positional information. The positional information, or toknow how feature points distribute on an object, isactually important information for its recognition. An
idea of spatial pyramid matching (SPM) [2] is proposedin order to take positional information of feature points
into account. But it is not very effective, since themethod segments an image into 2n by 2n regions with nooverlap some of which may contain only thebackground of the image.
The present paper proposes a method of describing
the feature of an object by BoF representation whichconsiders positional information of the features.
Although the original BoF representation is applied toan entire image, the proposed method employs multiple
overlapping windows on an image. The BoF
representation is applied to each of the windows torepresent an object in an image. In this way, thepositional information among obtained BoFs isemployed for recognizing an object interested.
2. BoF and VLAD
The idea of BoF is overviewed in the first placefollowed by giving the concept of VLAD (Vector ofLocally Aggregated Descriptors) [3] proposed asanother representation of BoF.
Given an object image, the SIFT operator [4] isapplied to the image to derive a number of feature

Published by Atlantis Press
Copyright: the authors
160

S. Morita et al.

points on the object. The point is described by a 128dimensional vector. It is then projected into a 128dimensional feature space. A number of object images
are respectively transformed into the feature space as aset of feature points. The feature space then contains alarge number of the feature points, to which clustering isapplied to define some hundreds or thousands ofprominent classes. Let the number of the class in thefeature space be denoted by M. A class Ci is represented
by a feature vector vi (i=1,2,...,M). The feature space isthen defined as a visual word dictionary (VWD) by theset V={vi| i= 1,2,...,M}: Vector vi is referred to a visualword (VW) within the dictionary.
An object image is then described using the VWD.
Given an object image, the SIFT operator is applied tothe image and the feature points are extracted from theobject. Once they are projected into the VWD, they
distribute around the VWs which represent the object.
Let the number of the feature points distributing arounda VW vi be denoted by fi. This is actually the frequency
of a histogram of the chosen VWs. The object is thencharacterized by a M-dimensional feature vectorw = (f1, f2, ..., fM)(1)

This is called Bag of Features (BoF). An object isfinally identified by the BoF w.
Instead of using the frequency of the VWs, anotherdescription of an object [3] is proposed using a VLAD
(Vector of Locally Aggregated Descriptors). If a featurepoint extracted from an object image by SIFT is denoted
by x, the VLAD feature vector is defined bywi   ( x  v i )(2)Fig. 1. The proposed method also introduces VLAD for
describing BoF. This means to put more emphasis on
the VWs which have many feature points around themthan the frequency description.
Locating windows on an image has three variations:(i) Random location; Windows are randomly located on
an image;(ii) Considering feature points distribution: Arranging
windows more at the spots where many feature points
distribute;(iii) Combining (i) and (ii): Arranging windows
randomly under the condition that the spots where there
are many feature points have priority in the arrangement.
Among the above three strategies, (ii) is natural andreasonable, since the present object recognition isfeature-points-based recognition. It is, however,important to consider some randomness to escape fromover-learning against training images. This is the reason
why (iii) is considered. (i) is conducted for thecomparison with (ii) and (iii).
The size of the window and the randomness in thewindows placement is determined experimentally.
Let the number of the visual words in a window Wk(k=1,2,...,K) be denoted by Mk. In the original idea ofBoF, frequency in the BoF histogram is employed forthe components of feature vector wk. Instead of using
the histogram, the present method introduces VLAD for
describing a BoF. The magnitude of the VLAD wkm at
visual word vkm (m=1,2,...,Mk) in window Wk becomeslarge, if many feature points distribute close to vkm in abiased way. After all, the overall feature vector w isdefined byv ( x )i

After all, the VLAD expression provides a 128Mdimensional feature vector of the formw = (w1, w2, ...,wM)

w = (w1,w2, ..., wk, ...,wK),(4a)

wk = (wk1,wk2, ..., wMk).

(4b)

(3)

The magnitude of the component wi depends largely on
the feature points distributing around VW vi.
3. Multiple-window BoF
The proposed method puts K mutually overlapping
windows (W1, W2, ...,WK) on an image in order toconsider positional relation among extracted featurepoints, which is a strategy different from the original
BoF [1]. It also differs from SPM [2] in the overlap ofthe windows. The idea of multiple windows is shown inFig. 1. Multiple-windows set on an object image.

Published by Atlantis Press
Copyright: the authors
161

Multiple-window Bag of Features

The dimension of the feature vector w is therefore 128M
(M=M1+M2+...+MK).
The recognition strategy employs a nonlinear SVM
based on one-versus-rest classification.
4. Experimental Results
An experiment was conducted using the images in anroad environment. The employed objects for recognition
are a pedestrian, a traffic signal, a car and a bicycle (SeeFig. 2). They are all principal objects in the roadenvironment. The number of images used for thetraining of a SVM is 800; 100 with each object and 400
negative images. On the other hand, the number ofimages used for test is 500; 100 with each object and
100 negative images. Used PC has a 3.40 GHz CPU
with 8 GB memories.
The first experiment, Exp_1, was done to examine
the performance of VLAD. The original method andSPM employing a frequency histogram for BoF
representation were compared to those employing
VLAD for BoF representation. The result is given inTable 1.
In the second experiment Exp_2, the proposed
method employing multiple windows is examined its
performance with respect to the three cases of windows
placement explained in the former section; (i) placementat random, P_R, (ii) placement considering featurepoints distribution, P_FPD, and (iii) placement
considering feature points distribution and randomness,
P_FPD&R.
The experimental result is shown in Table 2. In
Exp_2, the number of VWs is parameterized and it
varies from 50 to 500 per window.
As seen in Table 2, the recognition rate is themaximum when P_FPD&R is adopted for windows
placement and 200 VWs are employed with everywindow. The third experiment, Exp_3, was conductedunder the employment of multiple windows with
P_FPD&R, 200 VWs with each window, and VLAD
expression for BoF. The result is shown in Table 3.
The number of used windows in Exp_2 and Exp_3
is 10 whose size is approximately 1/3 of the entire
image. They are arranged as shown in Fig. 3, in whichwindows are placed where many feature points exist in(a), whereas randomness is considered in addition to thefeature points distribution in (b). These windows
placements are kept unchanged through Exp_2 andExp_3.

(a)(b)
Fig. 2. Objects for recognition: (a) Positive samples; a pedestrian, a traffic signal, a car and a bicycle, (b) negative samples.

Table 1. Result of Exp_1: Original BoF & SPM without/with
VLAD.
Methods

O-O-BoF*

Rec. rate [%]**BoF+VLAD

63.0

67.4

SPM

SPM
+VLAD

69.8

72.8

* Original BoF** Recognition rate

Table 2. Result of Exp_2: Multiple window BoF.
P_R
No. VWs / window

50

100200500

Recogn' rate [%]

68.6

72.8

69.4

68.4

No. VWs / window

50

100200500

Recogn' rate [%]

71.0

69.4

69.4

68.8

No. VWs / window

50

100200500

Recogn' rate [%]

71.8

73.0

73.2

67.6

P_FPD

P_FPD&R

Table 3. Result of Exp_3: Multiple window BoF with VLAD.
Proposed method
Recognition rate [%]Published by Atlantis Press
Copyright: the authors
162

74.8

S. Morita et al.

(a)experiments to make the superiority certain, since thedifference is not very large.
In the employment of BoF, various weights could beconsidered including frequency of the SIFT featurepoints [1], VLAD [3], TF-IDF [5] and weighted BoF [6].
But they don't give very high recognition rates togeneral objects. One may need to improve this in someway.

(b)Fig. 3. The windows employed in Exp_2 and Exp_3: (a) Case(ii), (b) case (iii).

Table 4. Average recognition rate with respect to threestrategies of windows placement.
Ave. rec. rate*

P_R
69.8

P_FPD
69.7

P_FPD&R
71.4*Average recognition rate [%]5. Discussion
In Exp_1, the positive performance of the VLAD wasrecognized as shown in Table 1. The idea of VLAD is toput emphasis on the VW which is characteristic to aparticular object more than frequency, and it worked
affirmatively in the recognition of the 10 objects
employed in the experiment.
On the other hand, in Exp_2, the maximum
recognition rate was 73.2% when strategy (iii) in section
III was employed as windows location and 200 visual
words were used with each of the 10 windows. When asingle window, an image itself, is employed, which is
the original way of using the BoF, the recognition rate is
63.0% as seen in Table 1. Although SPM is a multiple
window method, the recognition rate is worse, 69.8%,
than the proposed method which is more flexible inwindows placement than SPM. This fact indicates theeffectiveness of the proposed use of multiple windows
in the BoF-based object recognition.
Finally, the proposed method, employing multiple
windows and VLAD expression, achieved 74.8% of therecognition rate as given in Table 3. This is the best
result at the moment.
As for the three strategies of windows placement,strategy (iii) seems to act better than the other two,
which is seen in Table 4. It shows average recognition
rates with respect to each windows placement in Table 2.
It may, however, be necessary to perform more

6. Conclusion
In this paper, multiple window bag of features wasproposed which considered positional relation of thefeature points on an object. For BoF representation, thevector of locally aggregated descriptors, VLAD, wasalso employed for recognizing ten familiar objects in a
traffic environment. By effective placement of themultiple windows on an image, 74.8% of recognition
rate was achieved, which is satisfactory for generalobject recognition. However, the research should becontinued to raise the recognition rate more in order toput the method into practical use.
Acknowledgement
This work was supported by JSPS KAKENHI Grant
Number 25350477.
References
1. G. Csurka, C.R. Dance, L. Fan, J. Willamowski, C. Bray,
Visual categorization with bags of keypoints, in Proc.
ECCV: Learning in Computer Vision (2004), pp. 1-22.
2. S. Lazebnik, Beyond bags of features: Spatial pyramid
matching for recognizing natural scene categories, in
Proc. IEEE Computer Vision and Pattern Recognition(2006), pp. 2169-2178.
3. H. Jégou, et al., Aggregating local descriptors into a
compact image representation, in Proc. CVPR (2010).
4. D.G. Lowe, Distinctive image features from scaleinvariant keypoints, International Journal of Computer
Vision, 60(2) (2004), pp. 91-110.
5. W. B. Frakes, R. Baeza-Yates, Information Retrieval:
Data Structures & Algorithms. Prentice-Hall,Engelwood Clifs, (1992).
6. T. Manabe, J. K. Tan, H. Kim, S. Ishikawa, Recognizing
indoor objects by weighted bag of features, in Proc. IEEE
Tensymp (2014). (submitted)

Published by Atlantis Press
Copyright: the authors
163