349

Sensors and Materials, Vol. 28, No. 4 (2016) 349–358
MYU Tokyo
S & M 1185

High-Speed Detection of Down Staircase for Visually
Impaired People Using Illuminance of Point Pattern
Yusuke Kajiwara,* Keisuke Ohshima,1 and Haruhiko Kimura1
College of Information Science and Engineering, Ritsumeikan University,
1-1-1 Noji-higashi, Kusatsu, Shiga 525-8577, Japan
1Graduate School of Natural Science and Technology, Kanazawa University,
Kakuma, Kanazawa, Ishikawa 920-1192, Japan(Received August 6, 2015; accepted January 22, 2016)

Keywords: phase only correlation, Hough transform, illuminance of point pattern, detection of down
staircase

Visually impaired people are limited in movement both indoors and outdoors. To solve thisproblem, existing studies propose a navigation system that leads visually impaired people on theroute to their destination by detecting obstacles on the route and tracking their position using
information and communication technology. Visually impaired people recognize their position
and the route to their destination by combining these positioning methods. On other hand, existingstudies propose detection of obstacles by image processing, but this method take a long time.
Hence, we have proposed the high-speed detection of a down staircase for visually impaired people.
The proposed method quickly detects the boundary line between the down staircase and floors using
the illuminance of a point pattern. Moreover, the proposed method recognizes the down staircase
quickly because it removes images that are clearly different from the down staircase and applies
image processing to the pedestrian route only. Experimental results have shown that the proposed
method can recognize a down staircase and border line with more than 90% accuracy. As a result,visually impaired people can safely go down a flight of stairs using this method.

1.

Introduction

Visually impaired people are limited in their movement indoors and outdoors because theycannot see obstacles before them or the route from a starting point to a destination. Under current
condition, visually impaired people follow a route with a guide dog and braille guidance blocks.
However, a guide dog is very expensive, and braille guidance blocks cannot indicate the current
position and the direction for reaching a destination to a visually impaired person.
To solve this problem, existing research(1–9) has proposed a navigation system that leads visually
impaired people on the route to their destination by detecting obstacles on the route and tracking
their position using information and communication technology.
Methods for tracking a person's position are mainly Global Positioning System (GPS),(1) WiFi
positioning,(2) and Bluetooth Low Energy (BLE) positioning.(3) GPS can measure a person’s
position outdoors with an error of 0.5 m. WiFi positioning and BLE positioning can measure aperson’s position indoors with error between 1 and 10 m. Visually impaired people recognize their*

Corresponding author: e-mail: kajiwara@de.is.ritsumei.ac.jp
ISSN 0914-4935 © MYU K.K.

350

Sensors and Materials, Vol. 28, No. 4 (2016)

position and the route to their destination by combining these positioning methods. On the other
hand, existing studies(4–9) have been able to detect obstacles such as a bumpy road and a flight ofstairs. Sadi et al.(4) detected an obstacle with head-mounted ultrasonic sensors. Mahmud et al.(5)detected a obstacle with the tip of a white walking stick mounted with ultrasonic sensors. Mustapha
et al.(6) detected an obstacle with the tip of shoes mounted with ultrasonic sensors and infrared
sensors. Valaminck et al.(7) detected a door and a flight of stairs with two cameras and stereo
matching. Aladren et al.(8) and Murakami et al.(9) detected a flight of stairs with RGB-D sensors
such as Kinect. To lead visually impaired people to a safe route, the navigation system must
recognize the shape and size of an obstacle. Moreover, a navigation system should be composed ofa sensor that is small and lightweight to minimize the burden to users.
Existing systems using ultrasonic sensors cannot determine the size and shape of obstacles.
A navigation system using a camera or RGB-D sensors can recognize the shape and size of an
obstacle. The camera is small and lightweight, but RGB-D sensors are heavier than a camera.
For these reasons, we have proposed the high-speed detection of a down staircase for visually
impaired people. Visually impaired people cannot safely go down a flight of stairs by the stereo
matching method because they have to prepare to go down the stairs immediately before placing afoot on the down staircase owing to a long detection time. In addition, it is difficult for the stereo
matching method to match images of a floor and a down staircase that are painted a single color
or a repeating pattern. Hence, the proposed method detects quickly the boundary line between adown staircase and floor using the illuminance of a point pattern. Moreover, the proposed method
recognizes quickly the down staircase by matching part of an image after removing images that are
different from the down staircase image by the Hough transform. The stereo matching method isa phase-only correlation (POC). The proposed method can recognize a floor and a down staircasethat are painted a single color or have a repeating pattern because the floor and the down staircase
are marked with a specific illumination pattern with a laser pointer.
We explain the proposed method in § 2. In § 3, experimental results are shown. In § 4, we
discuss the experimental results. In § 5, the conclusion and suggestion for future work are given.

2.

High-Speed Detection of Down Staircase Using Illuminance of Point Pattern

To support the movement of visually impaired people, existing studies detect obstacles by thestereo matching method. The stereo matching method can estimate the distance between a camera
and target objects because it applies triangular surveying to target objects in images obtained by
two cameras. However, if some objects have similar color and shape, the stereo matching method
frequently mismatches target objects in images. A floor and a down staircase may be painted asingle color or have the same repeating pattern. Hence, stereo matching frequently mismatches thefloor and the stairs. In addition, stairs detection methods have a long detection time because theyuse complex image processing such as the Hough transform and stereo matching. The proposed
method recognizes the down staircase images with a single color and repeating pattern by markinga point pattern on the floor with a laser pointer. Moreover, the proposed method recognizes thedown staircase quickly by multistep detection.
Figure 1 shows the processing flowchart of the proposed method. First, the proposed method
detects the boundary line between the down staircase and floor quickly using the illuminance of apoint pattern with a laser pointer. Visually impaired people are warned when the proposed method
detects a boundary line. The walking of visually impaired people becomes slow after a warning.

351

Sensors and Materials, Vol. 28, No. 4 (2016)

Fig. 1.

Multistep detection of down staircase.

Hence, these methods provide enough time to recognize the down staircase exactly. Second, theproposed method recognizes the down staircase quickly by removing images different from thedown staircase image using the Hough transform. Third, the proposed method recognizes the down
staircase exactly by stereo matching. The proposed method recognizes the down staircase quickly
because it removes images that are different from the down staircase and applies image processing
to pedestrian routes only.
2.1

Detection of the boundary line by the illuminance of point pattern

To enable a person to go down a staircase safely, the system should detect the boundary line
between the down staircase and floor from images that are filmed at a place far from the down
staircase. In this case, the camera cannot photograph the steps of stairs such as Fig. 2 because
the steps are concealed by the floor. Hence, the down staircase cannot be detected by the Houghtransform and stereo matching. From the inverse square law, illumination intensity is inversely
proportional to square of distance. Thus, when visually impaired people radiate visible rays at
the down staircase, illumination intensity of the middle floor becomes lower than the illumination
intensity of the floor around the feet. For this property, the proposed method quickly detects theboundary line between the down staircase and the floor using the illuminance of the point pattern.
Figure 3 shows the outline of the down staircase detection method using the illuminance of thepoint pattern. First, the system removes both ends of an image as an excepted area. That width is
1/5 of the width of the image, and the height is the same as the height of the image. Second, theproposed method divides the image into lx × ly blocks. Third, the point pattern is extracted using theevaluation function on the basis of the color of the visible rays, as
E1(x,y) = pg(x,y),(1)

E2(x,y) = pg(x,y) − wpr(x,y),(2)

where pg(x,y) is the green component of the picture elements, pr(x,y) is the red component of thepicture elements, and w is the adjustment factor.
The proposed method should extract the green point pattern, because we used a laser pointer.
The term E1(x,y) extracts the green point pattern, and E2(x,y) evaluates the ratio of the green to
red. Natural light such as evening sun irradiates yellow visible rays. Yellow is composed of green
and red. If system extracts the green point pattern using E1(x,y) only, the system extracts the green

352

Sensors and Materials, Vol. 28, No. 4 (2016)

(a)

(b)
Fig. 2. (Color online) The proposed system detectsa boundary line between a down staircase and floor
from images that are filmed at a place far from thedown staircase. (a) The image is photographed 2 maway fron the borderline between the down staircase
and the floor. (b) The image is photographed 1 maway from the borderline between the down staircase
and the floor.

Fig. 3. (Color online) Detection of the boundaryline between the down staircase and floors using theilluminance of point pattern.

point pattern and the yellow visible ray. The yellow visible ray disturbs the proper extraction of
the green point pattern. The system should remove the effects of the yellow visible ray. Thus,
the proposed system extracts the green point pattern using E1(x,y) and E2(x,y). When E1(x,y) andE2(x,y) exceed the threshold, the point pattern elements are extracted. The thresholds Te(k)(e = 1,2, k = 1, 2,…, 5) for E1(x,y) and E2(x,y) are set on the basis of the number of point pattern elements
as 500, 1000, 1500, 2000, and 2500. Fourth, the number of the point pattern elements in the eachblock is counted. From the number of point pattern elements in each block, point pattern feature is
calculated asfe (k, i x , iy) =be (k, i x , iy )max be (k, i x , iy )(3)

i x∈I x ,iy∈Iy

where fe(k,ix,iy) is the point pattern feature, Ix is the number of the block on the x axis, Iy is thenumber of the block on the y axis, and be(k,ix,iy) is the number of point pattern elements in eachblock.
Fifth, the Euclidean distance ue(k,q) between fe(k,ix,iy) and the point pattern feature of thetemplate image as radiating visible rays to various flooring materials is calculated asfe (k, i x , iy ) − f (q, i x , iy ) 2 ,ue (k, q) =i x∈I x iy∈Iy

(4)

353

Sensors and Materials, Vol. 28, No. 4 (2016)

where f(q,ix,iy) is the point pattern feature of the template image radiating visible rays to flooringmaterials.
The term f(q,ix,iy) is calculated by the same procedure as Eq. (3). The thresholds for E1(x,y),and E2(x,y) are set on the basis of the number of point pattern elements in the template image
being 1500. The number of point pattern elements differs depending on the type of floor material.
The images of the radiated visible ray on each floor material in a building are learned as template
images. A robust feature to changing illuminance owing to changing floor materials is obtainedby learning the point pattern feature of the template image as radiating visible rays from various
flooring materials. When the threshold processing extracts exactly the point pattern elements,ue(k,q) becomes a minimum value. Hence, the threshold for extracting point pattern elements isdefined to regard Te(k), when ue(k,q) becomes minimum value, as TE. The point pattern featurefb(ix,iy) is extracted using TE. The robust feature for changing illuminance is extracted by changing
TE adaptively on the basis of the number of point pattern elements.
Finally, the boundary line between the down staircase and floor is detected on the basis offb(ix,iy). If there is a down staircase in front of visually impaired people, the proposed system
obtains an image with middle floor in upper half of the image. Six blocks in the upper half of theimage are defined as the detection area in Fig. 3. From the inverse square law, E1(x,y) and E2(x,y)
of the point pattern in the middle floor become smaller than E1(x,y) and E2(x,y) of the point patternin floor around the feet. In this case, many E1(x,y) and E2(x,y) values in detection area become lessthan TE. The number of point patterns in a block is defined as fb(ix,iy). If there is a down staircase
in front of visually impaired people, fb(ix,iy) in the detection area become less than the other fb(ix,iy).
Thus, fb(ix,iy) in the detection area counts. The number of fb(ix,iy) in the detection area is defined as
CB. Based on this property, if CB is more than the threshold TP, the input image is detected as theimage including the boundary line between the down staircase and floor.
2.2

Selection of image using the Hough transform

The step of a down staircase has many straight lines parallel to the horizontal line in the image.
On the other hand, an image different from the down staircase image does not have straight lines
parallel to the horizontal line. The images that are different from the down staircase images can be
distinguished by counting the straight lines parallel to the horizontal line in the image. The straightlines in the image are detected using the Hough transform. The processing time for the Houghtransform is less than that of POC. The proposed method recognizes the down staircase quickly
because the proposed method removes images that are different from the down staircase and applies
image processing to a pedestrian route only. Figure 4 shows the detection of the straight lines
parallel to the horizontal line in the image using the Hough transform. First, the straight lines in theimage are detected using the Hough transform. Second, when the angle θh between the horizontal
line and the straight line is θh > Tθ or θh < Tθ, that line is not considered. Moreover, when the lengthof the straight line is less than 0.1 times the width of the image, that line is also not considered.
Third, the straight line is connected to the nearest straight line. The distance between the twostraight lines is calculated asuRL(nl , nr ) =xR(nr ) − xL(nl ) 2 + yR(nr ) − yL(nl ) 2 ,

(5)

354

Sensors and Materials, Vol. 28, No. 4 (2016)

Fig. 4. (Color online) Detection of straight lines parallel to the horizontal line in image using the Houghtransform.

where uRL(nl,nr) is the Euclidean distance between two straight lines, xR(nr) and yR(nr) are the rightend coordinates of the nr-th straight line, and xL(nl) and yL(nl) are the left-end coordinates of nl-th
straight line.
The term θnear is defined as the angle between line nl and line nr. The lines nl and nr are selected
when uRL(nl,nr) becomes a minimum. When θnear is more than 0.1 radian, the two straight lines areconnected. If the number of connections is less than TRL, the system identifies an image that is not adown staircase. That image does not apply stereo matching method.
2.3

Down staircase recognition by stereo matching

Figure 5 shows that the proposed method recognizes the down staircase from a selected imagein § 2.2 by stereo matching. First, the stereo matching calculates the distance between cameras andtarget objects from images with two cameras installed in parallel asd=

LBd(xL − xR)(6)where xL is the x-coordinate of a target object in image gL with a camera on the left side, xR is the
x-coordinate of a target object in image gR with a camera on the right side, L is the focal distance ina camera, B is a distance between the left and right cameras, and d is the pixel pitch in the camera.
The 1/5 parts from both ends of gL are not considered. The value xL – xR is calculated using
block matching algorithms.(12) The block matching algorithm divides gL and gR into Nx × Ny blocks
and matches the block in gL and the block in gR. The term xL(nx,ny) is defined as the block of thenx-th in the x-axis and ny-th in the y-axis. The term xR(nx,ny) is defined as the block in gR with thehighest similarity to xL(nx,ny). The degree of similarity is calculated by POC. POC calculates thedegree of similarity from the phase information of the frequency, which is obtained from gL andgR by a one-dimensional Fourier transform. The block in gR with the highest similarity to xL(nx,ny)
is identified by a coarse-to-fine search (13). Second, the down staircase is recognized from xL –xR. The down staircase is composed of steps of constant height. From Eq. (5), xL – xR is inversely
proportional to the square of the distance between the camera and each step. Hence, when xL –xR changes regularly, the image is recognized as the down staircase. The degree of regularity is
calculated as355

Sensors and Materials, Vol. 28, No. 4 (2016)

Fig. 5.

(Color online) Recognition of down staircase by stereo matching.

c(x, y) =δ(n x , ny ),(7)

Td ≤ ux ≤ Tu
otherwise(8)

uc = |xR(nx,ny) − xR(nx,ny − 1)|,(9)n x∈N x , ny∈Nyδ(n x , ny ) =10

where uc is the Manhattan distance, and Td and Tu are thresholds.
When c(x,y) is greater than three, the image is recognized as the down staircase. Moreover, theproposed method can reproduce the down staircase from the straight line and the distance between
the camera and the down staircase.

3.

Evaluation of Experiment

3.1

Experimental methods

We evaluated the borderline detection method using the image including the boundary line
between the down staircase and floor (DSF) and other images (NDSF). The detection of the down
staircase using the Hough transform and stereo matching was evaluated using the down staircase
images (DS) and other images (NDS). The photography was done at Kanazawa University. Stairs
size conforms to enforcement ordinance of construction standard law. The numbers of DSF, NDSF,
DS, and NDS are 103, 105, 100, and 100, respectively. The image size was 512 × 384 pixels. The
DSF and NDSF were photographed 2 m away from the borderline between the down staircase andthe floor. The DS and NDS were photographed 1 m away from the down staircase. Table 1 shows
the PC specifications. The DSF, NDSF, DS, and NDS were photographed with a camera (FinePix
REAL 3D). The value of Tθ is set as 0.7; Td as 3; Tu as 9; Nx as 2; and Ny as 8.
3.2

Experimental results

Table 2 shows the results for DSF and NDSF. From the table, the mean of the f-measure was 0.92.
Table 3 shows recognition results for the DS and NDS. From the table, the mean of the f-measure
was 0.96.

356

Sensors and Materials, Vol. 28, No. 4 (2016)

Table 1
Specifications of PC used to evaluate experiments.
CPU
MemorySpec
2.8 GHz
4 GB

Table 2Detection results of the DSF and NDSF using the illuminance of point pattern.
Image
Precision Recall f-measure
DSF
NDSF
DSF
98
12
0.89
0.95
0.92
Detection
NDSF
5
95
0.95
0.89
0.92

Table 3
Recognition results of the DS and NDS using Hough transform and stereo matching.
Image
Precision Recall f-measure
DSF
NDSF
DSF
98
5
0.95
0.98
0.97
Detection
NDSF
5
95
0.98
0.95
0.96

4.

Discussion

Figure 6 shows examples of images that were correctly detected using the proposed method in §2.1. From the figure, the proposed method could correctly detect the images of a floor and down
staircase that are painted a single color or in a repeating pattern. Figure 7 shows examples ofimages that were incorrectly detected using the proposed method. From the figure, the proposed
method detected images incorrectly because the illumination intensity of the point pattern changed
for each floor material or shadow of a wall. The detection time was 0.17 s. The proposed method
could detect a down staircase about 2 m away. The person’s walking speed was 1.4 m/s. Hence,by the proposed method, visually impaired people could stop walking before the down staircase
is reached. Figure 8 shows examples of images that were incorrectly recognized using the Houghtransform and stereo matching. From the figure, the proposed method recognized incorrectly the
NDS images that have many straight lines parallel to the horizontal line in the image. The proposed
method in § 2.3 recognized the images incorrectly because the illumination intensity of the point
pattern changed for each floor material. In addition, misrecognized images include the down
staircase and other objects. Figure 9 shows a reproduction of the down staircase from the straightline and the distance between the camera and the down staircase. From the figure, the proposed
method can reproduce the down staircase using the Hough transform and stereo matching. We
evaluated the border line detection method using the image including a boundary line between DSF
and NDSF. The detection of the down staircase using the Hough transform and stereo matching
was evaluated on the basis of DS and NDS. In contrast, the Hough transform can recognize an up
staircase because straight lines are parallel to the horizontal line in an up staircase. Also the point
pattern and stereo matching method can recognize an up staircase because illumination intensityof the middle floor is lower than the illumination intensity of the floor around the feet. The

Sensors and Materials, Vol. 28, No. 4 (2016)

Fig. 6.

(Color online) Example of images were correctly detected by illuminance of point pattern.

Fig. 7.

(Color online) Example of images were incorrectly detected by illuminance of point pattern.

Fig. 8.

357(Color online) Example of images were incorrectly recognized by Hough transform and stereo matching.

Fig. 9. (Color online) Reproduction the down staircase from the straight line and distance between camera anddown staircase.

proposed method recognizes with certainty the down staircase even if the illuminance of external
light changes because it normalizes illumination intensity in images using Eq. (3). However, if
illumination intensity in an image changes locally, the proposed method decreases an identification
accuracy. To solve this problem, we need to correct the illumination intensity in images using themeasured illuminance of external light.

358

5.

Sensors and Materials, Vol. 28, No. 4 (2016)

Conclusion and Future Work

We proposed a detection method for the boundary line using the illuminance of a point
pattern. In addition, we proposed the high-speed detection of a down staircase using the Houghtransform and stereo matching. Evaluation experiments show that the proposed method canrecognize the down staircase and borderline with more than 0.9 identification accuracy. However,
we must improve its accuracy more for safe walking. In future work, we would like to improve
identification accuracy. For that purpose, we will correct the illumination intensity in images using
the illuminance of external light measured using the illuminance sensor within a smartphone. In
addition, we will reduce the effect of noise, such as from a handrail, using the direction and stateof visually impaired people measured using an accelerometer, an angular velocity sensor, and amagneto metric sensor within a smartphone.

References
1 Q. H. Nguyen and T. H. Tran: Int. Conf. Adv. Technol. Commun. (ATC) (2013) p. 398.
2 L. Wenzhong, H. Yuefei, F. Xiaoming, L. Sanglu, and C. Daoxu: IEEE Trans. Parallel Distrib. Syst. 26 (2015)
382.
3 T. I. Chowdhury, M. M. Rahman, S. A. Parvez, A. K. M. M. Alam, A. Basher, A. Alam, and S. Rizwan: 2015
Int. Conf. Netw. Syst. Secur. (NSysS) (2015) p. 1.
4 M. S. Sadi, S. Mahmud, M. Kamal, and A. I. Bayazid: 2014 Int. Conf. Electrical Eng. Inf. Commun. Technol.
(ICEEICT) (2014) p. 1.
5 N. Mahmud, R. K. Saha, R. B. Zafar, M. B. H. Bhuian, and S. S. Sarwar: 2014 Int. Conf. Inf. Electron. Vision(ICIEV) (2014) p. 1.
6 B. Mustapha, A. Zayegh, and R. K. Begg: 2014 Int. Conf. Comput. Commun. Control Technol. (I4CT) (2014)
p. 271.
7 M. Vlaminck, L. Jovanov, P. Van Hese, B. Goossens, W. Philips, and A. Pizurica: In 2013 Int. Conf. 3DImaging (IC3D 2013) (2013).
8 A. Aladren, G. Lopez-Nicolas, L. Puig, and J. J. Guerrero: IEEE Syst. J. PP (2014) 1.
		doi:10.1109/JSYST.2014.2320639
9 S. Murakami, M. Shimakawa, K. Kivota, and T. Kato: 2014 Joint 7th Int. Conf. Soft Comput. Intell. Syst. (SCIS)
and 15th Int. Symp. Adv. Intell. Syst. (ISIS) (2014) p. 1186.
10 R. O. Duda and P. E. Hart: ACM 15 (1972) 11.
11 C. D. Kuglin and D. C. Hines: Proc. Int. Conf. Cybernet. Soc. (1975) p. 163.
12 D. I. Barnea and H. F. Silverman: IEEE Trans. Comput. 21 (1972) 179.
13 P. J. Burt, J. R. Bergen, R. Hingorani, R. Kolczynski, W. A. Lee, A. Leung, and H. Shvayster: Workshop on
object tracking with a moving camera. In Visual Motion Proc. (1989) p. 2.