Front Environment Recognition of Personal Vehicle
using the Image Sensor and Acceleration Sensors forEveryday Computing
Takahiro Matsui1, Takeshi Imanaka1, Yasuyuki Kono1,1

Graduate School of Science and Technology, Kwansei Gakuin University, 2-1
Gakuen, Sanda, Hyogo, JAPAN{acl27598, bbv84420, kono}@kwansei.ac.jp

Abstract. In this research, we propose the method for detecting moving objects infront of the Segway by detecting running state for the Segway. Running state of thepersonal vehicle Segway is detected with both an image sensor and an acceleration
sensor mounted on the Segway. When objects are moving in front of the Segway, theimage sensor can capture the motion while the acceleration sensor shows a different
result. By analyzing the difference our method successfully recognizes moving
objects from environment.
Keywords: Segway, Image Sensor, Acceleration Sensor, Optical Flow.

1Introduction

This research examines the safety and comfort of the personal vehicle Segway [1]which uses electrical energy instead of gasoline in everyday environment. Unlike
other vehicles such as cars and bicycle, the Segway can be used to run not only on theroad in outdoors, but also indoors such as inside the buildings or factories shown in
Fig. 1 right. Safety must be especially considered for the Segway when driving insuch place, since they have high pedestrian traffic. To improve the safety of theSegway, a system to notify the danger to the rider and avoid the incident
automatically is necessary. This paper proposes the method to detect moving objects
in front of the Segway. Rider can easily recognize static obstacles and avoid them, but
it is difficult to avoid dynamic obstacles such as unexpected pedestrian and otherpersonal vehicles. Thus, if a system can recognize these obstacles and notify the riders
or avoid them automatically, safety of the Segway can be assured. Most of theresearches which use vehicle video system to recognize the front environments are
instead for use of cars, and are not applicable for slow vehicle like the Segway, which
requires to recognize dynamic obstacles in close range. Moreover, the operation of theSegway is different from conventional vehicles in the way that has no accelerationpedal and no brakes. A rider stands on the footplate between two wheels to adjust
speed, movement, and direction of the Segway. Vehicles other than the Segway do
not have these features.

Our proposed method detects the running state of Segway by employing both animage sensor(camera) aimed forward and an acceleration sensor. By integrating
sensor readings, our method detects moving objects in the frontal direction. In thisresearch, we focus on the tilting of the handle bar while driving the Segway. From theimage sensor mounted on the handle bar, the system analyzes the video captured fromit and detects the optical flow [2,3] to obtain the running state among the following
six states: forward, backwards, right turn, left turn, acceleration and deceleration.
When a moving object such as a walking person is within the range of the image
sensor in front of the Segway, it affects the trend of the optical flow, although thereadings from acceleration sensors are not affected by the object. Using these dataobtained from both sensors, we analyze whether the moving object actually exists
within the certain range which the incompatible information has been detected. Thismethod will contribute for safety in the future, notifying the riders to avoid themoving obstacles in front of the Segway.
Place where persons do not exist together to car

Place where person exists together to vehicle
Dangerous moving object

In vehicle camera

On vehicle camera

Fig. 1. A difference between a car and the Segway

2

Research Background

2.1

Related Works

Many of related works usually mount the image sensor on the car. Although theposture of the camera is overall stable in these works, that on the Segway changes
depending on the running state, i.e., the rider’s center of gravity. For example, when
the rider wants to turn right, he tilts the handle bar to the right, and the image sensor
also tilts to the right. The rider leans forward to gain speed, so the handle bar together
with the camera also leans forward. It is necessary to analyze the running state of
Segway by detecting the tilt of the image sensor which is affected by the change ofthe Segway posture.

A method for detecting three-dimensional position of moving objects with an invehicle camera is proposed [4]. The method in [4] is not suitable to the Segway
because the posture of the camera is not stable while driving. Another approach is todetect approaching vehicles by classifying horizontal edges in an image into twoclasses; ones on the ground and one above the ground [5]. The method in [5] is alsodifficult to apply to complicated everyday scenes.
2.2

Preparation

The image sensor is mounted on the handle bar as indicated in Fig. 2(A). The
acceleration sensor is mounted on the footplate. The image sensor is Glasshopper ofPoint Grey Research Inc shown in Fig. 3. The acceleration sensor is WAA-001 of
Wireless Technologies Inc shown in Fig. 4. The specification of the sensors is shownin Table 1 and 2 [6] [7].

AAcceleration sensor

PCFig. 2. Placement of sensors on the Segway

Fig. 3. Grasshopper

Fig. 4. WAA-001

Table 1. Spec of the Grasshopper
SensorSampling rate
Resolution
Interface
Dimensions
WeightGRAS-20S4C
Sony 1/1.8 progressive scan CCDs
30FPS
1624×1224
Dual 9-pin IEEE-1394b 800Mb/s ports
44×29×58mm
104g (The optical system is excluded)

Table 2. Spec of the WAA-001
WAA-001
Sampling rate
MAX 200Hz
Range of detection ±3g
Interface
Bluetooth (Class 2)
Dimensions
38×39×10mm
Weight17g

3Moving Object Detection by Sensor Fusion

In this research, the image sensor is mounted on the handle bar of the Segway andthe acceleration sensor is mounted on the footplate between two wheels. Runningstates of the Segway are detected from these sensors, and moving objects arerecognized from the discrepancy of the detection results.
The image sensor detects and analyzes optical flow patterns of each running states
while the acceleration sensor measures the acceleration of the Segway to determine
the running states. The optical flow shows the movement of feature points between
video frames expressed as vector. Considering the running state of the Segway as
moving forward, when the image sensor detects no moving objects, the optical flow
displays as shown in Fig. 5 left. When a moving object is within the field of view, theoptical flow is disrupted as shown in Fig. 5 right. On the other hand, the moving
object has no influence on the detection results of the acceleration sensor, whichmeasures the acceleration of the Segway for detection of the running states.

Fig. 5. Optical flows change when a moving object is recognized4Running state Detection using each Sensor

4.1

Posture Change and Running State of the Segway

The method to detect the running state is described here, namely; forwards,backwards, right turn, left turn, acceleration and deceleration.

Fig. 6. Forward

Fig. 7. Backwards

Fig. 8. Acceleration

Fig. 9. Deceleration

Fig. 10. Right turnFig. 11. Left turn

Details of optical flow pattern for each running state is described as follows.
In forward running state, a vanishing point and optical flows appear as shown inFig. 6. In backwards running state, a vanishing point and optical flows appear as
shown in Fig. 7. In accelerating running state, optical flows appear as shown in Fig. 8.
The flows rise from the bottom to the top as the handle bar leans forward. The camera
is faced downwards and a range of the image sensor moves to bottom. In deceleration
running state, optical flows appear as shown in Fig. 9. The flows descend from top tobottom as the handle bar is pulled back, and the camera is faced upward. In addition,
after the Segway has decelerated, note that the posture of the Segway returns to theupright position. In turning right running state, optical flows appear as shown in Fig.
10. In turning left running state, optical flows appear as shown in Fig. 11.
4.2

Running States Detection using the Acceleration Sensors

When running state of the Segway changes, the acceleration sensor data change
characteristically. Fig. 13 shows one example of the acceleration data. In addition, theline X shows the acceleration that is traveling direction of the Segway, and the line Y
shows the acceleration that is horizontal direction and vertical to X axis againsttraveling direction of the Segway shown in Fig. 12. The acceleration data change
remarkably when the running state changes.
For example, when the vehicle is running forward, the line X does not show visible
changes, but, large values are obvious along the positive direction when moving thevehicle from an idling state and along the negative direction during acceleration. Also,in line Y, large values are obvious along the positive direction when turning right, andalong the negative direction when turning left. Likewise, by detecting the significant
changes in the acceleration data, we can distinguish the running states from the imagesensor.
Fig. 13 shows readings from the acceleration sensor starting from the idling state,
turning right, moving forward for a while, and then turning left.

Y axisX axis

Acceleration sensor

Fig. 12. Three axial directions
mG
2000
Start

1000

Y-1000X-20005Idle

10Move forward

15

20

Right turn Left turn25

30 Second

Move forward

Fig. 13. An example of an acceleration sensors data
4.3Integrating two sensors information

This section compares image sensor data and acceleration sensor data obtained
simultaneously while riding the Segway. Fig. 14 shows the readings while theSegway is moving forward. When no moving objects are in the sight, the image
sensor simply detects the optical flow for “moving forward” as explained in section
4.1. No remarkable changes can be verified from the acceleration sensor data. Incontrast, Fig. 15 shows the case when the moving object exists in front of the Segway.
In this case, the optical flow shows different trends from that in Fig. 14, although theacceleration sensor obtains similar data with Fig. 14. The moving object which affects
the optical flows can be detected by examining the moment when such discrepancy
occurs.
Fig. 16 and 17 show examples while the vehicle is turning right. In this case,acceleration sensor data shows the same trend, although optical flow shows differenttrends.

X axis

Y axisFig. 14. Two sensors data when a moving object does not appear (moving forward)

X axis

Y axisFig. 15. Two sensors data when a moving object appears (moving forward)

X axis

Y axisFig. 16. Two sensors data when a moving object does not reflect (turning right)

X axisY axisFig. 17. Two sensors data when a moving object does not reflect (turning right)

5Conclusion

This paper described our method to detect front moving objects by using theincompatibility between readings from an image sensor and an acceleration sensor.
Moreover, the detection method of running state using two sensors to detect a moving
object is described. As future work, it is necessary to examine whether the safety ofthe personal vehicle can be achieved with our method.

Reference
1. Segway, http://www.segway.com
2. B.K.P Horn and B.G Schunck, "Determining Optical Flow", Artifical Intelligence,
Vol.17, pp.185-204, 1981
3. J.Y.Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker",Intel Corporation, Microprocessor Research Labs, 2000
4. K. Onoguchi, "Shadow Elimination Method for Moving Object Detection,"
Proceedings of International Conference on Pattern Recognition (ICPR'98),
pp.583-587, 1998
5. R.Okada, et al, “Obstacle Detection using Progressive Invariant and VanishingLines”, ICCV, pp.330-337, 2003
6. Point Grey Research, http://www.ptgrey.com/products/grasshopper/index.asp
7. ATR-Promotions, http://www.atr-p.com/sensor01.html