Development of image stabilization system for remote operation ofwalking robots
Ryo Kurazume

Shigeo Hirose

Tokyo Institute of Technology
2-12-1, Ookayama, Meguro-ku, Tokyo, Japan

Abstract

Walking robots have high adaptability for terrain variation, and thus, have been expected as e ective moving
platform on uneven terrain, stairs, forest, marshy surface, and on ice.
On the other hand, mobile robots that perform several
hazardous tasks such as mine detection or the inspection of an atomic power plant are typically controlled
by operators from distant places. For a remote operation system, use of visual information from a camera mounted on a robot body is very useful. However,unlike wheeled vehicles, the camera mounted on thewalking robot oscillates because of the impact of walking, and the obtained unstable images cause inferior
operation performance.
In this paper, we introduce an image stabilization system for remote operation of walking robots using a highspeed CCD camera and gyrosensors. The image stabilization is executed in two phases, that is, the estimation of the amount of oscillation by the combination of the template matching method and gyrosensors, and change of the display region. Pentium MMX
instruction is used for template matching calculation,and the estimated amount of oscillation is outputted inevery 12 [msec.]. Furthermore, developed image stabilization mechanism can be used an external attitude
sensor from the visual information, and the damping
control of the robot body while walking is also possible.
Experimental results showed stabilized images thateliminates the oscillation component are taken evenwhen the robot moves dynamically or in long distance,and veri ed that the performance of attitude control
using the developed image stabilization system is almost same as the case using an attitude sensor.
1Introduction

In comparison with wheeled vehicles, legged walking
robots have several interesting characteristics, for ex-ample,
1. the robots have high adaptability for the abruptchanges of terrain conditions.
2. foot positions can be chosen freely and discontinuously within the work space of the leg under therestriction of kinematic stable conditions.
3. the robots can stride over small obstacles.
4. the robots can change the moving direction without slippage at foot positions.
5. legs can be used as manipulator arms.
Therefore, the walking robots have been expected as
e ective moving platform on uneven terrain, stairs,forest, marshy surface, and on ice. Until now, as applications of walking robots, mine detection and removal system by quadruped walking robots in a forest[1] or academic investigation on steep slopes aroundthe crater [2], etc. are reported.
On the other hand, a mobile robot that operates instead of a human worker in hazardous or inaccessible
environment, such as disaster eld or on other planet,
is being developed. In the future, it is expected that
arti cial intelligence will be equipped with these mobile robots, and adequate tasks will be planned automatically from the sensory information and worldmodel. However, practical intelligent system for a mobile robot has not been developed yet, and thus, remote control systems by human operator will be utilized at present.
In order for the operator to recognize the surrounding environment around the robot appropriately andto control with high operation performance, acquisition of images from a camera mounted on the robot
body and presentation for the operator is indispensable . However, unlike wheeled vehicles, the camera
mounted on the walking robot oscillates because of theimpact of walking, and the obtained unstable images
cause inferior operation performance for the operator.
Therefore, image stabilization systems that eliminates
the oscillation component of images caused by the disturbance and presents the stabilized images for the operator is requisite for the practical remote operation
system for a walking robot.
Several techniques of image stabilization are considered, for example, i) special camera universal head orlens, ii) oscillation component removal by the image
processing, and iii) damping control of the robot body.
Technique (i) has already been used for the shake prevention of the video camera, but the mechanism thatcan be used even for the large angular velocity of walking robots, is light weight and small is costly. Thus, inthis paper, we develop a low cost and high speed image stabilization mechanism that uses the techniques(ii) and (iii). This system consists of high speed CCD
camera, gyrosensors and image processing system utilizing the Pentium MMX instruction, and stabilized
images are taken in every 12 [msec.] . In addition, thisimage stabilization mechanism can be used an external attitude sensor from the visual information, andthe damping control of the robot body while walking
is also possible.
The remained part of this paper is as follows: in section 2, we explain the developed mechanism and technique of image stabilization, and in section 3, we introduce the remote operation experiment. Next, insection 4, we report the damping control experimentof the walking robot using the developed image stabilization system.

CCD Camera

GyrosensorsFigure 1: 4-legged walking robot TITAN-VIII
Acquired images from CCD camera are processed
through remote computer and displayed on the HMD(Eyetrek, Olympus) of the operator.
Mounted high-speed CCD camera (ES310, Kodak)
can outputs the 80 progressive images of the 648x484
pixels in every second. Acquired images are transmitted to the main memory of the remote computer (Pentium III, 600 MHz) through image acquisition board(MaxPCI, Datacube), and processed using Pentium
MMX instruction. Therefore, in case that image processing time is 12 [msec.] or less, the whole throughput
becomes 12 [msec.], and is almost equal to the sampling time of the walking robot, 10 [msec.].
2.2

shows the adopted image stabilization methodthat combines the information from the gyrosensor
and CCD camera based on the template matching
method. In this system, we rst choose the speci cregions as initial templates, and pursue the movementof regions corresponding to the initial templates by
calculating the correlation between them, and nd thematching point. Here, the search area of corresponding region of each template is set to the point that isgiven by adding the matching point at previous timeand camera movement measured by the gyrosensor.
Then, actual display region that is slightly smaller
than the input image is shifted so that the templateis always located in the same position in the display
region, and consequently, the stable image can be obtained.
This system, at rst, roughly compensates the image
oscillation by the gyrosensors, and then, absorbs theremaining error that is due to the processing or communication delay et al. by template matching method
Fig. 222.1

Image stabilization system
Experimental

systemforremote

operation

Quadruped walking robot, TITAN-VIII, for remote
operation experiment is shown in Fig. 1. Thisrobot equips with a high speed CCD camera (ES310,Kodak), computer board (Pentium 200MHz, JapanData System), AD/DA board, Ethernet card, silicondisk, 3-axes attitude sensor, (Maxcube, Japan Aviation Electronics), and two gyrosensors (Gyrostar, Murata).
Gait planning, position and attitude control, and estimation of shape of oor are all executed in every
10 [msec.] by the equipped board computer, and therobot is controlled from the remote operator by using the joystick connected to the remote computer.

Image stabilization method

Search of the tmplate image
Initial position is set by gyro sensors
Display region

MAXCUBE TA7233
GU-3020
Oscillationof imageLinearactuator

Input image
Output image

Gyrostar
Template

Figure 2: Image stabilization method with CCD camera and gyrosensor
using the actual images that should be stabilized.
As described above, calculation of template matching is executed using MMX instruction of Pentium III
processor and image acquisition and change of display
region are done by image processing board (MaxPCI,Datacube). Processing time of correlation calculationfor one template is 0.22 [msec.], where template sizeis 16x16 pixels and search region is 5 neighborhoodpixels.
2.3

Comparison of image stabilization
using various sensors

As shown in Fig. 3, we developed the experimentalsystem to verify the e ectiveness of proposed systemand compared the image stabilization ability for various sensors and their combination.
This experimental system consists of a plate that rotates around a horizontal axis by the linear actuatorin constant amplitude and frequency, and three attitude sensors, (Maxcube JAE, GU-3020 Datatec, andTA7233 Tamagawa Seiki) gyrosensor (Gyrostar, Murata), and CCD camera. By driving the linear actuator in reciprocating way, image taken from CCD
camera oscillates and attitude of CCD camera to compensate the oscillation is measured by these sensors,respectively. The experiment is carried out for following ve conditions, and each image stabilization
performance is compared by changing the frequencyof oscillation.
1. Attitude of CCD camera is measured by attitudesensors.

CCD camera
Potentiometer

Figure 3: Experimental system
2. Gyro sensor is used.
3. Template matching method using acquired CCD
camera image (without gyrosensor).
4. The combination of CCD camera and gyrosensor. Search region is shifted in constant distance( 5 pixels) to estimated direction of camera movement measured by gyrosensor.
5. The combination of CCD camera and gyrosensor. Search region is shifted in estimated distanceto estimated direction of camera movement measured by gyrosensor.
As a result of the experiment, though it was possiblethat the oscillation was suppressed using methods (1)and (2) for any attitude sensors, it was not possiblethat the oscillation was perfectly eliminated becauseof the measurement error and time delay caused bycommunication and calculation processes. In method(3), the template could be tracked up to 5 degrees
of amplitude , 1.5 [Hz] vibration of frequency and47 [deg./sec.] of maximum angular velocity. but
often missed over 1.5 [Hz]. Moreover, method (4)could track up to 3 [Hz] of vibration frequency and 94[deg./sec.] of maximum angular velocity, and method(5) could track up to 4 [Hz] and 126 [deg./sec.].
Here, in methods (3), (4), and (5), template size is
16x16 pixels and search region is 5 neighborhood.
In method (4), template searching ability is raised byshifting the search region with 5 pixels toward themoving direction estimated by the gyrosensor. Furthermore, in method (5), the distance and the direction of search region are both estimated by gyrosensor.
From the result of these experiments, it was proventhat method (5) which combines CCD camera and gy-rosensor has highest image stabilization performanceagainst impulsivity, and we adopted this method indeveloping the image stabilization system.

3

Experiment of remote operation for walking robot

In order to verify the ability of developed image stabilization system, we performed the experiment of remote operation by using a walking robot named TITAN VIII shown in Fig. 1.
In the experiment, acquired image is divided into 100
regions of the size 16x16 pixels, beforehand. Then, ineach region, eigen values 1 and 2 of following matrix  are calculated and some regions where min(1 ,2 ) > are chosen as initial template images(trackability [4]).
thresh02 1XBC (1)=2@ A
2i;js@ Ii;j@ Ii;j@ Ii;j@x@x@y@ Ii;j@Ii;j@ Ii;j@x@y@y

Next, SAD (Sum of Absolute Di erence) is calculated
between acquired image and template image using
Pentium MMX instruction within 5 pixels neighborhood of each acquired region. Then the regions wherethe minimumvalue of SAD is less than threshold value
are categorized as able to be tracked. This process isexecuted for every template image and the o set valueof display region is set as the average of the di erence
between template position that shows the minimum
SAD value and initial position of template. Furthermore, when the number of trackable templates is lessthan the particular threshold number, calculation of
trackability and selection of appropriate template areexecuted automatically, and the system can continue
the image stabilization even after the robot moves long
distances and initial templates go out of the acquired
image.
Fig.
4 shows an example of the remote operationexperiment. Left hand of Fig. 4 shows the stabilized
images and right hand shows raw images. As a resultof experiment, stabilized images that eliminates theoscillation component are taken even when the robot
moves dynamically or in long distance.

Figure 4: Results of image stabilization
4Use of stabilization system as
attitude sensor

Developed image stabilization system can be utilized
as external attitude sensor of the robot body since
the attitude of CCD camera can be estimated fromthe o set of template position and focus length. In
the coordinate system shown in Fig. 5, the velocity(u; v) of the point (x; y) on the image plane placed on
the z axis can be derived using the linear and angularvelocity of robot body, (V ; V ; V ) and (! ; ! ; ! ) asfollows.
u=fVx

Vv=f yZZxyx

f (1 + x2)!y + fy!zxVz+ fxy!

yVz+ f (1 + y2 )!

zxx

fxy!yyfx!zz(2)(3)

In Fig. 5,  is the robot body coordinate systemand these axes are parallel to the axes of the camera
coordinate system BCωz

Left image

Z

Forward image

Forward image
Vz

Left imagex
Mirror(u,v)
yX

ΣCxy

Yawωxωy
ΣB

Pitch

Vx

VyRollY

Figure 7: Split images

Figure 5: Camera and robot coordinate systems
Here, maximumvalues of actual linear and angular velocity of walking robot in above experiment are 0.01
[m/sec.] and 100 [deg./sec.], respectively, and therefore, the component corresponding to linear velocity,and, can be neglected if Z is suÆciently large. Consequently, (! ; ! ; ! ) is derived as
the linear equation of (u; v), and calculated by leastsquare method if (u; v) is measured in several points.
However, ! is likely to be a ected by measurement
error because the coeÆcient value is quite small, andaccurate estimation of ! is diÆcult. Thus, we put amirror in front of camera lens as shown in Figs. 6,7,and split the acquired image into front and left sideimages.
VxVyxVz

ZyVz

ZxyzzzMirrorFigure 6: images partition by a mirror
On the left side image shown in Fig. 5, the image

velocity (u; v) is written as
xVx
fy!x f (1 + x2)!y + fxy!z (4)XV yVx+ fx!x fxy!y + f (1 + x2)!z (5)v= f yXand (!x ; !y ; !z ) can be given as linear equation of(u; v) if the linear velocity can be neglected same as
u= fVzabove. Thus, we tracked the several templates onfront and left images by the developed image stabilization system, and estimated (! ; ! ; ! ) precisely using
least square method of these four equations. Actually,
since we can measure the o set of template from theinitial position, the o set of camera attitude from theperiod when the initial templates are chosen are estimated. Thus, the current attitude of robot body (roll,pitch, and yaw in Fig. 5) are given as addition of theinitial attitude and measured o set values.
Fig. 8 shows the results of the experiment in whichrobot body is controlled to be stabilized by using theattitude sensor (Maxcube, JAE) and the image stabilization system, respectively. Vertical axis shows thevariance of attitude of robot body and horizontal axis
shows the feedback gain. From these experiments, theperformance of attitude control using developed image
stabilization system is almost same as the case using
an attitude sensor. In particular, it is veri ed that therotational angle around gravity axis is estimated much
more accurately than conventional attitude sensors because there is no drift after long time measurement.
x5yzConclusion

For the remote operation system of the walking robot,
use of visual information from a camera mounted on

2

Variance [rad ]8x10-4

6Pitch4Roll

2000.04

0.08

0.12

Feedback gain(a) Attitude sensor (Maxcube)

2Variance [rad ]8x10-4

6

iment using a quadruped walking robot, TITAN-VIII.
Furthermore, we showed that the developed image stabilization system can be utilized as the attitude sensor of robot body, and introduced an accurate attitude
measurement mechanism by splitting the images using
a mirror. Experimental results showed that the performance of attitude control using the developed image
stabilization system is almost same as the case using
an attitude sensor. In particular, the rotational angle
around gravity axis is estimated much more accurately
than conventional attitude sensors because there is nodrift. Thus, combination of the proposed system andconventional attitude sensors is expected to realize ahigh performance and low cost attitude sensor for amobile robot.
Future works will be the discrimination of image oscillation to the components that are caused by the undesired oscillation such as impact and desired oscillation such as operation command or gradient change ofslope, and development of appropriate fusion systemof sensory information from CCD camera and othersensors.

4PitchReferences

2Roll
000.040.08

0.12

Feedback gain(b) Image stabilization system

Figure 8: Comparison of attitude estimationa robot body is very useful. However, unlike wheeledvehicles, the camera mounted on the walking robot oscillates because of the impact of walking and discontinuity of supporting legs, and the obtained unstable
images cause inferior operation performance.
In this paper, we introduce an image stabilization system for a remote operation of walking robots using ahigh speed CCD camera and gyrosensors. This system, at rst, roughly compensates the image oscillation by the gyrosensors, and then, absorbs the remained error that is due to the processing or communication delay et al. by template matching method using
the actual images that should be stabilized. We also
reported the performance of the developed image stabilization system through vibration experiments usinga CCD camera and several sensors, and walking exper-[1] Hirose, Kato, Development of quadruped walking
robot with the mission of mine detection and removal, Proc. ICRA, TAII-7-4, 1998
[2] Red Whittaker, David Wettergreen, John Bares,
Con guration of a Walking Robot for Volcano Exploration, pp.487-492, '93 ICAR, 1993
[3] Y. S. Yao and R. Chellappa,Selective Stabilization of Image Acquired by Unmanned Ground Vehicles,IEEE Trans. on Robotics and Automation,
Vol.13,No.5,pp.693-708,1997[4] C. Tomasi and T. Kanade, Shape and Motion
from Image Streams: A Factorization Method 2. Point Features in 3D Motion, Technical Report
CMU-CS-91-105, Carnegie Mellon Univ., Pittsburg, Penn., Jan. 1991