REMOTE CONTROL OF A NETWORKED MOBILE ROBOTUSING AN IMMERSIVE LOCOMOTION INTERFACE
Kazumasa Yamazawa, Masaki Tawada, and Naokazu Yokoya
Graduate School of Information Science
Nara Institute of Science and Technology
8916-5 Takayama, Ikoma, Nara, Japan 630-0192
E-mail: yamazawa@is.naist.jp
ABSTRACT

2. PROTOTYPE SYSTEM

Recently, the concept of behavioral media using a mobilerobot has emerged in tele-robotics, where the mobile robot
behaves as a user’s second body or an agent in a remotesite. An important issue in behavioral media is how tocontrol the robot intuitively immersing a user into a remotesite. This paper describes a new approach which combines
locomotion interface and immersive projection display of aremote site for controlling a networked mobile robot. We
have developed a prototype system and have evaluated theusability of the system with experiments.

2.1. Overview of prototype system
Fig. 1 shows the configuration of prototype system. A user
walks on a treadmill to control a remote mobile robot.
Walking information is received by a PC for treadmill
control. The PC controls the treadmill and transports
walking information to the mobile robot via network. In themobile robot site, a PC for mobile robot moves the robot
based on user’s walking, and transports omni-directional
video acquired by an omni-directional camera to the user’ssite via network. The transported omni-directional video isreceived by a PC for video reception, and is transported to agraphics WS via DV recorder. The omni-directional video istransformed to appropriate perspective video, and isprojected onto an immersive display. The user can control
the remote mobile robot by repeating the above process.
The configuration of the prototype system is described inthe following.
Mobile robot (Wheelchair of Misawa: M-Smart)
Misawa M-Smart is used as the base of mobile robot. MSmart is an electric wheelchair which is usually controlledwith the joystick on the right armrest. The mobile robot has
been rebuilt so that it can be controlled by digital input froma laptop PC. Fig. 2 (left) shows the appearance of themobile robot. The mobile robot can move forward-andbackward and rotate right-and-left.

1. INTRODUCTION
Behavioral media using a mobile robot is expected to havean important role in some applications such as rehabilitation,
virtual tour, active surveillance, and tele-operation inextreme environments. One of the most important issues
concerning behavioral media is how to control the robot
intuitively immersing a user into a remote site.
A number of remote control and tele-operation
techniques have been investigated for controlling a mobile
robot in robotics. Typical devices for controlling the robot
are foot pedals[1], joysticks[2,3,4], and a mouse[5,6].
Immersion into a remote site has been achieved bypresenting omni-directional stereo views on a head-mounted
display[7,8].
In this paper, we describe a new approach which
combines locomotion interface and immersive projection
display of a remote site for controlling a networked mobilerobot. In our approach, a user controls a mobile robot in aremote site simply walking on a treadmill located in an
immersive projection display environment. Locomotion
interface based on walking behavior provides the user withintuitive control of the robot and immersive projection
display of omni-directional live video does the user with thefeeling of rich presence in the remote site. The rest of thispaper describes a prototype system and its evaluation withexperiments.

Graphic WS
DV recorderVideoOmni-directional
cameraPC formobile robot

VideoVideoand Audio

Projectors

Videoand audio

Audio

Speaker
PC for video reception

Immersive
Display

Mirror

User

User’scentroid position

User’scentroid position

Mobile robot

Speed of treadmill

Treadmill

PC for treadmill

User’s site

Mobile robot site

Fig. 1. Configuration of prototype system.

1­4244­0367­7/06/$20.00 ©2006 IEEE2117

ICME 2006

DV recorder
It receives the stream, sends the video to Graphics WS, andoutputs the audio to the speakers.

Omnidirectional camera
Immersive display

Treadmill
Wheel chair

Fig. 2. Appearance of mobile robot (left), immersive
display, and treadmill (right).
Omni-directional camera (HyperOmni Vision)
The omni-directional camera is mounted on the mobile
robot so that it is located at almost the same height (about160cm) as user’s viewpoint. The omni-directional camera
acquires a 360-degree view around the robot. The omnidirectional camera satisfies the single viewpoint constraintand thus the acquired omni-directional video can betransformed to common perspective video as well as
panorama video suitable for presentation[9].
PC for mobile robot (SONY: PCG-R505FR-D)
It is carried on the mobile robot, and controls the robot
based on user’s walking information which is received vianetwork. Simultaneously, it transports the acquired omnidirectional video and audio to the user’s site via network.
Treadmill (SOLIDRAY: WalkMaster)
The belt of the treadmill can be moved into all directions on
a 2D plane. Therefore, the user can walk in all directions on
the 2D plane. The user wears the magnetic sensors on bothknees. The belt is moved so that the centroid of both knees
is kept at the center of the treadmill. The user can walk at
the fixed point on the treadmill [10]. The position of thecentroid is used not only for controlling the treadmill but
also for controlling the mobile robot by transporting theposition of the centroid via network.
PC for treadmill
It controls the treadmill based on information from themagnetic sensor. Simultaneously, it transports user’s waking
information to the mobile robot site.
Immersive display (SOLIDRAY: VisualValley)
It consists of three large slanted screens (see Fig. 2 (right))
and corresponding twelve projectors (four projectors perscreen). It displays video transported from the mobile robot.
Graphics WS (SGI: ONYX3800)
It transforms omni-directional video to common perspective
video for the immersive display.
PC for video reception
It receives the stream which includes video and audio via
network and sends the stream to the DV recorder.

2.2. Network transmission
The system has two network transmissions. One is videoand-audio stream transmission. The other is operation
instruction transmission. In video-and-audio stream
transmission, the mobile robot site is the sender and theuser’s site is the receiver. On the other hand, in operation
instruction transmission, the user’s site is the sender and themobile robot site is the receiver.
For transporting omni-directional video, we use
DVcommXP software made by Fatware Co. DVcommXP isbased on RTP Payload Format for DV Video (RFC 3189)[11]. DVcommXP can transport high-resolution DV data
(720x480, 30fps) using 30Mbps bandwidth via network. In
this study, we set the framerate to 10fps for bandwidth ofwireless network. The mobile robot site sends the omnidirectional video from the omni-directional camera mounted
on the mobile robot. The user’s site receives the omnidirectional video and transfers the video to the graphic WS
via the DV recorder. The graphic WS transforms omnidirectional video to common perspective video and projects
it on the immersive display.
The user’s walking action is estimated by using
magnetic sensors worn on both knees. Walking informationis then transported to the PC on the mobile robot by using
TCP/IP. At the mobile robot site, the PC transforms thereceived walking action to adequate instructions for themobile robot and controls the mobile robot. The mobilerobot can move only forward-and-backward and can rotate
right-and-left as mentioned earlier. Therefore, the user’s
forward-and-backward translation and right-and-lefttranslation should be reflected to forward-and-backward
translation and right-and-left rotation of the mobile robot.
2.3. Video presentation
The transformed video is presented on the immersive
display. First, the graphic WS transforms the omnidirectional video transported from the mobile robot via
network to twelve common perspective videos
corresponding to the number of projectors in real-time using
an image warping technique [12]. Next, each common
perspective video is presented on the immersive display via
the associated projector. Moreover, for presenting a
backward scene of the mobile robot, its common
perspective video is generated and is presented on the upper
part of the front screen as a rearview (see Fig. 3(b)).

2118

Fig. 4. Appearance of mobile robot site (left) and user’ssite (right) in experiment.

(a) Omni-directional image.

(b) Common perspective images for three screens.
Fig. 3. Omni-directional image and corresponding
common perspective images projected on three screens.
3. EXPERIMENTS
3.1. Overview of experiments
We have experimented with the prototype system. The
user’s and mobile robot sites are located in different
buildings. The distance between both sites is about 200m.
The user wears magnetic sensors on both of knees, watches
the video capturing the mobile robot site, and controls themobile robot using the proposed locomotion interface. The
network between both buildings is a wired 100Mbps
connection. The network between the building and mobilerobot is a wireless 54Mbps connection. We have set theframerate of video to 10 frames/sec. adjusting DVcommXP
and have set the maximum speed of mobile robot to 2 km/h.
Fig.4 shows the appearance of experimental environments.
We have confirmed that the user can control the remotemobile robot by walking. The time delay from acquiring anomni-directional image to presenting it on the screen via
network is about 1 second. The time delay from thebeginning of user’s walking to the beginning of themovement of mobile robot is also about 1 second. The delay
is mainly caused by network delay. In the present
implementation, the user must consider the delay of at most
2 seconds. Evaluation of the prototype system is describedin the following section.
3.2. Evaluation of usability
3.2.1. Tasks for evaluating usability
For evaluating the usability, we have defined the following
three tasks. Subjects know the size and moving mechanismof the remote robot before the tasks. Subjects control themobile robot for tasks and some quantitative evaluation iscarried out.
Task 1: Stopping in front of an obstacle
The mobile robot approaches an obstacle as close as
possible and stops in no contact. The robot moves only
straight forward. The distance between the starting point

and the obstacle is 4.25m. The purpose of this task is toexamine how much the delay affects the operation requiring
the sense of distance.
Task 2: Moving to a target point
The mobile robot moves to a target point that the mobile
robot cannot reach only by moving forward from the initialstate of position and orientation. There is no obstacle in thistask. When the task starts, the user can see the target point
via the immersive display. The user can control the mobile
robot within his/her discretion. The distance between thestarting point and the target point is 5.60m. The user needs
to move forward, rotating when the target point is not aheadof the mobile robot. The purpose of this task is to evaluate
the intuitiveness of correspondence between user’s
movement and mobile robot movement.
Task 3: Avoiding obstacles and moving to a target point
The mobile robot avoids obstacles and moves to a target
point. The distance in a straight line between the starting
point and the target point is 6.70m. It is required to control
the mobile robot without a collision with obstacles. The
robot is required a rather long movement for avoidingobstacles. When the task starts, the user can see the target
point via the immersive display. The purpose of this task isto evaluate the total maneuverability of the system.
The number of subjects for each task is nine. All of thesubjects are beginners for using the system.
3.2.2. Results and discussion
Table 1 shows the results of task 1 including the operation
time and the distance from the stopping point to the obstacle.
The negative value means the distance between the obstacle
and the stopping point after collision. The robot moves
about 1m in the total time delay of 2 seconds when it runs at2km/h. The results are within 1m. The subjects who are
required much time can control the robot close to theobstacle. It should be noted that subjects are required to pay
attention to the time delay when the distance between therobot and the obstacle is less than the distance to which therobot moves in the time delay.
Fig. 5 shows the results of task 2 including theoperation time and the trajectory of the mobile robot. Few
subjects changed the direction of the robot to the target
point at first and moved straight forward. Many othersubjects moved and rotated to the target point
simultaneously. Since the distance between the starting
point and the target point is large enough, the subjects could2119

Table 1. Results of task1: Operation time and distance between
stopping point and obstacle.
Operatora b c d e fghi
Operation time (sec.) 14 16 12 22 17 12 13Distance (cm)

between the robot and obstacles, and (iv) the time delay islong.
4. CONCLUSION

22 15

1 -35 40 -5 70 35 100 -10 10

adjust the direction of the robot to the target point
repeatedly. It is confirmed that subjects can control therobot intuitively corresponding subjects’ movement and themobile robot movement when the subjects do not need to
pay attention to obstacles.
Fig. 6 shows the results of task 3 including theoperation time and the trajectory of the mobile robot. In thistask, the skill of controlling the robot correctly for avoiding
obstacles is required. Some subjects are required much time
for the task. We observed that the reasons for this wereeither the speed is reduced for controlling the robot, orchanging direction is repeated for avoiding the obstacles.
Many subjects have reported that the task is difficult due tothe followings: (i) there is no sense other than vision fromthe environment around the robot, (ii) the resolution ofpresented video is low, (iii) it is difficult to feel the distance

GGS

G

S

a:18secS

b:35secG

c:42secGSGS

d:19secS

f:21sec

e:22sec
GGSGS

S

g:26sec
h:26sec
i:23sec
Fig. 5 Results of task 2: Operation time and trajectory.
GSG

S

a:41secS

b:93secGS

c:91sec
GGSS

d:70sec

f:132sec

e:135sec
GSG

GS

GS

g:50sec
h:50sec
i:71sec
Fig. 6 Results of task 3: Operation time and trajectory.

We have constructed a remote control system for a mobile
robot with a locomotion interface and an immersive display.
The usability of the system has been evaluated withexperiments.
Future work includes the followings. For the time delay,
the graphics WS will show the estimated present location ofthe mobile robot on the immersive screen by drawing CG ofthe robot. The mobile robot should be equipped withobstacle detecting sensors. When the sensor detects a close
obstacle, the mobile robot and the treadmill should stop forfeedback to a user.
5. REFERENCES[1] D. G. Caldwell, A. Wardle, and M. Goodwin, “Tele-presence:visual, audio and tactile feedback and control of a twin armed
mobile robot,” Proc. IEEE Int. Conf. on Robotics and Automation,
Vol. 1, pp. 244-249, 1994.
[2] I. Elhajj, N. Xi, and Y. Liu, “Real-time control of internet
based teleoperation with force reflection,” Proc. Int. Conf. on
IEEE Robotics and Automation, vol. 4, pp. 3284-3289, 2000.
[3] P. Jensfelt and S. Kristensen, “Active global localization for amobile robot using multiple hypothesis tracking,” IEEE Trans. on
Robotics and Automation, vol. 17, pp. 748-760, 2001.
[4] D. A. Bell, J. Borenstein, S. P. Levine, Y. Koren, and J. Jaros,“An assistive navigation system for wheelchairs based upon
mobile robot obstacle avoidance,” Proc. IEEE Int. Conf. on
Robotics and Automation, vol. 3, pp. 2018-2022, 1994.
[5] D. Schulz, W. Burgard, D. Fox, S. Thrun, and A.B. Creemers,“Web interfaces for mobile robots in public places,” IEEE
Robotics and Automation Magazine, Vol. 7, No. 1, pp. 48-56, 2000.
[6] H. Hirukawa, T. Matsui, H. Onda, K. Takase, Y. Ishiwata, andK. Konaka, “Prototypes of teleoperation systems via a standard
protocol with astandard human interface,” Proc. IEEE Int. Conf.
on Robotics and Automation, Vol. 2, pp. 1028-1033, 1997.
[7]. N. Yokoya, K. Yamazawa, and H. Takemura, “Interactivemedia-oriented applications of an omnidirectional video camera,”Proc. World Automation Congress, Vol.13, pp.11-16, 2002.
[8]. R. Kurazume and S. Hirose, “Development of image
stabilization system for remote operation of walking robots,” Proc.
IEEE Int. Conf. on Robotics and Automation, Vol.2, pp.1856-1861,2000.
[9]. K. Yamazawa, Y. Yagi, and M. Yachida, “Omnidirectional
imaging with hyperboloidal projection,” Proc. Int. Conf. onIntelligent Robots and Systems, Vol.2, pp.1029-1034, 1993.
[10] H. Iwata, “Walking about virtual environments on an infinitefloor,” Proc. IEEE Virtual Reality, pp.286-293, 1999.
[11]. K. Kobayashi, A. Ogawa, S. L. Casner, and C. Bormann,“RTP payload format for DV (IEC 61834) video,” Internet
Requests For Comments, RFC 3189, 2002.
[12]. Y. Onoe, K. Yamazawa, H. Takemura, and N. Yokoya,“Telepresence by real-time view-dependent image generation from
omnidirectional video streams,” Computer Vision and Image
Understanding, Vol.71, No.2, pp.154-165, 1998.

2120