The Design of Internet-Based RobotPHONE
Dairoku Sekiguchi1, Masahiko Inami2, Naoki Kawakami1 and Susumu Tachi1
12Graduate School of Information Science and Technology,
The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, JAPAN{dairoku, kawakami, tachi}@star.t.u-tokyo.ac.jp

Department of Mechanical Engineering and Intelligent Systems,
The University of Electro-Communications
1-5-1 Chofugaoka, Chofu-shi, Tokyo 182-8585, JAPANinami@computer.org

Abstract
RobotPHONE is a Robotic User Interface (RUI) thatuses robots as physical avatars for interpersonalcommunication. The shape and motion of RobotPHONE
is continuously synchronized by a bilateral control
method. Using RobotPHONE, users in remote locations
can communicate shapes and motion with each other. Inthis paper, we will present the design of an InternetBased RobotPHONE. To use RobotPHONE via theInternet, we developed a time delay compensation
control method. In this paper we also discuss this newcontrol method.
Key words: Robotic User Interface, Bilateral Control
Method, Internet, Communication Device

1. Introduction
RobotPHONE is a communication device based on theconcept of a Robotic User Interface (RUI). In contrast toa Graphical User Interface (GUI), which uses thecombination of WIMP (Window, Icon, Menu, PointingDevice), RUI utilizes a robot as an interface for a humanbeing. The concept of a GUI that originated from NLS(On-Line System) by Douglas C. Engelbart [1] and Altoby Alan Key [2] changed the way computers are used.
GUI simplified the use of a computer and played an
important role in the spread of computers for public use.
However, because GUI is based on WIMP, recently theinteraction method used by GUIs has been recognized aslimited when interacting with our real world situations.
NaviCam [3] and Tangible Bits [4] are attempts toovercome such GUI limitations; by using a physicalobject that exists in the real world as the interface. Manyof these attempts use a see-through head mounted
display (HMD) or a projector to output information to
the user. Therefore, considering that direct interaction
with such outputted information is still limited, webelieve that an output method that makes use of a real
object hasn’t yet been established.
On the other hand, personal robots such as pet robots [5]are a good example of utilizing a real object. Contrary toa computer graphics (CG) character on a computer
display, these robots have a physical body and thatexistence attracts people. A robot can be regarded as acomputer with a physical body that enables it to interact
with the real world. Hence, considering its strong
capability to interact with the real world, a robot would
be efficient interface having an input and an output
method for the real world. As well, if we regard therobot as a general-purpose machine, it is possible toconsider that it is a universal interface. Robotic User
Interface (RUI) is the terminology for this type ofinterface. That is, the concept of using a robot as aninterface between the real world and the informationworld can be referred to as a RUI. An intelligent robot asa physical entity for an Artificial Intelligence agent or ahaptic feedback robot arm used in virtual reality (VR)systems are good examples of a RUI.
RobotPHONE [6] is a RUI system for interpersonal
exchanges that uses robots as agents for physicalcommunication. The RobotPHONE system employs
robots that are called shape-sharing device. The shapeand motion of remote shape-sharing devices are always
synchronized.
Operations to the robot, such as
modification of posture, or input of motion, are reflectedto the remote robot in real-time. Therefore, users ofRobotPHONE can communicate and interact with eachother by exchanging the shape and motion of the robot.
NetworkSynchronize the shape and motion

Fig.1 The concept of RobotPHONEIf an ordinary telephone is considered to be a device fortransferring voice, RobotPHONE is a device fortransferring motion. Further, because RobotPHONE uses
a physically existing robot as the input and output
interfaces, a user of the RobotPHONE system candirectly feel the force of a user at the other side of thesystem. Therefore, users of RobotPHONE can get
motion information not only as a visual sensation but
also as a tactile sensation; it is thus possible to say that
RobotPHONE is a new type of telephone that cansimultaneously transfer visual, haptic and auditory
information. It can also be said that RobotPHONE is asystem that makes it possible to have an object exist
virtually in a remote place on behalf of a user.
RobotPHONE transfers the existence of the user not byattempting to transmit the user them self but to transmit
the user's substitute. Consider a mother giving a child astuffed animal to keep the child company at night. Thisis a form of communication aided by a physical entity.
RobotPHONE can facilitate this kind of communication.
In this paper, we present the system design of our
RobotPHONE, which can stably communicate overcommunication links, such as the Internet, with timevarying delays.

Fig.3 The mechanism of the RobotPHONE
RobotPHONE software running on each PC controls theRobotPHONE through the USB connection andsynchronizes the motion of two RobotPHONEs indistant places by exchanging data over the Internet. The
RobotPHONE software also transfers voice data inaddition to motion data. Therefore a user cancommunicate using motion and voice. The
RobotPHONE software uses the RCTP 2.0 protocol [7]for transferring motion data. Fig.4 shows the screen
image of the RobotPHONE software.

2. System overview
Fig. 2 shows an overview of the Internet-based
RobotPHONE system.
RobotPHONE

RobotPHONE

ControlControlPCPCUSBCommunicationUSBCommunicationInternetFig.2 System overview
As shown in Fig.2, this system uses two teddy bear-likerobots (RobotPHONE) as a shape-sharing device. This
humanoid type robot has 2 degrees of freedom at eacharm, 2 degrees of freedom for the head for 6 degrees offreedom in total (Fig. 3). All DC motors of theRobotPHONE are controlled by microcontroller insidethe device. The RobotPHONE onboard controller has a
USB connector that is used to connect it to a PC. For
voice communication, the RobotPHONE has amicrophone and a speaker, which can be linked to thePC’s sound device.

Fig.4 RobotPHONE software

3. Bilateral control with time delay
In this section, we explain the control method, used intheInternet-based
RobotPHONEsystemforsynchronizing the motion of the RobotPHONE.
Ordinarily, to synchronize motion between distantplaces, as the RobotPHONE does, the bilateral control
method would be used. However, if there is a time delayin the communication line, it is well known that such a
communication delay will destabilize the system if aconventional control method like a symmetrical bilateralcontrol method is used. This will especially be a major
problem when the RobotPHONE is used over acommunication line, such as the Internet, with timevarying delays. To overcome this problem, several
methods have been proposed. Anderson and Spong [8]proposed a time delay compensating bilateral control law
based on scattering theory. Oboe and Fiorini [9]proposed a PD-type controller. However, these methods
are relatively complex schemes; therefore, if thesemethods are applied to the RobotPHONE, it will be amore complex system than necessary. In addition, as thetime delay becomes longer, these methods tend toincrease the equivalent damping and inertia of thesystem for system stability, and so the system becomessticky and heavy as a result.
As described earlier, the goal of the RobotPHONE is tobe a communication device that can be used like anordinary telephone; thus, a user should be able to always
manipulate the RobotPHONE in the same manner
regardless of any communication line time delay. Hence,
it is difficult to apply the above methods toRobotPHONE. We therefore decided to use a newmethod that would be the most suitable for the
RobotPHONE.

4. External force transferring bilateral control
methodTo create a suitable bilateral control method for the
RobotPHONE, we first focused on the fact that previous
approaches were mainly designed for a teleoperation
system. In a teleoperation system, the operator performs
some tasks like grasping some object at the slave side at
a place distant from the master side. Hence, in ateleoperation system it is very important to match as
much as possible both position and force between themaster and the slave. On the other hand, theRobotPHONE is a communication device, whichtransfers motion to each other, and is not intended to dosome complex task at a distant place. Thus, it can be said
that the precise reflection of the position between themaster and the slave is not so important with the
RobotPHONE. Therefore, we thought that we could use
a new bilateral control scheme, which transfers motionand force to each other but does not actively match
positions between the master and the slave. Our basic
idea is as follows:

voice echo canceller.
Based on this idea, we developed a new control methodreferred to as the external force transferring bilateralcontrol method for the Internet-based RobotPHONE. An
impedance representation of the external force
transferring bilateral control method is as follows.

Fig.5 Impedance representation of the external force
transferring bilateral control method
In Fig.5, the dynamics of the master and the slave
systems are described as follows:
Fm − F 1 = MmX&&m + BmX&m + KmXm
F 2 − Fs = MsX&&s + BsX&s + KsXs

(1)
(2)

Where Xm and Xs are the respective positions of themaster and slave arms, Mm and Ms are the respectiveinertias, Bm and Bs are the respective dampings, Km andKs are the respective elasticities, F1 and F2 are therespective motor torques, Fm is the master input torqueand Fs is the slave output torque.
In addition, we consider the following impedance
representation of the operator, master-slave system andenvironment.

Each device only sends information caused byexternal force.

Fig. 6 Impedance representation of the operator, masterslave system and environment

By following this scheme, only movement information
caused by an applied force is sent to the opposite device
and, the opponent device does not send back any
information caused by the received information.
Therefore, this control system can be basically regarded
as a pair of two independent control systems and it canbe said that communication line time delay does notaffect the system. A similar kind of idea can be seen in aIn Fig. 6, the dynamics of the operator and theenvironment are given as follows:

Fop − Fm = MopX&&m + BopX&m + KopXm(3)

Fs − Fobj = MeX&&s + BeX&s + KeXs

(4)

Where Mop and Me are the respective inertias of theoperator and the environment, Bop and Be are therespective dampings, Kop and Ke are the respective
elasticities, Fop is the operator applied torque and Fe isthe torque from the environment.
From (1) and (3), we get(F 1 = Fop − MopX&&m + BopX&m + KopXm− MmX&&m + BmX&m + KmXm

())

(5)

From (2) and (4), we get((F 2 = Fobj + MeX&&s + BeX&s + KeXs+ MsX&&s + BsX&s + KsXs))

position offset to the transferred motion.
As noted, we intend to apply this control method to theRobotPHONE communication device. Thus, we thinkthat this position inconsistency would be negligible.
Furthermore, in an actual situation using theRobotPHONE, users will send motions to each other notcontinuously but intermittently. Therefore, byintroducing a method that moves each arm to the default
position in a motionless period, we can periodically reset
position inconsistencies between master and the slave.

5. Experimental Results(6)

The following diagram shows the control scheme of ourexperimental system for verifying the external force
transferring bilateral control method.

Let us consider the situation where external force isapplied to the master and this information is transferredto the slave without load. In this case, the equation (5)becomes

Fop = MmX&&m + BmX&m + KmXm

(7)

Where F1 is 0 and Mop, Bop and Kop is 0. Then, theequation (6) becomes
F 2 = MsX&&s + BsX&s + KsXs

(8)

Fig.7 The bock diagram of the experimental system

Where Fobj is 0 and Me, Be and Ke is 0. A simple control
scheme for the external force transferring bilateralcontrol method is then given by
(9)
(10)

Thus the motion information of the master and slavearms are described as follows

master130

500msslave

120
Positions

F 1 = Fs
F 2 = Fm

140

110

100

MmX&&m + BmX&m + KmXm = MsX&&s + BsX&s + KsXs

(11)
90

Where Fop is Fm. This equation shows that the impedance
parameters must be equal between the master and slavearms for synchronizing positions and motions of themaster and slave arms. In addition, if another parameterslike the impedance parameters of the operator or theenvironment are not 0, unlike the above assumptions,this also leads to position inconsistency between themaster and the slave.
However, we think that it is possible to minimize
difference of positions and motions between the masterand the slave by matching the impedance parameters
between them as much as possible. If the impedance
parameters are sufficiently matched between the masterand the slave, temporal impedance mismatch between
the operator and the environment, or the existence of an
external force (Fobj) are accumulated as position
deviation between the master and the slave, and act as a805001000

1500

2000
Time (ms)

2500

3000

3500

Fig.8 The external force transferring bilateral control
method with 500ms time delay
Fig. 8 shows the experimental results of the externalforce transferring bilateral control method with 500ms
time delay. As seen in Fig.8, the slave arm successfully
traced the motion of the master arm, and at the end ofeach arm motion, a 500ms gap, which corresponds to thetime delay, is observed. After this gap, each arm
remained in the same position and no oscillation wasobserved. We performed this experiment with up to 5
seconds time delay. Through all experiments, this systemshowed no unstable behavior under all the time delay
conditions we tested.

Master

6. Related work

SlaveObstacle

Bilateral control

Fig. 9 Experimental setup with obstacle
Fig. 9 shows the next experimental setup. In thisexperiment, an obstacle was placed under the slave arm
to be hit by the operation from the master arm.
160

140

Slave hit the obstacle

Positions120

Force feedback motion
100

master
80

500ms
500

1000

1500

2000

2500

3000

inTouch was tested with Internet connection [12], using
motion prediction and control parameter tuning for timedelay compensation; the method seemed to be sensitive
to increases of the time delay.

7. Conclusion

slave

60

There are several works that share haptic sensation orthe position of the object for interpersonalcommunication. Ishii’s inTouch [10] synchronizes therotation of rollers under the hand. Fogg’s HandJive [11]synchronizes the movement of a balloon and a stick
gripped in the user’s hand. PsyBech(PhysicallySynchronized Bench) by Ishii [12] synchronizes theposition of an object on the workbench. However, theseworks synchronized only the motion or position of theobject. We think that the combination of shape andmotion is especially important, and this combination
results in the embodiment of realistic communication.

3500

Time (ms)Fig.9 The slave hit the obstacle with 500ms time delay
Fig. 9 shows these results. At first, the slave arm
followed the motion of the master arm with 500ms
delay. Then, the slave arm hit the obstacle, halted its
motion, and sent back the external force information to
the master. At the same time, the master already stopped
its motion; the information from the slave took 500ms toreach the master because of communication line timedelay between the master and the slave. 500ms after thetime of the hit on the obstacle, the external force
information reached the master and the master output
force according to the information from the slave. As
shown in Fig.9, this force output from the mastergenerated a counter motion to the inputted motion by theoperator and becomes force feedback to the operator.
Through a series of such events as above, both masterand the slave showed no oscillation. This experiment
was also conducted with up to 5 seconds time delay;very stable behavior was observed under all time delay
conditions we tested.
We implanted an external force transferring bilateralcontrol system and experimentally confirmed that thiscontrol method is stable under various time delay
conditions.

In this paper we briefly explained our concept of RUI
and showed our design for the Internet-based
RobotPHONE. To compensate for the communicationline time delay observed in the Internet, we proposed a
new bilateral control method called the external force
transferring bilateral control method for the Internetbased RobotPHONE. We discussed this control methodand implemented it in an experimental system. Throughthe implementation, we experimentally confirmed its
stability under various time delay conditions, and wedemonstrated the feasibility of our Internet-based
RobotPHONE design.

References
1.

2.
3.

4.

5.

6.

D C. Engelbart, and W. K. English: A ResearchCenter for Augmenting Human Intellect,
Conference Proceedings of the 1968 Fall Joint
Computer Conference, San Francisco, CA, Vol. 33,
pp. 395-410, 1968
A. C. Key: Microelectronics and the Personal
Computer. Scientific American 237, no. 3, pp. 230244, Sep. 1977
J. Rekimoto: NaviCam: A Magnifying Glass
Approach to Augmented Reality Systems, Presence:
Teleoperators and Virtual Environments, Vol. 6,No. 4 pp.399-412, MIT Press, 1997
H. Ishii, and B. Ullmer.: Tangible Bits: Towards
Seamless Interfaces between People, Bits andAtoms, in Proceedings of CHI '97, pp. 234-241,
1997
M. Fujita and H. Kitano: Development of an
Autonomous Quadruped Robot for RobotEntertainment, Autonomous Robots vol.5, pp.7-8,
Kluwer Academic Publishers, 1998
D. Sekiguchi, M.Inami, S. Tachi: RobotPHONE:
RUI for Interpersonal Communication, CHI2001
Extended Abstracts, pp. 277-278, 2001

7.

D. Sekiguchi, W. Teng, Y. Yanagida, N.
Kawakami, S. Tachi: Development of R-Cubed
Manipulation Language - The design of RCML 2.0system, Proc. of ICAT 2000, pp. 44-51, 2000.
8. R. J. Anderson and M. W. Spong, Bilateral control
of teleoperators with time dlay. IEEE Trans. on
Automatic Control, 34(5), pp. 495-501, 1998
9. R. Oboe, P. Fiorini, A design and control
environment for internet-based telerobotics, Int. J.
Robotics Res., 17(4), pp. 433-449, 1998
10. S. Brave and A. Dahley, inTouch: A Medium for
Haptic Interpersonal Communication, Extended
Abstracts of CHI '97, pp. 363-364, ACM Press,
1997
11. B. J. Fogg, L. Cutler, P. Arnold and C. Eisback,
HandJive: a device for interpersonal hapticentertainment, Proceedings of CHI '98, pp. 57-64,
ACM Press, 1998
12. S. Brave, H. Ishii and A. Dahley, Tangible
Interface for Remote Collaboration andCommunication, Proceedings of CSCW '98, pp.
169-178, ACM Press, 1998