Toward the Next Generation Telexistence
Susumu Tachi
The Univers�ty of Tokyo
Department of Mathemati回I Engineering and Information Physics
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656Japan
http：仇阿w.st町.tル加kyo.aりplAbstract: Telexistence (tel・existence) is a concept named for血e technology which enables a hum組
being to have a real time sensation of being at血e place o血er白血the place where he or she actuallyexists，加d is able to interact wi血由eぉmote 姐d/or virtual environments. He or she C佃 ”telexist ” （tel・exist) in a real world where the robot exists or in a virtual world that a computer has generated. It is also
possible to telexist in a mixed environment of real ar凶virtual, which can be called augmented
telexistence. It enables a human operator to have a sensation of being in aぉmote real environment wherea surrogate robot exists, while being augmented by a vi武田1 environment synthesized by a computer,
whose structure is based on由e sensor infonnation on board the robot. In仕tis keynote paper，血e conceptof telexistence is explained and experimental telexisten田町御ms are in回duced. The concept oftelexistence, i.e 叩 virtual existence in a remote or computeトgenerated environment, has developed into anational R&D scheme of R-Cubed侭eal・time Remote Robotics) for the advanced and comfortable life of也e 21st century networlc society. R-Cubed can be said to be the effort toward出e next generation oftelexistence. Based on the scheme the National R&D Project of ”Humanoid and Human Friendly
Robotics，”HRP in short, w部 launched in April 1998. This is an e妊M to integrate telerobotics, networlc
technology and virtual reali勿into networked telexistence. A next generation telexistence technology
using E勤1P (Head Mounted Projector) and re住0 ・reflective screen is also proposed and feぉibility
experiments are conducted.
Keywords: tele対stence, tele-existence, telepresence, virtual reality, master-slave manipulation

1. Introduction
It has long been a desire of human beings to project
themselves in the remote environment, i.e., to have asensation of being present or existing in a different place
other than the place they really exist at the same time.
Another dream has been to amplify human muscle powerand sensing capability by using machines while 民seIVinghuman dexterity with a sensation of direct operation.
In the late 1960s research and development program wasplanned on a powered exoskeleton由at a man would wear
like a garment. A concept of H紅diman was proposed byGeneral Elec国C Co., for example，出at a man wearing theHardiman exoskeleton would be able to command a set ofmechanical muscles that multiply his stren併1by a f泊ctorof 25, yet in this union of man and machine he would feel
object and forces almost as江he were m必rect contact.

IMEKO-XV WorldCongress

However. the project was unsuccessful because of thefollowing reasons: (1) it is potentially quite dangerous towear a powered exoskeleton when we consider potential
malfunction of the machine. (2) Space inside the machineis quite valuable to store computers, controllers，印刷atorsand energy source of the machine, which eliminated出espace for a human operator. Thus, the design proved
impractical in its original form.
ラWi出血e advent of science and technology, however, it
has become possible to challenge for the realization of thedreams again with a different concept. The concept ofpr，句ec出1g ourselves by using robots, computers andcybernetic human interlace is 回lled telexistence （鈴1・existence）.官邸concept expan也to include projection in
a computer-generated v註陶al environment. Fi♂m 1illus回tes血e original idea of telexistence using血eoriginal figure published in 1982 in Japanese and in 1984
in English [1].

ー 173 -feasibility of the method was evaluated [2r Figure . 3
shows the telexistence vehicle during an experiment.

I)

The first prototype telexistence master slave system forremote manipulation experiments was designed anddeveloped, and a preliminary evaluation experiment oftelexistence was conducted. An experimental telexistencesystem for real and/or virtual environments was designed
and developed, and by conducting an experimentcomparing a telexistence master-slave system withFig. 1 Principle of Telexistence [1 ].
The concept of the telexistence was proposed by theauthor in 1980 and it played the role of the fundamental
principle of the eight year Japanese National Large Scale
Project of ”Advanced Robot Technology in Hazardous
Environment，”which started in 1983 together with theconcept of the Third Generation Robotics. Through thisproject theoretical consideration has been done andsystematic design procedure has been established.
Experimental hardware telexistence system haven been
made and the feasibility of the concept has been
demons廿ated.
In our first reports [I], the principle of the telexistence
sensoηdisplay was proposed, and its design procedure
was explicitly defined. Experimental visual display
hardware was built, and the feasibility of the visualdisplay with the sensation of presence was demonstrated
by psychophysical experiments using the test appartus(Fig.2).

Fig. 3 Mobile Telexistence Vehicle [2].
conventional master-slave system, efficacy of thetelexistence master-slave system and the superiority of thetelexistence method was demonstrated experimentally(Fig.4) [3, 4, 5].

Fig. 4 Telexistence Surrogate Anthropomorphic Robot(TELESAR) at Work (5].

Fig. 2 First Experimental Apparatus Constructed .
A method was also proposed to develop a mobile
telexistence system, which can be remotely driven withthe auditory and visual sensation of presence. A prototype
mobile tele-vehicle system was constructed and the2. Augmented Telexistence
Telexistence can be divided into ti町o categories:
telexistence in the real world that. actu�!Iy �)fi府保adist叩ce, and is connected via a robot to the place whereー174-仇fEKO・XV World Cong同時

0「尚治il�ier is located; and附xis肌e in the virtual worldJtb.;1t cloes not actually exist but is creat巴d by a computer.
The former can be called ”transmitted reality，” while thet!1attet is”S戸1thesized reality. ” The synthesized reality can
もe classified into two, i.e., a virtual environment as a' model of the real world and a virtual environment of animaginary world. Combination of transmitted reality andsynthesized reality, which is called mixed reality, is also
possible and has a great importance in real applications.
This we call augmented telexistence to clari命theimportance of harmonic combination of real and virtualworlds in this paper.
Augmented telexistence can be used in several situations.
Take for instance, of controlling a slave robot in a poor
visibility environment. An experimental augmented
telexistence system using a virtual environment iscons甘ucted. The environment model is also cons甘ucted
食om the design data of the real environment. Whenaugmented reality is used for controlling a slave robot, themodeling errors of the environment model must becalibrated. A model-based calibration system using image
measurements is proposed for matching the realenvironment and a virtual environment. The slave robot
has impedance control mechanism for contact tasks andfor compensating for the errors that remain even after thecalibration. An experimental operation加a poor visibility
environment was successfully conducted by using Telesar(Fig.4) and the virtual Telesar (Fig.5). Figure 5 shows thevirtual telexistence anthropomorphic robot used in theexperiment and Fig. 6 shows how the real environment isaugmented by the computer model [6,7].

Fig. 6 An Experimental Augmented Reality.
However, it is difficult for everyone to telexist合間ly
through commercial networks like the Internet or the next
generation world-wide networks.

3. R-Cubed & HRP
In order to realize the society where everyone can freely
telexist anywhere through network, Japanese Ministry ofInternational Trade and Industry (MITI) proposed a long­range national R&D scheme, which is dubbed R・Cubed(Real-time Remote Robotics) in 1995 [8,9].
Based on the scheme and after two-year feasibility study
called Human Frien社ly Network Robot (FNR), which wasconducted from April 1996 till March 1998, NationalApplied Science & Technology Project，”Humanoid andHuman Friendly Robotics (H貯），＇’ has Just been launched.
It is a five-year project toward the realization of so-called
R-Cubed Society by providing humanoids, control
cockpits and remote control protocols.
Figure 7 shows an example of an artist ’ s image of afuture use of R Cubed System. In this example, ahandicapped person climbs a mountain with his合iends
using networked telexistence.
酬In an R-Cubed system, each robot site has its server ofits local robot. The robot type varies合om a humanoid(high end) to a movable camera (low end). A virtual robot
can also be a controlled system to be telexisted.

Fig. 5 Virtual TELESAR at Work.
Quantitative evaluation of the telexistence manipulation
system was conducted through tracking tasks by using atelexistence master slave system designed and developed.
Through these experimental studies, it has been
demonstrated that a human being can telexist in a remoteenvironment and/or a computer-generated environment byusing the dedicated telexistence system [5].

/MEKO-XV World Congress

Each client has its teleoperation system. It can be acontrol cockpit with master manipulators and a head
mounted display (HMD) or CAVE Automatic VirtualEnvironment (CAVE) on the high end. It is also possible
to use an ordinary personal computer system for its
control system on the low end. In order to support the low
end users to control remote robots through networks,
RCML/RCTP (R-Cubed Manipulation Language / R­
Cubed Transfer Protocol) is now under development [9].

-175-To standardize the following control scheme, a language
called RCML, which describes a remote robot’ s features

If the user has a device such as 6 degrees-of.・企eedom(DOF) position/orientation sensor to indicate the robot­
manipulator’s endpoint, the user can employ that instead
ofthe conventional GUI panel (See Fig.8).

4. The Next Generation
A Head Mounted Display (HMD) and a CAVE 訂e two
typical virtual reality visual displays. Although由ey arequite useful displays, it is also ue that白ey have somedemerits. The former has a problem of回deoff of highresolution and wide field of view, and the latter has aproblem of a shadow of user’s body on a virtualenvironment and interaction of user’s virtual body with
their real body.
甘In our laboratory at ぬe Unive凶q ofTo防o, a new typeofvisual display is being developed [10). It is called X’tal
vision, and it uses retro-reflective material as its screen.

Fig. 7 Mountain Climbing using R-Cubed.
and its working environment, has been propo叩d. A
communication protocol RCTP, which is designed to
exchange control commands, status da旬，and sensory
information between the robot and the user, has also beendeveloped.
一♂：叩：竺竺佐野…

！出；A projector is arranged at the conjugate position of auser’s eye, and an image is projected on a screen made of,painted with, or covered with re甘0・，reflective material. A
pinhole is placed in 合ont of the projector to secureadequate depth of focus (Fig.9).
軒。J 闘torι：�ぷニムーI '-"·'·"'

S.n制『Driveri竺竺三三ζ正二i

in：：：了七1
1－·閥w’叫円

r.r-1；－－鞠0四，. ...

,,,.. .. ,三選立こ些コ1Fig. 8 An Example of an RCML Browser 18].
With a Web browsers a user accesses a Web site
describing information of a robot in the form ofhypertext
and icon graphics using WWW browser. Clicking on anicon downloads the description file, which is written in
RCMLお，rmat, to the user’s computer and launches theRCML browser. The RCML browser p釘ses thedownloaded file to process the geometry information,
including the arrangement of the degrees of合eedom of
the robot, controllable parameters, available motion
ranges, sensor information, and other pertinent
information .. The browser decides what kind and how
many devices 訂e required to control the remote robot. It
白 en generates a graphical user interface (GUI) panel tocontrol the robot, plus a video window that displays theimages ” seen ” by the robot and a monitor window thatlets users observe the robot's status合om outside the robot.

Fig. 9 Principle of X'tal Vision.
The retro-reflector screen together with the pinhole
assures that由e user always sees images with accurate
occlusion relations. It means白紙if the user’s body has
re 0・reflector on it, their body becomes a part of thevirtual environment and it disappears, and their virtual
body replaces it, while if it does not have re甘o-reflector
on it, it will occlude the virtual environmept ，�ithout any
troublesome hindrance shadow on the virtual environment.
甘In the construction of X ’tal vision, screeµ s:hape昇A.�earbitraη，， i.e., any shape is· possil;,J号.、It fo. qµe,.t9 Jhecharacteristics of the re甘0・reflector 如 g the Ptl\lWls}:inJhe
By using 、 νJh,e 事!lffieco吋ugate optical system.

-176-JMEKO-XVw。，1d C<酬唱同盟characteristics of x ’ta! vision, binocular stereo vision
becomes possible using only one screen with an arbitrary
shape. This can be mounted on a head of a user, which wecall HMP (Head Mounted Proj 巴ctor) System.

Thus mutual telexistence becomes possible by using
X'tal vision method, i.e., not only the user sees otherpeople naturally but also the user of the robot can beobserved naturally by other people. We are now in theprocess of feasibility study of the proposed method using
Telesar.

Almost eighteen years have passed since our first idea
and concept of telexistence, and it is now possible totelexist in the remote environment and/or virtualenvironment with a sensation of presence. We can havefeelings that we are present in several real places and c釦
work and act. However, those people in the place where
someone telexists using a robot see only the robot but
they can not feel that the person presents. It is useless touse TV display on board the robot to show the face of theuser. It is just comical and far合om reality.

Figure 11 shows an example of how a robot can be seen
by a human being who wears a HMP. It can be seen as if
the robot is a human being telexisting in the robot. Figurell(A) shows an miniature of the HONDA Humanoid
Robot, while Fig. 1 l(B) illus廿ates the robot painted with阿佐0 ・reflective material. Figures ll(C) and (D) show how
they are seen by a human being wearing a HMP. The
telexisted robot just looks like the human operator of therobot, and telexistence can be naturally done.

Figure IO illustrates the proposed method of mutual
telexistence using X'tal vision HMP (Head MountedProjector) in order to solve the above problem, i.e., tomake a telexisted robot look like the user of the robot.
This is an effort toward the next generation telexistence.

5. Conclusion

A human user ”A ” uses his telexistence robot ” A ” at the
remote site where another human user ”B ” is present. Theuser ”B ” also uses another telexistence robot ”B ” ，which
exists in the site where the user ” A ” works. Both robots
are painted with retro-reflective material and can act as
screens, and they are controlled by their users asconventional telexistence robots.
Remote scenery sensed by cameras on board the robots
A ” and ”B ” are sent to HMPs of human users ” A ” and” ”B ，respectively. 3・D image observed by the telexistence
robot ”A ” is projected and seen by the human user ” A ”
with a sensation of presence, while 3・D image observedby the telexistence robot ”B ” is projected and seen by thehuman user ” B” with a sensation of presence.
”References

MoUon Con加l/T副ile c。耐olv;sual 畠Audi旬rylnfo1官官tion
JD Image ob時四回byτelexlste何回Robot＇官－
Is pre羽田佳d and S居”
by man Usef· 、w詑ha s聞satl n p『眉剖鴎”切。 。’

Telex1s包neeRob ’A＂・。r。

Tele：剖stence R bor·a··
is s田n .. ；，靴脂HumanU曽f'B"'bythe p叫回日目ond陪al image of'B 目、”the robot目官＂.

τelexis旬nee Rob。r’A＇’
格seen a, if It隠HumanU蝿r"A"bythe proiect陶n。f realimage or官、n柏eroboγ＇A・I

V国,al &Aud陶ry Information。

Motion C ntrol/Tac臨Cont,olFig. 10 Next Generation Telexistence [11].
The telexistenc巴 robot ” B ” is seen as if it is the humanuser ” B ” by the projection of real image of ” B ” on therobot ” B ” ，while the telexistence robot ” A ” is seen as if itis the human user ”A ” by the projection of real image of” ”
A on the robot ” A” ．

イ＇MfKO-XVWorld Cong開店

Virtual means “ existing in e任ect or in essence though
not in actual fact or form." Thus virtual reality must havethe essence of the reality in its computer-generated
environment or a transmitted remote environment so that
it is effectively the reality itself. One of the mostpromising technologies today is the integration of virtual
realty and robotics on the network. It is called neれ/\forked
robotics in general and R-Cubed (Real-time RemoteRobotics) in paロicular. R-Cubed is a Japanese nationalR&D scheme toward the realization of the next generation
telexistence through various kinds of networks including
the Internet. Japanese Ministry of International Trade andIndustry (MITI) launched the 5・year Pr供ct ” Humanoidand Human Friendly Robotics (HRP〕 ” in April 1998. Thisis the first step toward the realization of R-Cubed, thenext generation telexistence, and the results are quite
much expected.

[l�S.Tachi, K.Tanie, K.Komoriya and M.Kaneko: Tele­existence (I): Design and evaluation of a visual displaywith sensation of presence, Proceedings of the 5thSymposium on Theory and Practice of Robots andManipulators (RoManSy '84), pp.245-254, Udine, Italy,(Published by Kogan Page London), June 1984.
[2]S.Tachi, H.Arai, I.Morimoto and G.Seet: Feasibility
experiments on a mobile tele-existence system, TheInternational Symposium and Exposition on Robots (19thISIR), Sydney, Australia, November 1988.
[3]S.Tachi, H.Arai and T.Maeda: Development of an
Tele-existence
Slave
Robot,
Anthropomorphic
Proceedings of the International Conference on Advanced
Mechatronics (ICAM), pp.385・390, Tokyo, Japan, May
1989.
[4]S.Tachi, H.Arai and T.Maeda: Tele-existence masterslave system for remote manipulationラ IEEE International
Workshop on Intelligent Robots and Systems (IROS’90),
pp.343・3 48, 1990.

ー177-[5]S.Tachi and K.Yasuda: Evaluation Experiments of aTele-existence Manipulation Sysytem, Presence, vol.3,no.I, pp.35・44, 1994.
[6]Y.Yanagida and S.Tachi: Virtual Reality System withCoherent Kinesthetic and Visual Sensation of Presence,
Proceedings of the 1993 JSME International Conference
on Advanced Mechatronics (ICAM), pp.98・ 103, To防o ラJapan, August 1993.
[7]K.Oyama, N.Tsunemoto, S.Tachi and T.lnoue:
Experimental study on remote manipulation using virtualreality, Presence, vol.2, no.2, pp.112 ・124, 1993.
[8]S.Tachi: Real-time Remote Robotics- Toward
Networked Telexistence : IEEE Computer Graphics andApplications, vo.18, Nov-Dec98, pp.6-9, 1998.
[9]Y.Yanagida, N.Kawakami and S.Tachi: Developmentof R-Cubed Manipulation Language ・ Accessing realworlds over the network, Proceedings of the 7thInternational Conference on Artificial Reality and Tele­
Existence (ICAT'97), pp.159・164, Tokyo, December 1997.
[10] N.Kawakami, M.Inami, T.Maeda 如d S.Tachi: Media
X ’ta] -Projecting virtual environments on ubiquitous
object-oriented retro-reflective screens in the realenvironment-, SIGGRAPH'98, Orlando, FL, July 1998.
[ll]S.Tachi: Augmented Telexistence, Mixed Reality,
pp.251・260, Published by Springer-Verlag, 1999.

Fig. 11 (A) Miniature of the HONDA Humanoid
Robot, (B) Painted with Retro-reflective Material, (C)An Example of Projecting a Human Image on it, (D)Another Example (11].

ー178-

IMEKO-XV WorldCongress