Proceedings of the 5th Virtual Reality International Conference (VRIC2003)
pp.69/1-69/9, Laval Virtual, France, May 13-18, 2003

Telexistence and Retro-reflective Projection
Technology (RPT)
Susumu TACHI, Ph.D.
Professor
The University of Tokyo
Department of Information Physics & Computing
7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656 Japan
http://www.star.t.u-tokyo.ac.jp/
Abstract. This paper reviews how the original telexistence (tele-existence) technology has been developed, and describes a
newly developed method of mutual telexistence using projection technology onto retro-reflective objects dubbed RPT(Retro-reflective Projection Technology). Telexistence is fundamentally a concept named for the general technology that enables
a human being to have a real-time sensation of being at a place other than where he or she actually exists, and being able tointeract with the remote environment, which may be real, virtual, or a combination of both. It also refers to an advanced type of
teleoperation system that enables an operator at the control to perform remote tasks dexterously with the feeling of existing in asurrogate robot working in a remote environment. Telexistence in the real environment through a virtual environment is also
possible. Although conventional telexistence systems provide an operator the real-time sensation of being in a remoteenvironment, persons in the remote environment have only a feeling that a surrogate robot is presented, not the operator. Mutual
telexistence aims to solve this problem so that the existence of the operator is apparent to persons in the remote environment by
providing mutual sensations of presence. This has become possible by the use of RPT.

1. Introduction
Telexistence (tele-existence) is a technology thatenables us to control remote objects andcommunicate with others in a remote environment
with a real-time sensation of being there by using
surrogate robots, remote/local computers andcybernetic human interfaces. This concept has
been expanded to include projection of ourselves
into computer-generated virtual environments, andalso the use of a virtual environment for theaugmentation of the real environment.
The author proposed the concept of the telexistence
in 1980, and the concept of telexistence became thefundamental principle of the eight year Japanese
National Large Scale Project of "Advanced Robot
Technology in Hazardous Environments," whichstarted in 1983 together with the concept of Third
Generation Robotics. Through this project, wemade theoretical considerations, establishedsystematic
designprocedures,developed
experimental hardware telexistence systems, anddemonstrated the feasibility of the concept.
Thanks to the efforts of twenty years of research anddevelopment, it has nearly become possible forhumans to use a humanoid robot in the remoteenvironment as if it was an other self, i.e. they areable to have the sensation of being just inside therobot in the remote environment.
Althoughexisting telexistence systems succeeded in providing
an operator a real-time sensation of being in a remoteenvironment, persons in the remote environment did
not have the feelings that the human operator ispresented, but only a surrogate robot. Mutual
telexistence aims at solving this problem so that theexistence of the operator is apparent by the persons
in the remote environment by providing mutual
sensations of presence.
This paper reviews the original telexistence
technology, and introduces a method of mutual
telexistence based on the projection of real-time
images of the operator on the surrogate robot, basedon RPT (Retro-reflective Projection Technology).

2. Short History of Telexistence
Figure 1 shows the concept of telexistence in realenvironments, virtual environments, and the realenvironment through a virtual environment(augmented telexistence). The following describes
the research and development conducted in order torealize the concept.
Our first report [1] proposed the principle of thetelexistence sensory display, and explicitly definedits design procedure. The feasibility of a visualdisplay with a sensation of presence was

VRIC 2003 Proceedings –69/1

demonstrated through psychophysical measurements
using experimental visual telexistence apparatus.
A method was also proposed to develop a mobile
telexistence system that can be driven remotely withboth an auditory and visual sensation of presence.
A prototype mobile televehicle system wasconstructed and the feasibility of the method wasevaluated [2].

Fig.1 Concept of Telexistence.

In 1989, a preliminary evaluation experiment oftelexistence was conducted with the first prototype
telexistence master slave system for remotemanipulation. An experimental telexistence system
for real and/or virtual environments was designed
and developed, and the efficacy and superiority ofthe telexistence master-slave system overconventional
master-slave
systemswasdemonstrated experimentally [3, 4, 5].

To use augmented reality in the control of a slave
robot, a calibration system using image
measurements was proposed for matching the realenvironment and the environment model [6,7].
The slave robot has an impedance control
mechanism for contact tasks and to compensate forerrors that remain even after calibration. An
experimental operation in a poor visibility
environment was successfully conducted by using ahumanoid robot called TELESAR (TELExistence
Surrogate Anthropomorphic Robot) (Fig.2) and its
virtual dual. Figure 3 shows the virtual TELESAR
used in the experiment and Figure 4 shows themaster system for the control of both real TELESAR
and virtual TELESAR.
Experimental studies of tracking tasks demonstrated
quantitatively that a human being can telexist in aremote and/or a computer-generated environment byusing the dedicated telexistence master slave system[5].
A networked telexistence paradigm called R-cubed(Real-time Remote Robotics) was proposed in 1986,and several research efforts towards the goal arebeing conducted, including a real-time remote robot
manipulation language dubbed RCML [6,7].

Fig. 3 Virtual TELESAR at Work (1993).

Fig. 2 Telexistence Surrogate Anthropomorphic Robot(TELESAR) at Work (1988).

Augmented telexistence can be effectively used innumerous situations. For instance, to control aslave robot in a poor visibility environment, an
experimental augmented telexistence system wasdeveloped that uses a virtual environment model
constructed from design data of the real environment.
VRIC 2003 Proceedings –69/2

Fig. 4 Telexistence Master (1989).

Telexistence technology was adapted in METI
(Ministry of Economy, Trade and Industry)'s national
five year HRP (Humanoid Robotics Project:
1998-2003) for a new type of cockpit system tocontrol a humanoid bipedal robot, as shown in
Figure 5. The telexistence cockpit was completed
for this project in March 2000 (Fig.6). It consists ofthree main subsystems: an audio/visual display
subsystem, a teleoperation master subsystem, and acommunication subsystem between the cockpit andthe humanoid robot [10,11,12].

Fig.5 HRP Humanoid Robot at Work (2000).

Various teleoperation tests using the developedtelexistence master system confirmed thatkinesthetic presentation through the master systemwith visual imagery greatly improves both theoperator's sensation of walking, and dexterity atmanipulating objects.
If the operator issued a command to move the robot,
the robot actually walked to the goal. As the robot
walked around, real images captured by a wide fieldof view multi-camera system were displayed on four
screens of the surrounded visual display. This
made the operator feel as if he or she was inside therobot, walking around the robot site.

Fig.6 Telexistence Cockpit for Humanoid Control (2000).

A CG model of the robot in the virtual environment
was represented and updated according to the current
location and orientation received from sensors on thereal robot. The model was displayed on thebottom-right screen of the surround visual display,
and when augmented to the real images captured by
the camera system, it supported the operator'snavigation of the robot. Since the series of real
images presented on the visual display are integrated
with the movement of the motion base, the operator
feels the real-time sensation of walking, or stepping
up and down.
Through these efforts of more than twenty years, it
has become nearly possible for a human to use ahumanoid robot in the remote environment as if it
was his or her other self. Persons can control therobot by just moving their body naturally, without
using verbal commands. The robot conforms to theperson’s motion, and through the sensors on board
the robot the human can see, hear and feel as if theysensed the remote environment directly. Persons
can virtually exist in the remote environment without
actually being there in a sense.
For observers in the remote environment, however,the situation is different. They see only the robot
moving and speaking. Although they can hear thevoice and witness the behaviour of the humanoperator through the robot, it does not actually look
like him or her. This means that the telexistence is
not yet mutual.
In order to realize mutual
telexistence, we have been pursuing the use of
projection technology with retro-reflective material
as a surface, which we call RPT (Retro-reflective
Projection Technology).

VRIC 2003 Proceedings –69/3

3. Retro-reflective Projection Technology(RPT)
A Head Mounted Display (HMD) and a CAVE(CAVE Automatic Virtual Environment) are twoclassic virtual reality visual displays. Although
they are quite useful displays, it is also true thatthey have some shortcomings. The former has a
tradeoff of high resolution and wide field of view,and the latter has problems concerning the user’s
body casting shadows on a virtual environment,
and the interaction between the user's real bodyand the virtual interface.
In addition, both
displays have problems concerning occlusion whenin use under the augmented reality condition, i.e.
virtual objects and real objects are mixed. The
problems of conventional displays HMD and
CAVE are shown in Figure 7 (C) and (D),respectively.
Figure 7 (A) shows a virtual vase and a virtual
ashtray on a virtual desk. When a real hand isplaced between two virtual objects, an ideal
occlusion should be depicted as in Figure 7 (B), i.e.,
the real hand occludes the virtual vase and is
occluded by the virtual ashtray. However, a real
hand cannot occlude the virtual vase nor be occludedby the virtual ashtray when an optical see-through
HMD is used to display virtual objects, and the hand
and the ashtray look as if they are transparent. Thisis simply due to the fact that the physical display
position is always just in front of the eyes of the userwhen HMD is used.

On the other hand, the virtual ashtray cannot occlude
a real hand when IPT (Immersive Projection
Technology) like CAVE is used, as shown in Figure
7 (D). This is due to the fact that the display
position of virtual objects is always at the screen
surface, which is one to two meters away from thehuman user when CAVE and other IPT (Immersive
Projection Technology) displays are used.
In our laboratory at the University of Tokyo, a new
type of visual display is being developed called X'tal(pronounced crystal) vision [16,17], which uses
retro-reflective material as its projection surface.
We call this type of display technology RPT(Retro-reflective Projection Technology).
Under the RPT configuration, a projector is arranged
at the axial symmetric position of a user's eye withreference to a half-mirror, with a pinhole placed in
front of the projector to ensure adequate depth offocus, as shown in Figure 8. Images are projected
onto a screen that is either made, painted, or covered
with retro-reflective material [17].
Projector

Retroreflector

Pin Hole

Half Mirror
Operator

Fig.8 The Principle of RPT system.

Fig. 7 (A) A virtual vase and a virtual ashtray on a virtual desk;(B) An ideal occlusion when a real hand is placed between two
virtual objects; (C) Unfavorable results when optical

A retro-reflective surface projects back the projected
light just to the direction of projection, whileconventional screens normally used for IPT scatterprojected lights in all directions ideally as aLambertian surface (Fig.9). Figure 10 shows how aretro-reflective surface behaves. It is covered withmicroscopic beads of about 50 micrometers indiameter, which reflect back the incident light to theincident direction. It can also be realized with amicrostructure of prism-shaped retro-reflectors
densely placed on a surface.

see-through HMD is used; (D) Unfavorable results when IPT(Immersive Projection Technology) like CAVE is used.

VRIC 2003 Proceedings –69/4

Fig. 9 Three typical reflections.

Fig. 12 Head Mounted Projector (HMP) with RPT.

Fig. 10 Retro-reflective surface densely covered withmicroscopic beads with about 50 micrometer diameter. Ideally,refractive index should be 2.

The retro-reflector screen, together with the pinhole,ensures that the user always sees images withaccurate occlusion relations. In the construction ofan RPT system, screen shapes are arbitrary, i.e., any
shape is possible. This is due to the characteristics ofthe retro-reflector, and the pinhole in the conjugate
optical system. By using the same characteristics ofan RPT system, binocular stereovision becomes
possible using only one screen with an arbitrary
shape. The projector can be mounted on the headof a user, which we call an HMP (Head Mounted
Projector) system as shown in Figure 11. Figure 12
shows a general view of a prototype HMP.

Figure 13 shows an example of an image projected
on a sphere painted with retro-reflective material.
Figure 14 shows an example of projecting a virtual
cylinder onto a Shape Approximation Device(SAD) [18], which is a haptic device that enables
the user to touch geometrical shapes as if they
were real.
The use of the SAD as theretro-reflective screen enables us to feel just as wesee by using a HMP.

Fig.13 Projected image on a spherical retro-reflective screen.

In the figure, (A) illustrates the principle of SAD,
(B) shows an image to be displayed, (C) is anactual SAD, and (D) indicates the image projected
onto SAD, which can be touched as it is seen.

Fig.11 Principle of stereo display using RPT with Head Mounted
Projector (HMP).

VRIC 2003 Proceedings –69/5

Fig. 16 Another Example of the application of RPT to medicine.

Fig.14 Projected image on a Surface Approximation Device (SAD).
A: Principle of SAD; B: Image; C: Actual SAD; D: Image

Figures 17 and 18 show application examples ofusing RPT for optical camouflage. In Fig. 17,
pre-captured a background image is projected so thata retro-reflective object grasped by a human appears
to be transparent.

projected on SAD, which can be touched as it is seen.

Figure 15 shows an example of the use of RPT for
augmented reality. Pre-captured x-ray and/or MRI
data can be superimposed onto a human patient sothat a surgeon can have open surgery related
information even under the minimally invasive
surgery
environment.
Bysuperimposing
ultra-sonic data, real-time presentation of inner
body information is also possible through RPT.

Fig. 17 An optical Camouflage Application using RPT.

Fig.15 An Augmented Reality Application using PRT.

Figure 16 shows another example of applying RPT
to medicine.

Figure 18 shows how optical camouflage can beachieved using real-time video information. Figure
19 shows how RPT is applied to realize the situationof Figure 18. The coat is made of retro-reflective
material so that the coming light is reflected back tothe same direction that it comes from. Microscopic
beads on the surface of the coat have the function ofretro-reflection.
A half mirror makes it possible for a spectator to see
virtually from the position of the projector. An HMP
projects an image of the background scenery
captured by the video camera behind thecamouflaged subject. A computer calculates theappropriate perspective and transforms the captured
image to the image to be projected on the subject
using image-based rendering techniques. Since thecloak the subject is wearing is made of a special
retro-reflective material, which reflects back theincident light just the same direction it comes from,VRIC 2003 Proceedings –69/6

an observer looking through a half mirror sees a very
bright image of the scenery so that he is virtuallytransparent.

robots A' and B', and are sent to the HMPs of humanusers A and B respectively, both with a sensation ofpresence. Both telexistence robots A' and B' areseen as if they were their respective human users by
projecting the real image of the users onto theirrespective robots.
Motion Control / Tactile Control
Visual & Auditory Information

HMP

3D image observed by
Telexistence Robot B’is projected and observedby Human User B witha sensation of presence.

3D image observed by
Telexistence Robot A’
is projected and observed by
Human User A with asensation of presence.

Human
User A

Telexistence
Robot A’

Telexistence Robot B’
Telexistence Robot B’ isobserved as if it is Human
User B by the projectionof real image of B on
the robot B’.

Fig. 18 Another example of application of RPT to opticalcamouflage.

Telexistence Robot A’ is HMP
observed as if it is Human
User A by the projectionof real image of A on
the robot A’.

Human
User B

Visual & Auditory Information
Motion Control / Tactile Control

Fig. 20 Concept of Robotic Mutual Telexistence (adopted from[13]).

Fig. 19 A schematic diagram of the RPT system used for the
optical camouflage in Fig.18.

4. Mutual Telexistence using RPT
More than twenty years have passed since weinitially proposed the concept of telexistence, and it
is now possible to telexist in the remote and/or
virtual environment with a sensation of presence.
We can work and act with the feeling that we arepresent in several real places at once. However, inthe location where the user telexists, people see only
the robot but cannot feel that the person is actuallypresent. Simply placing a TV display on board therobot to show the face of the user is not verysatisfying, since it appears mostly comical and far
from reality.
By using RPT, the problem can be solved as shownin Figure 19 [13]: suppose a human user A uses his
telexistence robot A' at the remote site where anotherhuman user B is present. The user B in turn uses
another telexistence robot B', which exists in the sitewhere the user A works. 3-D images of the remotescenery are captured by cameras on board bothFig. 21 (A) Miniature of the HONDA Humanoid Robot,(B) Painted with Retro-reflective Material,(C) Example of Projecting a Human onto it,(D) Another Example (adopted from [13]).

Figure 21 presents an example of how mutual
telexistence can be achieved through the use of RPT.
Figure 20(A) shows a miniature of the HONDA
Humanoid Robot, while Figure 21(B) shows therobot painted with retro-reflective material. Figures
21 (C) and (D) show how they appear to a human
wearing an HMP. The telexisted robot looks justlike the human operator of the robot, andtelexistence can be naturally performed [13]. We
are currently in the process of a feasibility study for

VRIC 2003 Proceedings –69/7

the proposed method using TELESAR.

5. Conclusions
Projection technology on Retro-reflective surfaces iscalled RPT (Retro-reflective Projection technology),
which is a new approach to augmented reality (AR).
The first demonstration of RPT was made at
SIGGRAPH98, followed by demonstrations at
SIGGRAPH99 and SIGGRAPH2000. In thiskeynote paper, the principle of RPT is explained, andapplications to AR are discussed.
Mutual telexistence is one of the most important
technologies for the realization of networked
telexistence, because users "telexisting" in a robot
must know whom they are working with over the
network. A method using RPT, especially an HMP(Head Mounted Projector) and a robot withretro-reflective covering was proposed and proved to
be a promising approach toward the realization ofmutual telexistence.
Acknowledgments: The research presented here
has been conducted in part under the CREST
Telexistence Communication Systems Project,supported by JST (Japan Science and Technology
Corporation). This work has been supported bymany members and ex-members of Tachi
Laboratory of the University of Tokyo. Special
thanks to Dr. Naoki Kawakami, Dr. Masahiko
Inami and Dr. Dairoku Sekiguchi for theircontribution to the project.

References[1] S.Tachi, K.Tanie, K.Komoriya and M.Kaneko:
Tele-existence (I): Design and evaluation of a visualdisplay with sensation of presence, Proceedings of the 5th
Symposium on Theory and Practice of Robots andManipulators (RoManSy '84), pp.245-254, Udine, Italy,(Published by Kogan Page London), June 1984.
[2] S.Tachi, H.Arai, I.Morimoto and G.Seet: Feasibility
experiments on a mobile tele-existence system, The
International Symposium and Exposition on Robots (19thISIR), Sydney, Australia, November 1988.
[3] S.Tachi, H.Arai and T.Maeda: Development of an
Anthropomorphic
Tele-existence
Slave
Robot,
Proceedings of the International Conference on Advanced
Mechatronics (ICAM), pp.385-390, Tokyo, Japan, May
1989.
[4] S.Tachi, H.Arai and T.Maeda: Tele-existence masterslave system for remote manipulation, IEEE International
Workshop on Intelligent Robots and Systems (IROS'90),
pp.343-348, 1990.
[5] S.Tachi and K.Yasuda: Evaluation Experiments of aTele-existence Manipulation Sysytem, Presence, vol.3,
no.1, pp.35-44, 1994.
[6] Y.Yanagida and S.Tachi: Virtual Reality System with
Coherent Kinesthetic and Visual Sensation of Presence,
Proceedings of the 1993 JSME International Conference
on Advanced Mechatronics (ICAM), pp.98-103, Tokyo,
Japan, August 1993.
[7] K.Oyama, N.Tsunemoto, S.Tachi and T.Inoue:
Experimental study on remote manipulation using virtual
reality, Presence, vol.2, no.2, pp.112-124, 1993.
[8] S.Tachi: Real-time Remote Robotics- Toward
Networked Telexistence : IEEE Computer Graphics andApplications, vo.18, Nov-Dec98, pp.6-9, 1998.
[9] Y.Yanagida, N.Kawakami and S.Tachi: Developmentof R-Cubed Manipulation Language - Accessing realworlds over the network, Proceedings of the 7thInternational Conference on Artificial Reality andTele-Existence (ICAT'97), pp.159-164, Tokyo, Japan,
December 1997.
[10] T.Nishiyama, H.Hoshino, K.Suzuki, R.Nakajima,
K.Sawada and S.Tachi, "Development of Surrounded
Audio-Visual Display System for Humanoid Robot
Control", Proc.of 9th International Conference ofArtificial Reality and Tele-existence (ICAT'99), pp.
60-67, Tokyo, Japan, December 1999.
[11] T.Nishiyama, H.Hoshino, K.Suzuki, K.Sawada and
S.Tachi, "Development of Visual User Interface
Embedded in Tele-existence Cockpit for Humanoid Robot
Control", Proc. of IMEKO 2000 World Congress, Vol.XI(TC-17 & ISMCR2000), pp.171-176, Vienna, Austria,
September 2000.
[12] S.Tachi, K.Komoriya, K.Sawada, T.Nishiyama,
T.Itoko, M.Kobayashi and K.Inoue: Development ofTelexistence Cockpit for Humanoid Robot Control,
Proceedings of 32nd International Symposium on
Robotics (ISR2001), pp.1483-1488, Seoul, Korea, April
2001.
[13] S.Tachi: Augmented Telexistence, Mixed Reality,
pp.251-260, Published by Springer-Verlag, 1999.
[14] S.Tachi: Toward Next Generation Telexistence,
Proceedings of IMEKO-XV World Congress, vol.X(TC-17 & ISMCR'99), pp.173-178 , Tokyo/Osaka, Japan,
June 1999.
[15] S.Tachi: Toward the Telexistence Next Generation,
Proceedings of the 11th International Conference on

VRIC 2003 Proceedings –69/8

Artificial Reality and Tele-Existence, (ICAT2001), pp.1-8,
Tokyo, Japan, December 2001.
[16] N.Kawakami, M.Inami, T.Maeda and S.Tachi: Media
X'tal -Projecting virtual environments on ubiquitous
object-oriented retro-reflective screens in the realenvironment-, SIGGRAPH'98, Orlando, FL, July 1998.
[17] M.Inami, N.Kawakami, D.Sekiguchi, Y.Yanagida,
T.Maeda and S.Tachi: Visuo-Haptic Display using
Head-Mounted Projector, Proceedings of IEEE Virtual
Reality 2000, pp. 233-240, New Brunswick, NJ, March
2000.
[18] S.Tachi, S.Tachi, T.Maeda, R.Hirata and H.Hoshino:
A construction method of virtual haptic space, Proceedingof the 4th International Conference on Artificail Reality
and Tele-Existence (ICAT'94), pp.131-138, Tokyo, Japan,
July 1994.
[19] M. Inami, N. Kawakami, Y. Yanagida, T. Maeda and
S. Tachi, Method of and Apparatus for Displaying an
Image, US PAT. 6,283,598 2001.
[20] M. Inami, N. Kawakami, Y. Yanagida, T. Maeda and
S. Tachi, Method and Device for Providing Information,
US PAT. 6,341,869 2002.
[21] S.Tachi: Two Ways of Mutual Communication:
TELESAR
and
TWISTER,in
S.Tachied.
Telecommunication, Teleimmersion and Telexistence,
IOS Press, ISBN 1-58603-338-7, pp. 3-24, 2003.

VRIC 2003 Proceedings –69/9