SICE Journal of Control, Measurement, and System Integration, Vol. 1, No. 5, pp. 335–344, September 2008

TELEsarPHONE: Mutual Telexistence Master-Slave Communication
System based on Retroreflective Projection Technology
Susumu TACHI ∗ Naoki KAWAKAMI ∗ Hideaki NII ∗ Kouichi WATANABE ∗ Kouta MINAMIZAWA ∗Abstract : TELEsarPHONE is a conceptual prototype of a mutual telexistence system, designed for face-to-face telecommunication via robots. Because of the development of telexistence technology, we can acquire a feeling that we arepresent in several actual remote places using remote robots as our surrogates and can work and act freely there. However,people in the place where someone telexists using a robot see only the robot, and they cannot feel the existence of thetelexisting person. Mutual telexistence aims to solve this problem so that the existence of a telexisting person (visitor)
is apparent to the people in the remote environment by providing mutual sensations of presence. On the basis of theconcept of mutual telexistence, we have designed and developed a prototype of a telexistence master-slave system forremote communication by applying retroreflective projection technology. In the TELEsarPHONE system, the face andchest of the slave robot TELESAR II are covered by retroreflective material. To provide the feeling of existence, thereal-time image of the visitor is projected onto the robot so that people can see the visitor in real time.
Key Words : telecommunication, teleoperation, telexistence, virtual reality, augmented reality, retroreflective projection,
human robot systems, system integration

1.

Introduction

Telexistence (tele-existence) is a fundamental concept thatenables a human being to have a real-time sensation of being
at a place other than where he or she actually is and being able
to interact with a remote environment, which may be real, virtual, or a combination of both. It also refers to an advanced
type of teleoperation system that enables an operator to perform remote tasks dexterously with the feeling of existing in asurrogate robot working in a remote environment.
Before the concept of telexistence was proposed, there wereseveral systems that aimed for a similar goal. In the US, Sutherland [1] proposed the first head-mounted display system, which
led to the birth of virtual reality in the late 1980s. This concept
was the same as telexistence in computer-generated virtual environments. However, it did not include the concept of telexistence in real remote environments. In Italy, Mancini et al.[2]developed a mobile teleoperated robot system, Mascot, as early
as in the 1960s. In France, Vertut et al.[3] developed a teleoperation system for use in deep submergence technology in
1977; this system controlled submarines. Although these remote robots were not humanoid-type robots and no sensationof presence was provided in a strict sense, the systems wereclosely related to the concept of telexistence in real remote environments, and they could be regarded as the forerunners oftelexistence.
In order to intuitively control a remote humanoid robot, it isimportant to locally provide the operator with a natural sensation of presence as if the operator were actually present at the
remote site, by means of visual, auditory, and haptic sensations.
The concept of providing an operator with a natural sensation∗

Department of Information Physics & Computing, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656,
Japan
E-mail: tachi@tachilab.org(Received June 17, 2008)of presence to facilitate dexterous remote robotic manipulation
tasks was termed “telepresence” by Minsky [4] in USA and“telexistence” by Tachi et al.[5] in Japan.
The concept of telexistence was proposed and patented inJapan in 1980 [6], and became the fundamental guiding principle of the eight-year Japanese national large scale project “Advanced Robot Technology in Hazardous Environments,” which
was initiated in 1983 together with the concept of third generation robotics. Through this project, theoretical considerationsof telexistence were made, systematic design procedures were
established for it, experimental hardware telexistence systems
were developed, and the feasibility of the concept of telexistence was demonstrated.
Through the eﬀorts of twenty-five years of research and development in the US, Europe, and Japan [7]–[23], it has almost
become possible for humans to use humanoid robots in a remote environment as if they were their other persons, i.e., humans are able to have the sensation of being inside the robots
in the remote environment.
Although conventional telexistence systems succeeded inproviding an operator with a real-time sensation of being in aremote environment, human observers in the remote environment did not have the sensation of the presence of the humanoperator; rather, the observers saw only a surrogate robot. Mutual telexistence addresses this problem so that the existence ofthe operator is apparent to people in the remote environment by
providing mutual sensations of presence [24]–[26].
A method for mutual telexistence based on the projection ofreal-time images of the operator onto a surrogate robot, which
in turn is based on RPT (retroreflective projection technology),
was first proposed in 1999 [20], and the feasibility of the concept was demonstrated by constructing an experimental mutual telexistence system using RPT in 2003 [24] and 2004 [25].
However, the demonstrated system had only a 6-DOF (degreeof freedom) head mechanism and no manipulation mechanism.

c 2008 SICEJCMSI 0005/08/0105–0335 ⃝336

SICE JCMSI, Vol. 1, No. 5, September 2008

In this study, we have developed a master-slave manipulation
system with the function of mutual telexistence. We also propose the concept of using a mutual telexistence system as a newcommunication tool as an extension of the telephone and videophone and employ the system to demonstrate the feasibility ofthe proposed concept.

2.

Concept of TELEsarPHONE System

The history of communication shows that communication
methods have evolved over the past few decades. The telephone
is a communication device that uses sound for communication
and a videophone uses both sight and sound for communication. The next generation of communication devices will carryout communication by haptics along with sight and sound.
“Body presence” is an element that is necessary to convey
the feeling that a remote participant is present at a site. Many
audio-visual systems that attempt to convey this feeling havebeen proposed and implemented, such as remote meeting systems using videophones. However, such systems do not aﬀect
any physical action in the remote environment; that is, they are
only eyes and ears located at the remote location. To achieve a
realistic sensation, it is necessary that the person “telexists” at
the remote location.
For this purpose, it is necessary to set up a surrogate bodyof a person and an artificial system that (1) performs physicalactions in the remote environment on the basis of the person’s
motion through the surrogate body and (2) lets the person feelsensations felt by the surrogate body in the remote environment.
We propose to let a surrogate robot participate at a site, and weestablish a system in which a remote participant feels the robot
as his or her representative and acts via the surrogate robot as if
it were a part of his or her body.
We have been conducting research on telexistence and a technology for operating surrogate robots in a remote environment,
and have found that a robot can be used as a communication
tool. Traditionally, robots are classified into Astro-Boy-type
robots with self-directed intelligence and Gundam-type robots
with human operation. However, we have confirmed the existence of a third type of robot, an alterego robot, which acts andcommunicates on behalf of its operator.
“Mutual existence” refers to participants at a site being awareof the existence of a remote participant when they encounter asurrogate robot that is operated by the remote participant. Ourearlier studies on telexistence focused on a remote participantacquiring a realistic sensation, and we developed a demonstration system that was successful in transferring the realistic sensation to the remote participant. However, the participants at thesite felt as if they were talking with a robot, and the existence ofthe remote participant was surprisingly weak. While the voice,sound, and action of the remote operator were transferred tothe site, essential visual information was not conveyed. The
most important elements of nonverbal communication, such as
facial expression, were not transferred, and as a result, the “existence” became weak. To strengthen existence, the face of theremote participant should be projected onto the head of the representing robot in the correct direction. Furthermore, assumingthat several people are standing around the robot, displaying asingle image on the head by using LCD monitor [26] is insuﬃcient. Rather, multiple images should be projected in diﬀerent
directions.

Fig. 1

Conceptual image of TELEsarPHONE.

TELEsarPHONE is a conceptual prototype of a mutual telexistence system, designed for face-to-face telecommunication
via robots (Fig. 1). TELEsarPHONE is composed of three elements: a humanoid robot, a cockpit for operating the robot, anda viewer system. Both the robot and the cockpit are located at
diﬀerent sites and connected by a high-speed communication
network. The video, audio, and tactile information of what the
robot sees and feels is sent to the cockpit, where the information is reconstituted three-dimensionally. Conversely, the robot
reflects the actions of the operator sitting in the cockpit, and theoperator feels as if he or she is actually at the place where therobot is and is also able to behave as he or she wants.
The viewer system is intended for those who are communicating with the operator through the robot. The body of therobot contains a screen onto which images of the operator thatare captured in the cockpit are projected. The use of the RPT,
which enables observers to individually observe a proper image
from their own perspective, enables observers in the area surrounding the robot to feel and communicate as if the operator
is actually there.

3. Mutual Telexistence Master-Slave System for
Telecommunication
To implement the concept of TELEsarPHONE, a new prototype of a mutual telexistence master-slave system for communication has been designed and developed. The mutual telexistence master-slave system is based on the RTP and composedof 3 subsystems - slave robot TELESAR II, master cockpit, andviewer system - as is shown in Fig. 2.
The robot constructed for this communication system iscalled “TELESAR II (Telexistence surrogate anthropomorphic
robot II).” In order to use this system for telecommunication,
we have designed the robot by focusing on reproducing humanlike realistic movement. TELESAR II has two human-sizedarms and hands, a torso, and a head. Its neck mechanism has 2
DOFs, which can rotate around pitch and roll axes. There are
2 CCD cameras located in its head for stereoscopic vision. It
also has 4 pairs of stereo cameras on top of the head for a 3D
surround display for the benefit of an operator. A microphonearray and a speaker are also employed for auditory sensationand verbal communication. Each arm has 7 DOFs, and eachhand has 5 fingers with a total of 8 DOFs.
To control the slave robot, we have developed a master cockpit for TELESAR II. The cockpit consists of two master arms,
two master hands, multi-stereo display system, speakers and a

337

SICE JCMSI, Vol. 1, No. 5, September 2008

microphone, and cameras for capturing the images of an operator in real time. In order that an operator can gesture smoothly,
each master arm has 6-DOF structures so that the operator’s elbow is free from constraints. To control the redundant 7 DOF
of the anthropomorphic slave arm, we place a small orientation
sensor on the operator’s elbow. Therefore, each master arm
can measure 7-DOF motion for each corresponding slave arm,
while force is fed back from each slave arm to each corresponding master arm with 6 DOFs.
The master arm is lightweight and impedance control is carried out so that the operator feels as if he or she is inside theslave robot. It is important that the master can apply an exact
force to an operator and the slave robot maintain safe contact
with humans in a remote environment. The impedance-controltype master-slave system adopted by us can achieve the force
presentation. Moreover, by using the slave, we can maintainsafe compliant contact with humans because the slave is subjected to impedance control. The motion of the head on the
robot is synchronized with the motion of the operator’s head;these motions are measured by a head tracker in the mastercockpit. The operator can easily control the hands of TELESAR II because the motion of the operator’s hands is measured
by the master cockpit and controlled by master-slave methods.
In the case of autonomous robot system, such a system should
perform precise computation to prevent the collision of the arm,
hand or torso of the robot. In the case of telexistence, however,the system does not require collision detection. The operator
calculates it subconsciously. This is a remarkable feature of
the telexistence system. Despite this, we calculate the collision limit, and collision can be prevented even if the operator
fails to avoid collision (fail-safe or safety intelligence). Figure 3 shows the general outline of the master-slave impedancecontrolled teleoperation system used in this study.
The most distinctive feature of the TELEsarPHONE systemis the use of the RPT viewer system. Both the motion and thevisual image of the operator are important factors for feelingthe existence of the operator at the place where the robot is
working. In order to view the image of the operator on theslave robot such that the operator is inside the robot, the robot
is covered with retroreflective material and the image captured
by a camera at the master cockpit is projected on the TELESAR
II. TELLESAR II acts as a screen, and a person seeing throughthe RPT viewer system observes the robot as if it is the operator
because of the projection of the real image of the operator onto
the robot.

4.

Telexistence Surrogate Anthropomorphic Robot:
TELESAR II

4.1

Slave Robot Arm

TELESAR II has two 7-DOF arms, as shown in Fig. 4. Eacharm is designed such that its weight is minimum in order thatit can move rapidly and is safe for human use. By uniting the
housing parts of a harmonic drive gear system with other parts,
such as the rotational axes of joints, we have ensured that theentire mechanism of the arm is very lightweight. The weightof the arm is 7.3 kg and its payload is 0.5 kg. The maximum
velocity of the arm is 1.2 m/s. The slave arm supports suﬃcient
payload and speed for mutual telexistence by using gestures,and since it is considerably lighter than existing arms [27], thepotential danger of injury due to malfunction is also greatly re-

Projector
System for
RPT Viewer
3

CameraSystem for
RPT Viewer

Stereo MultiLiquid Crystal
Display System

Stereo Multicamera System
4x2

4x2

3

Network

Slave Arm
Slave Hand

Master Arm
Master Hand

Slave Robot

Fig. 2Master Cockpit

Schematic diagram of TELESAR II master-slave manipulation
system.
Control PC

Local ImpedanceControl
Kv

Networked
Shared Memory
Impedance
Position, Posture, ForceControl PC
Local ImpedanceControl
JT

Kv

S
Inversekinematics

J
Kp

Kp
kinematics
Encoder

Encoder

Motor

Motor
Force/TorqueSensorFig. 3Force/TorqueSensorSchematic diagram of impedance-controlled teleoperation.

duced. At the same time, our slave arm has a larger range ofmobility in the joints above the elbow. Three shoulder joints
J1, J2, and J3 have a mobility range of -180 degree to 180 degree, 0 degree to180 degree, and -180 degree to 180 degree,
respectively. The reduction ratio of the harmonic drive of eachjoint of the slave arm is set to 50 to maintain back-drivability.
The maximum force at the tip of the slave arm is 164 N.
The motor driver controls the DC motors in the joints bymeans of a torque control mode, according to commands received from a DA board. The angular velocity and posture ofeach joint are measured using an encoder attached to the motor.
The neutral point of each joint is defined by photo-interrupters,
whose signals are read by the AD board. The slave arm’s
control system is connected to the master arm system throughshared memory. The control system for the master arm is thesame as that for the slave arm.
The distribution of the joints of the arm replicates the structure of the human arm in order to facilitate operation by telexistence using kinesthetic sensation. This human-mimicked structure is also useful for interaction with people because the operator can fully use the sensation of congruity.
4.2 Slave Robot Hand
Each slave hand has 5 fingers. Its thumb has 3 DOFs while
the remaining 4 fingers have 1 DOF. The hand has also abduction DOF, and altogether 8 DOFs. The hand weighs 0.5 kg.
The size of the hand is similar to that of a typical human, i.e., alength of 185 mm, width of 100 mm, and thickness of 35 mm.
All parts such as motors, gears, and encoders are packed in-

338

SICE JCMSI, Vol. 1, No. 5, September 2008

Fig. 4

Fig. 5

Slave arm (left: overview; right: structure of right arm).

Slave hand (left: right hand with right arm; right: right-hand palm).

side the hand mechanism. The hand is connected to the control
system, i.e., a servo amplifier and control computers, with only
one cable.
Figure 5 shows the general view of the slave hand used in ourstudy.

5. Telexistence Cockpit
5.1

Master Arm

The salve arm of TELESAR II and a general anthropomorphic slave arm are built as 7-DOF mechanisms; these arms havethe same structure as a human arm. The master arm used as theteleoperation system is also usually built a 7-DOF structure.
However, it is normally diﬃcult to achieve the free motion of
the operator if the master arm has active 7 DOFs because it
tends to restrain the operator’s elbow mechanically. When we
think of force to be applied to the operator’s hand, it is alonga maximum of six axes, i.e., 6 DOFs, even the motion of thehuman arm has 7 DOF. Thus, we have designed the master arm
such that it eﬀectively performs the function of force presentation in these six axes; we have also built the master arm as a
6-DOF mechanism.
While the force feedback mechanism is suﬃcient for 6 DOFs,
it is necessary to have 7 DOFs for the measurement of humanarm motion. The master arm has a cantilever beam structure as
serial links; therefore, if the DOF of the master arm increases,
the length of the cantilever beam and the total weight of theactuators also increase, thereby decreasing the rigidity and stability of the master arm. Since the measurable movement of
the master arm that follows the operator’s hand has 6 DOFs,
we use a new lightweight posture sensor composed of an acceleration sensor [28] to measure the final DOF, which is critical
to identifying the posture of the operator’s entire arm. Altogether, the master arm serves as a master system with 7 DOFs
for measurement of the arm’s posture and 6 DOFs for force
presentation. Since the posture sensor is very lightweight compared to the mechanical restraints on the operator’s elbow, thesensor enables considerably high movement of the opeartor’s
arm without any undesirable load on it.

Fig. 6Master arm (left: structure; right: overview).

An exoskeleton structure is widely used because it can be
adopted for the movement of an operator with minimal size requirements, which is an essential requirement for correspondence to various everyday actions of humans. Structure andgeneral view of the master arm are shown in Fig. 6.
A potentiometer and an encoder are installed in each joint,
and the operator’s initial posture is computed from the output
signals of the potentiometers. During movement, the joint angles and angular velocities of the operator’s arms are computedfrom the output signals of the encoders. The three axes of thejoints in the master arm’s wrist cross at one point and a 6axis force sensor (MINI 4/20, BL AUTOTEC) is attached tothat point. Output signals of this sensor are used to measure
the force acting between the wrists of the master arm and operator in the direction of the rotating axis and the torque thatacts around the axis. The exoskeleton-type multifingered master hand is attached to the tip of the master arm and a bilateralsystem including fingers is realized.
A gravity compensation system is realized by suspending awire at the tip of the master arm, which enables the manipulator
to yield the maximum performance. The tension of the wire is
20.6 N. Because of this gravity compensation system, the actuators of the joints of the master arms do not have to compensate
for the gravitational torque applied to the master arm. Therefore, the master arm is able to present forces to the operator’s
hand with a small output torque and high accuracy. A passive
link that has two joints is attached above the master arm. A
constant force spring runs through the link. Constant tension
acts on the wire by passing it through a pulley at the tip of thespring. The wire is attached to the tip of the master arm. Since
the joints of this link are parallel to the direction of gravity, thelink can follow the master arm smoothly by means of the wirethat runs through the master arm, while maintaining a horizontal posture. The maximum force at the tip of the master arm is
239 N.
As our master arm has 6 DOFs and the slave arm has 7
DOFs, a simple symmetric servo cannot be employed between
corresponding joints. Our two requirements are precise hand
movement: and communication by gestures. For satisfying thefirst requirement, the position and orientation of the slave arm’s
wrist must coincide with those of the master. 6DOFs are used
for this purpose. For satisfying the second requirement, theslave’s posture must be as similar as possible to that of the operator. The remaining 1 DOF is used for this purpose. It should
be noted that the conventional method based on the pseudo inverse of the Jacobian matrix is not appropriate because it does
not satisfy the second requirement.

339

SICE JCMSI, Vol. 1, No. 5, September 2008

Fig. 8Fig. 7

Acceleration sensor attached to operator’s upper arm along swivel
axis.

There are some possible techniques for measuring an operator’s posture, such as the use of markers and cameras, which
are adopted in general motion capture systems. However, theoptical method has two problems: time delay and occlusion.
Therefore, we consider another simpler method mentioned below to avoid these problems. By this measurement, the masterside has 7 DOFs for position and orientation and 6 DOFs forforce feedback.
As the operator’s arm can be regarded as a redundant manipulator, conventional methods to solve the inverse kinematics of aredundant arm can be applied to measure the operator’s posture.
One of the popular methods is to define the swivel angle of anarc of a circle, which the elbow traces; this angle lies on a plane
whose normal is parallel to the wrist-to-shoulder axis. Given
the wrist position, orientation, and elbow swivel angle, an algorithm can compute the joint angles analytically. It should be
noted that the wrist position and orientation of the operator are
identical to those of the master manipulator. In order to acquire
the remaining information (swivel angle), we use an acceleration sensor (ADXL202E, Analog Devices, Inc.) attached to theoperator’s upper arm. The sensor has a suitable high-frequency
response (1 kHz), while its small size and low weight permit
the operator to move his or her arm freely. We use an axis of
the sensor corresponding to the change in the swivel angle, as
shown in Fig. 7.
5.2

Master hand (left: structure; right: overview).

Master Hand

We have developed a new type of master hand, as shown in
Fig. 8. It has the following two features. One is the compact
exoskeleton mechanism of the master hand’s finger contrived tocover wide workspace of an operator. The exoskeleton mechanism can be placed either over (parallel joint) or beside (coaxial
joint) an operator’s finger. The former placement has a disadvantage that the master arm’s finger obstructs the motion of theoperator’s finger when the operator’s finger is bent. However,the latter placement is diﬃcult because there is little space toplace the exoskeleton mechanism. To solve this problem, we
have proposed a “circuitous joint” that coincides the joint axisof the master hand with that of the operator by extending thelink length in proportion to the joint angular displacement, asshown in Fig. 9 [29].
The other feature of the master hand is the encounter-typeforce feedback [30]. An encounter-type device remains at thelocation of the object in the remote envirunment and waits foran operator to encounter it. As shown in Fig. 10, our encountertype master hand’s finger usually follows the operator’s finger

Fig. 9Fig. 10

Basic scheme of circuitous joint.

Encounter-type master-slave hand system.

without physical contact. It enables the operator to touch nothing when the slave hand does not touch anything. When theslave hand touches an object, the master finger stops its movement so that the operator’s finger touches a plate of the masterhand. The plate provides both the feeling of contact and an
appropriate resistive force. Therefore, our master hand is able
to provide both perfect unconstrained motion and natural touchsensation.
5.3 3D Display System
If we use a head-mounted display (HMD) for displaying the3D (three-dimensional) scenery of the place where a surrogate
robot is working, people at that place inevitably see an operator
with a HMD, which is not preferable from the viewpoint of thedesired face-to-face communication.
Therefore, we have constructed a 3D display system consisting of four 3D displays (SynthaGram 204: 20 inch lenticulartype LCD display) arranged in front and on the left, right, andbottom, thereby forming a T-shape. Because of lenticular lenses
placed on the display surface of the LCD display, the operator can view a stereoscopic image without wearing any special
glasses such as shutter glasses or polarized glasses.
3D camera system is located on top of the robot. Since
the display system is fixed, the camera system should also be
fixed. The 3D camera system consists of four pairs of CCDs (8340

SICE JCMSI, Vol. 1, No. 5, September 2008

Fig. 11

Fig. 12

RPT (left: retroreflection; center: principle of RPT; right: overview of RPT system).

Example of RPT view (left: TELESAR II without projection; center: with projection of an
operator; right: operator at the control).

CCDs). The pairs of cameras are for front, right, left, and bottom views. Each image captured by these cameras is transferred
to its respective display. The system provides an approximatedegocentric view of the operator.

6. RPT Viewer System
In our laboratory at the University of Tokyo, a new type of visual display termed an RPT display is being developed, whichuses retroreflective material as its projection surface [31]–[34].
The retroreflective surface functions as a special screen. In the
RPT configuration, a projector is arranged at the axial symmetric position of a user’s eye with reference to a halfmirror, witha pinhole placed in front of the projector to ensure an adequate
depth of focus. Figure 11 shows the principle of the RPT system and a general view of the system.
The face and chest of TELESAR II are covered by retroreflective material. A ray coming from a particular direction isreflected in the same direction on the surface of the retroreflective material. Because of this characteristic of the retroreflective material, an image is projected onto the surface of TELESAR II without distortion. Since we use many RPT projectors
in diﬀerent directions and project diﬀerent images corresponding to the cameras placed around an operator, observers canview the corresponding images of the operator.
Figure 12 shows an example of projecting images of an operator to its surrogate robot.

7. Feasibility Experiments
In order to demonstrate the feasibility of the concept of the
TELEsarPhone system, we constructed a hardware system andexhibited it at Expo 2005 Aichi Japan as one of the prototype
robots at the Morizo and Kiccoro Exhibition Center from June
9 through June 16 [35].

Fig. 13Booth layout (left: cockpit booth; right: robot booth).

Two booths were constructed. One was a cockpit booth andthe other was a robot booth (Fig. 13). We assumed that the
cockpit booth was located in Tokyo and the robot booth was
located in Paris. The robot booth was assumed to be a store
specializing in stuﬀed animals, and the store had a telexistence
communication robot in order to greet a foreign customer without actually having to travel. A person in the robot booth, which
was supposed to be located on a street in Tokyo as a future extension of a telephone booth on the street, could visit the store
in Paris without actual travel using the telexistence communication system (Fig. 14).
The person (operator) would sit down and wear master arms
and hands and log into the surrogate robot in Paris through a
dedicated network. Then, the robot would move its arms andhands corresponding to the operator’s motion. The operator
could obtain a 3D view of the shop by using the autostereoscopic display in front of him or her. The operator could communicate with the clerk in the shop using a headset comprising
a microphone and speaker. Using the master-slave communication system, the operator could communicate with the clerk
by gestures or shake hands with the clerk. The operator could

SICE JCMSI, Vol. 1, No. 5, September 2008

Fig. 14

General view of the booths (left: cockpit booth; center: robot booth; right: human robot
communication).

Fig. 15

Mutual telexistence using RPT (left: without RPT viewer; center: through RPT viewer; right:
observer viewing through RPT).

Fig. 16

not only select the stuﬀed animal of his or her choice but also
handle the selected stuﬀed animal using the master-slave manipulation system.
Thus, the person in Tokyo could acquire the feeling of visiting the store in Paris and select the merchandise he or she
desired to buy. The clerk in Paris could also see the visitor at
the shop by using the RPT viewer system. Figure 15 shows anexample of such views. In the Aichi Expo, we set up 3 RPTviewers in front and on the right side and center of the booth.
Observers could simultaneously see the front view, right-handside view, and left-hand side view of the face of the operator.
Three cameras were arranged corresponding to the RTP
viewers. Three persons were able to see the operator’s images
simultaneously. One person was facing the front side of the operator’s face, while the remaining two people were diagonallyfacing the left- and right-hand sides of the operator’s face. They
could see the corresponding images of the operator, i.e., frontal,
diagonally left, and diagonally right, respectively. As the operator turned his or her face, he or she could communicate with
one of the three people face to face.
The main features of the proposed system are as follows:(1)nonverbal communication is possible in addition to the conventional verbal communication and (2) face-to-face communication is possible under the condition that several people arepresent.
In order to evaluate the above mentioned features of the proposed system, the following additional experiments were con-341

Hand gestures.

Fig. 17

Handshaking experiment (left: handshaking with slave hand;right: operator with master hand).

ducted.
(1) Nonverbal communication using hand gestures and handshakes was carried out. Several hand gestures such as the pointing gesture, iconic gestures, and emblematic gestures such as
pointing, thumbs up, ok, etc., were easily made using the developed master-slave system. Figure 16 shows some examplesof the realization of such gestures.
A handshake action was performed, and the positions andforces of the slave arm and the master arm were measured. Figure 17 shows an example of a handshake with a surrogate robot.
Figures 18 and 19 show examples of the measured position
and force in the upward and downward directions respectively,
during a handshake. As indicated by this figure, motion is initiated by the slave because the action of a handshake with asurrogate robot requires initiative on the part of a person shaking hands with the slave. The graph shows that the position of
the master follows the position of the slave without any delay
and the force of the master is delayed compared to the force of342

SICE JCMSI, Vol. 1, No. 5, September 2008

Fig. 22

Fig. 18

Normalized positions of slave and master during a handshake.

Fig. 19

Normalized forces of slave and master during a handshake.

Reflectance of retroreflective material as a function of the angle
between projection and observation.

incident angle of more than 60 degrees, which is suﬃcient fora human-sized robot.
Figure 21 shows an example of a projected image observed
from several positions around the robot. Clear images of theperson at the control are obtained from any angle because of
the reflectance characteristics shown in Fig. ??.
In order to estimate the maximum number of people who
could view the person’s image simultaneously, the following
measurement was performed. White uniform image was projected onto the retroreflective surface of the robot, and reflectance was measured as a function of the angle between the
projector and an observer, as shown on the left-hand side ofFig. ??. The result is shown on the right-hand side of Fig. 22.
This indicates that virtually the entire reflected light is within
an angle of 3 degrees. This implies that when we separate two
observation points at an angle of 3 degrees, no interference occurs. If we simply divide the view of 360 degrees by 3 degrees,
120 simultaneous viewing points are obtained.

8.

Fig. 20

Reflectance of Retroreflective Material as a Function of Incident
Angle of Projection.

the slave by around 0.2s. At the beginning of the handshakeaction, antagonistic force is produced on the master side and nomovement occurs for approximately 3s; then, the person on themaster side follows the motion of the person on the slave side.
These movements have been clearly recorded in Fig.17.
(2) Figure 20 shows the characteristics of the retroreflective
material used to cover the surface of the robot. A reflectanceof 100% with reference to normal incidence is obtained for an

Fig. 21

Conclusion

The concept of using a robotic mutual telexistence system fornatural face-to-face communication was proposed, and its feasibility was demonstrated by constructing a mutual telexistence
master-slave system using the RPT.
In order to carry out face-to-face communication between
two people who are located in diﬀerent places, person “A” must
be able to see another person “B” face to face, and vice versa.
We proposed a system in which person “A” used a surrogate
robot, which was located at the place where person “B” waspresent, and the surrogate robot was covered with retroreflective material so that a real-time image of “A” could be projected
onto it. It was demonstrated that not only was “A” able to see“B” face to face but also “B” was able to see “A” face to face

Projected images observed from various angles.

SICE JCMSI, Vol. 1, No. 5, September 2008by using the proposed RTP-based mutual telexistence method.
It was also proved that nonverbal communication actions
such as gestures and handshakes could be performed in addition to conventional verbal communication because of the useof a master-slave manipulation robot as the surrogate of a human.
It was also shown that person “A,” who visited a place wherea surrogate robot was located, could be seen naturally and simultaneously by several people standing around the surrogate
robot.
References[1] I.E.Sutherland: A Head-Mounted Three Dimensional Display,
Proceedings of the Fall Joint Computer Conference, pp.757764, 1968.
[2] H.A.Ballinger: Machines with Arms, Science Journal, October, pp.58-65, 1968.
[3] J.Charles and J.Vertut: Cable Controlled Deep Submergence Teleoperator System, Mechanism and Machine Theory,
pp.481-492, 1977.
[4] M.Minsky: Telepresence, Omni, Vol.2, No.9, pp.44-52, 1980.
[5] S.Tachi and M.Abe: Study on Tele-Existence (I), Proceedingsof the 21st Annual Conference of the Society of Instrument andControl Engineers (SICE), pp.167-168, 1982 (in Japanese).
[6] S.Tachi, K.Tanie, and K.Komoriya: Evaluation Apparatus of
Mobility Aids for the Blind, Japanese Patent 1462696, filed
on December 26, 1980; An Operation Method of Manipulators with Functions of Sensory Information Display, Japanese
Patent 1458263, filed on January 11, 1981.
[7] R.L.Pepper, R.E.Cole, and E.H.Spain: The Influence of Camera and Head Movement on Perceptual Performance under Direct and TV-Displayed Conditions, Proceedings of the SID,
vol.24-1, pp.73-80, 1983.
[8] S.Tachi, K.Tanie, K.Komoriya, and M.Kaneko: Tele-Existence(I): Design and Evaluation of a Visual Display with Sensationof Presence, Proceedings of the 5th Symposium on Theory andPractice of Robots and Manipulators (RoManSy ’84), pp.245254, Udine, Italy, (Published by Kogan Page, London), June
1984.
[9] J.D.Hightower, E.H.Spain, and R.W.Bowles: Telepresence: A
Hybrid Approach to High Performance Robots, Proceedingsof the International Conference on Advanced Robotics (ICAR’87), pp.563-573, Versailles, France, October 1987.
[10] L.Stark et al.: Telerobotics: Display, Control and Communication Problems, IEEE Journal of Robotics and Automation,
Vol.RA-3-1, pp.67-75, February 1987.
[11] M.S.Shimamoto:TeleOperator/TelePresence System (TOPS)
Concept Verification Model (CVM) Development, N.K. Saxena, ed., Recent Advances in Marine Science and Technology,’92, pp.97-104, 1992.
[12] S.Tachi, H.Arai, I.Morimoto, and G.Seet: Feasibility Experiments on a Mobile Tele-Existence System, The International
Symposium and Exposition on Robots (19th ISIR), Sydney,
Australia, November 1988.
[13] S.Tachi, H.Arai, and T.Maeda: Development of an Anthropomorphic Tele-Existence Slave Robot, Proceedings of the International Conference on Advanced Mechatronics (ICAM),
pp.385-390, Tokyo, Japan, May 1989.
[14] S.Tachi, H.Arai, and T.Maeda: Tele-Existence Master Slave
System for Remote Manipulation, IEEE International Workshop on Intelligent Robots and Systems (IROS’90), pp.343-348,
1990.
[15] S.Tachi and K.Yasuda: Evaluation Experiments of a TeleExistence Manipulation System, Presence, Vol.3, No.1, pp.3544, 1994.
[16] Y.Yanagida and S.Tachi: Virtual Reality System with Coherent

343

Kinesthetic and Visual Sensation of Presence, Proceedings ofthe 1993 JSME International Conference on Advanced Mechatronics (ICAM), pp.98-103, Tokyo, Japan, August 1993.
[17] K.Oyama, N.Tsunemoto, S.Tachi, and T.Inoue: Experimental
Study on Remote Manipulation Using Virtual Reality, Presence, Vol.2, No.2, pp.112-124, 1993.
[18] S.Tachi: Real-Time Remote Robotics- Toward Networked
Telexistence, IEEE Computer Graphics and Applications,Vol.18, pp.6-9, 1998.
[19] S.Tachi, K.Komoriya, K.Sawada, T.Nishiyama, T.Itoko,
M.Kobayashi, and K.Inoue: Telexistence Cockpit for Humanoid Robot Control, Advanced Robotics, Vol.17, No.3,
pp.199-217, 2003.
[20] S.Tachi: Augmented Telexistence, Mixed Reality, pp.251-260,
Published by Springer-Verlag, 1999.
[21] S.Tachi: Toward Next Generation Telexistence, Proceedingsof IMEKO-XV World Congress, Vol.X (TC-17 & ISMCR’99),
pp.173-178 , Tokyo/Osaka, Japan, June 1999.
[22] S.Tachi, K.Komoriya, K.Sawada, T.Nishiyama, T.Itoko,
M.Kobayashi, and K.Inoue: Development of Telexistence
Cockpit for Humanoid Robot Control, Proceedings of 32nd International Symposium on Robotics (ISR2001), pp.1483-1488,
Seoul, Korea, April 2001.
[23] S.Tachi: Toward the Telexistence Next Generation, Proceedings of the 11th International Conference on Artificial Reality and Tele-Existence, (ICAT2001), pp.1-8, Tokyo, Japan, December 2001.
[24] S.Tachi: Telexistence and Retro-Reflective Projection Technology (RPT), Proceedings of the 5th Virtual Reality International
Conference (IVRIC2003), pp.69/1-69/9, Laval Virtual, France,
2003.
[25] S.Tachi, N.Kawakami, M.Inami, and Y.Zaitsu: Mutual Telexistence System using Retro-Reflective Projection Technology,
International Journal of Humanoid Robotics, Vol.1, No.1, pp.
45-64, 2004.
[26] T.Nishiyama, H.Hoshino, K.Sawada, A.Baba, T.Sekine,
W.Yamada, A.Terasawa, RNakajima, Y.Tokunaga, andM.Yoneda: Communication Agent Embedded in Humanoid
Robot, Proceedings of SICE Annual Conference 2003
(SICE2003), pp.1342-1347, 2003.
[27] R.Tadakuma, Y.Asahara, H.Kajimoto, N.Kawakami, andS.Tachi: Development of Anthropomorphic Multi-D.O.F.
Master-Slave Arm for Mutual Telexistence, IEEE Transactions on Visualization and Computer Graphics, Vol.11, No.6,
pp.626-636, 2005.
[28] S.Nakagawara, R.Tadakuma, H.Kajimoto, N.Kawakami, andS.Tachi: A Method to Solve Inverse Kinematics of Redundant
Slave Arm in the Master-Slave System with Diﬀerent Degrees
of Freedom, Proceedings of the 2004 International Symposium
on Measurement, Control, and Robots, NASA Johnson Space
Center, Houston, Texas,USA, 2004.
[29] S.Nakagawara, I.Kawabuchi, H.Kajimoto, N.Kawakami, andS.Tachi: An Encounter-Type Multi-Fingered Master Hand Using Circuitous Joints, Proceedings ICRA, 2005.
[30] S.Tachi, T.Maeda, R. Hirata, and H. Hoshino: A Construction
Method of Virtual Haptic Space, Proceedings of the 4th International Conference on Artificail Reality and Tele-Existence(ICAT’94), pp.131-138, Tokyo, Japan, July 1994.
[31] N.Kawakami, M.Inami, T.Maeda, and S.Tachi: Media
X’tal -Projecting Virtual Environments on Ubiquitous ObjectOriented Retro-Reflective Screens in the Real Environment-,
SIGGRAPH’98, Orlando, Florida, USA,July 1998.
[32] M.Inami, N.Kawakami, D.Sekiguchi, Y.Yanagida, T.Maeda,
and S.Tachi: Visuo-Haptic Display Using Head-Mounted Projector, Proceedings of IEEE Virtual Reality 2000, pp.233-240,
New Brunswick, New Jersey, USA, March 2000.
[33] S.Tachi: Two Ways of Mutual Communication: TELESAR and344

SICE JCMSI, Vol. 1, No. 5, September 2008

TWISTER, in S. Tachi, ed., Telecommunication, Teleimmersion and Telexistence, IOS Press, ISBN 1-58603-338-7, pp.324, 2003.
[34] S.Tachi: Telexistence: Next-Generation Networked Robotics,
S. Tachi,ed.,Telecommunication, Teleimmersion and Telexistence II, IOS Press, ISBN 1-58603-519-3, pp. 3-38,2005.
[35] N.Kawakami, D.Sekiguchi, H.Kajimoto, and S.Tachi: TelesarPHONE - Communication Robot Based on Next Generation
Telexistence Technologies -, Proceedings of 36th International
Symposium on Robotics (ISR2005), Tokyo, Japan, 2005.

Susumu TACHI (Member, Fellow)
He received his B.E., M.S., and Ph.D. degrees in mathematical engineering and information physics from the
University of Tokyo in 1968, 1970, and 1973, respectively. He joined the Faculty of Engineering of the University of Tokyo in 1973, and in 1975, he moved to the
Mechanical Engineering Laboratory, Ministry of International Trade and Industry, where he served as the directorof the Biorobotics Division. In 1989, he rejoined the University of Tokyo,
and is currently Professor at the Department of Information Physics andComputing. From 1979 to 1980, Dr. Tachi was a Japanese Government
Award Senior Visiting Scientist at the Massachusetts Institute of Technology, Cambridge, USA. His present research covers robotics, virtual reality and augmented reality with special focus on telexistence (TELESAR
and TWISTER), Real-time Remote Robotics (R-Cubed) and Retroreflective Projection Technology (RPT), including optical camouflage. Prof.
Tachi is a founding director and a fellow of the Robotics Society of Japan(RSJ), the 46th president and a fellow of the Society of Instrument andControl Engineers (SICE), a fellow of the Japan Society of Mechanical Engineers (JSME), and is the founding president of the Virtual Reality Society of Japan (VRSJ). From 1988 he serves as Chairman of the
IMEKO (International Measurement Confederation) Technical Committee
17 on Measurement in Robotics. He initiated and founded ICAT (International Conference on Artificial Reality and Telexistence) in 1991 and IVRC
(International-collegiate Virtual Reality Contest) in 1993. He is a memberof IEEE VR Steering Committee and served as General Chair of IEEE VR
2001 and 2002.

Naoki KAWAKAMIHe received the PhD degree in advanced interdisciplinary studies from the University of Tokyo. He is a lecturer in the Department of Information Physics and Computing at the University of Tokyo. His research interest isvirtual reality.

Hideaki NII (Member)He received the PhD degree from the University ofElectro-communications. He is a research associate in
the Department of Information Physics and Computing
at the University of Tokyo. He is a member of the Society of Instrument and Control Engineers (SICE). His
research interests include vision, and robotics.

Kouichi WATANABEHe received the B.E. degree in Information and Image
Sciences from the Chiba University, Japan, in 2004, andthe M.E. degree in Information Physics and Computing
from the University of Tokyo, Japan, in 2006. Currently,
he is a candidate for the Ph.D. of Information Physics
and Computing at the University of Tokyo. His researchinterests include virtual reality and robotics.

Kouta MINAMIZAWA
He received his M.E. degree in information science andtechnology from the University of Tokyo in 2007. He isa Ph.D. student at the University of Tokyo. He has been
receiving a JSPS research fellowship for young scientists
from 2007. His research interests include haptic interface, human-computer interaction, and virtual reality. He
is a student member of IEEE, VRSJ, JSME, and IPSJ.