Transmission of Existence by Retro Reflective Projection Technology
Using Handheld ProjectorKeisuke Takeshita∗

Kouichi Watanabe†

Takumi Yoshida‡

Kouta Minamizawa§

Susumu Tachi¶

The University of Tokyo

Keio UniversityThe University of Tokyo

Keio University

Keio UniversityA BSTRACT
Our research aims to enable persons to communicate remotely withnumerous participants while moving around freely in public spaces.
To achieve this goal, we developed a novel remote communication
system based on a mutual telexistence master-slave system. In thispaper, we propose a system that transmits the operator’s sense ofexistence to the participant using retro-reflective projection technology and a handheld projector, which solves the problem of alimited sense of the surroundings. Although stereoscopic presentation by motion parallax did not occur in this study, this system wasexpected to give the participant a stereoscopic view via the surrogate robot using a handheld projector.
Index Terms: H.5.1 [Information Interfaces and Presentation]:
Multimedia Information Systems—Artificial, augmented, and virtual realities
1I NTRODUCTION

In remote communication, mutual telexistence[1] is the concept inwhich the sense of existence of the self is transmitted to a remoteenvironment, providing the sense that the self is actually in the remote environment. Our aim is to enable remote persons to communicate with numerous participants while moving around freely inpublic spaces. Figure 1 shows the primary concept of our research.

Figure 1: The concept of our research.

In the proposed remote communication system, it is necessary to(1) transmit the presence of the remote environment to the operator,(2) transmit the sense of existence of the participant in the remoteenvironment to the operator, and (3) transmit the sense of existenceof the operator to the participant. Previous research has addressed
the first two points using a surrogate robot[2]. In this paper, we∗ e-mail: keisuke takeshita@ipc.i.u-tokyo.ac.jp† e-mail: kouichi@tachilab.org‡ e-mail: takumi@tachilab.org§ e-mail: kouta@tachilab.org¶ e-mail: tachi@tachilab.org

IEEE International Symposium on Virtual Reality Innovation 201119-20 March, Singapore
978-1-4577-0037-8/11/$26.00 ©2011 IEEEfocus on transmitting the sense of existence of the operator to theparticipant. In order to achieve this, the images of the operator haveto be three-dimensional to sufficiently represent the sense of existence to the participant. From a psychological perspective, humanemotions such as enjoyment are transmitted through both the eyes
and facial expressions[3]. Similarly, from physiology, we know thatthe forehead cortex and temporal pole are activated when others arewithin the line of sight in order to estimate their intentions[4]. This
information highlights the requirement to transmit visuals of theoperator’s face, including their eyes and facial expression.
There have been several studies focused on transmitting thesense of existence of the operator to the participant. Anybots
QB[5]and Texai[6] have transmitted the identity and expression of
the operator by displaying their face on a display mounted on a surrogate robot. However, the display was not visible when the participant viewed the surrogate robot from the rear. In the transmissionof an ideal sense of existence, there should be a post-figure of theoperator. Furthermore, in the previous studies, everyone sees thesame image when they look at the surrogate robot from different
directions. Given that the surrogate robot ideally would accurately
represent the operator, the appearance of the face must change according to the direction from which the surrogate robot is being
viewed. Therefore, the accurate transmission of sense of existence
is not possible using this previously studied method. Ishiguro et al.
developed the robot Geminoid[7], which closely resembles a human being. They state that Geminoid transmits more sense of existence than a video conference. However, in remote communication
systems that use a humanoid robot, the robot must closely resemble
the operator so that the participant knows who the operator is. As
a result, we are not aiming toward a system of this type because it
is cost-prohibitive to prepare surrogate robots for each person using
the system. Fuchs et al. developed Shader Lamps Avatars[8], whichtransmit the sense of existence of the operator through projectionsof the operator’s facial image on a model of a human head. Using this method, the three-dimensionality of the face is transmitted,
which allows participants to view the operator from any direction.
However, short-range communication, such as a handshake, is difficult, because it is impossible for there to be anything between theprojector and the head model. In TELESAR2[1], the sense of existence is conveyed using retro-reflective projection technology[9].
Using this method, the image is different at every angle, and therefore, contradictory occlusion does not occur. However, the viewpoint is limited to the position where the projector was placed. The
presentation of a three-dimensional operator is not possible as theprojecting image of the operator is a two-dimensional image.
In this study, we focus on retro-reflective projection technology that enables short-range communication, like a handshake. InTELESAR2, the fact that the viewpoint was limited to the position
where the projector was placed disturbed the transmission of thesense of existence. This problem is solved by using small projectors, which have recently become available. The small projector
can be used in two ways: handheld and head-mounted. For thisstudy, we opted for the easily usable handheld type. It is known thatthree-dimensionality occurs through motion parallax[10]. The reproduction of motion parallax seems to impart a stereoscopic viewto the participant who sees the surrogate robot via the handheld pro-71

jector. In this paper, we propose a system that transmits the senseof existence of the operator to the participant using retro-reflective
projection technology and a handheld projector.
2PROPOSED METHOD

Our aim is to enable people to communicate with remote participants while moving around freely in public spaces. To achieve thispurpose, we focused on a master-slave system using a surrogate
robot. In the proposed remote communication system, it is necessary to transmit the following:
1. The presence of the remote environment to the operator.
2. The sense of existence of the participant in the remote environment to the operator.
3. The sense of existence of the operator to the participant.
To achieve these three requirements, we suggest a system that transmits the sense of existence of the operator to the participant. The
requirements of the system for transmitting the sense of existence
are thus examined.
To transmit the sense of existence of the operator to the participant, it is important to transmit a three-dimensional image of theoperator, particularly their facial expression and eyes. During communication, the facial expression must also be visible to multiple
participants, because it is assumed that there will be numerous participants surrounding the surrogate robot. Similarly, it should beusable at the close distance required for physical contact. In addition, it should be possible to use the system regardless of who theoperator is. As a result, we focused on retro-reflective projection
technology, which can satisfy all of these requirements. In TELESAR2, it became an issue that the viewpoint was limited to the position where the projector was placed, as this disturbs the transmission of sense of existence. This problem is solved in our system
by using a handheld projector. Additionally, the reproduction ofmotion parallax should give a stereoscopic view to the participants,
who view the surrogate robot via a handheld projector.
We will now describe the projection of the operator to the surrogate robot using retro-reflective projection technology in order tosatisfy the further requirements of the system. In retro-reflectiveprojection technology, a retro-reflector is used as the projection
plane. A retro-reflector strongly reflects the incident light in thedirection of the light source. Using this, the image is presenteddifferently when viewed from different angles to account for multiple participants. For the projection of the operator to the surrogate
robot using retro-reflective projection technology, it is necessary toaffix a retro-reflector as a projection plane to the head of the surrogate robot and to project the image of the operator captured by thecamera from the projector near the eye.
In order to project the face of the operator on the projection
plane, it is necessary to affix a projection plane to the head of thesurrogate robot and that projection plane must be of a sufficient size.
In addition, the projector, which would be held in the left hand to
free the right hand for communication, must be of a sufficiently
small size and weight for handheld portability. For the communication with the operator, the handheld projector must not cover
either the eyes or the face of the person who holds it. However, it isacceptable for one eye to be covered, because half mirrors are necessary in front of the eye. For the reproduction of motion parallax,it is necessary that the camera captures whether the operator moves
the position of the projector of the surrogate robot. Accordingly,
the camera must have a movable mechanism. To assess the direction of movement, the measurement of the relative position of theprojector and the retro-reflective projection plane is required. On
the other hand, the operator needs an immersive display to transmit
the sense of existence to the participant. However, the display must72

not cover the face of the operator, in order to capture the image ofthe operator’s face. Furthermore, the camera capturing the imageof the operator must not hinder the display.
In an ideal system that satisfies the requirements described, theparticipant is transmitted the identity of the operator and their facialexpression. Furthermore, the participant should not sense that thereis the projected image of the operator on a projection plane but that
there is the operator.
3 SYSTEM CONFIGURATION
Figure 2 shows the configuration of the system that presents thesense of existence of the operator. The relative position of the projector and the retro-reflective projection plane affixed to the surrogate robot is measured using a motion capture system, i.e., OptiTrack (Natural Point Inc). By placing a marker in the projector andprojection plane, it is possible to acquire each position at a frame
rate of 100 fps within an error of 0.2 mm.

Figure 2: System configuration.

We adopted an immersive full-color autostereoscopic display,”TWISTER”[2], as the display that the operator uses and six movable camera units (Figure 3) installed with TWISTER for capturing
the image of the operator. By using TWISTER, movable camera
units can capture an image outside of the display. Movable camera
units are moved on a circular rail with a 1.2-m radius that centers on
the operator. The height is equal to the viewpoint of the operator forthe circular rail. A movable camera unit, Firefly MV (Point GreyResearch Inc.), is mounted and pointed towards the operator. This
movable camera unit follows the position of the projector for thesurrogate robot, and captures the image of the operator for projection. The position of the movable camera units was controlled using
PID control. Though the Firefly MV has a maximum resolution of752x480 pixels, a resolution of only 480x350 pixels was used forthe image of the operator, with a central focus on their eye. The
image of the operator is preserved in the computer that controls themovable camera unit at 30 fps and it is transmitted to the projector
via the network.

Figure 3: Movable camera unit.

The handheld projector is composed of a projector, outer pack-aging, half mirror, and marker (Figure 4). The projector is a
SHOWWX (Microvision Inc., refresh rate: 60 Hz, resolution:
640x480 pixels, vertical angles of view: 25 degree) laser projector. The outer packaging was designed according to the requirement that coverage of the operator’s face is minimized. As a result,the projector was located near the face. The marker was placedfor OptiTrack. The dimensions of the handheld projector were 60mm x 150 mm x 175 mm, and the weight was 220 g, including theprojector and its battery.
The projection image was made using OpenGL and OpenCV. After the captured image of the operator is loaded, it is stuck as a texture on the plane calculated from the projector and surrogate robot
position/posture that was obtained from OptiTrack. This plane wasalways vertical for the optic axis direction of the projector, and thesize was decided from the angle of field of the camera that captures
the image of the operator. This image is converted considering theconfiguration of the projector and half mirror. The refresh rate of
the projected image is 60 Hz, which is equal to the refresh rate of
the projector.

Figure 6: Result of projection of the operator. upper left: operator image from the outside TWISTER, upper right: location of participants
and surrogate robot, lower-left: image of the operator from participant 1, lower-right: image of the operator from participant 2

Figure 4: left: Handheld projector unit, right: Image of using projector
unit.

Figure 5 shows the retro-reflective projection plane. It was madealong an omnidirectional camera device. The projection plane is ofa sufficient size to project the face of the operator in its real scale.
The height of the projection plane is set at the height at which eye
contact between the operator and the participant was possible. The
marker for OptiTrack placed in the upper part. So that the camera
is not blocked, a hole of 9-mm diameter was made in front of lens.

Figure 5: Retro reflective projection plane.

4RESULT AND DISCUSSION

Figure 6 shows a demonstration image of the proposed system. By
using the constructed system, it was confirmed that the expressionof the operator was transmitted and that it was possible to observe
the operator from any viewpoint. Furthermore, this system achieved
the previously described requirement that it must be able to be used
at close proximity and regardless of the identity of the operator.
However, a person who tested the system reported that they only
sensed through the projection. In other words, stereoscopic presen-tation by motion parallax did not occur. As a result, the sense ofexistence of the operator was not transmitted to the participant.
This cause considers two. The first is difference between the location of movable camera unit relative to the operator and the location of hand-held projector relative to surrogate robot. The movable
camera units follow the position of the projector for the surrogate
robot, and capture the image of the operator for projection. The delay of following, however, is not avoided. To solve the problem, it
is necessary to convert the image which compensates for differenceof the relative position assuming the delay of tracking. The second
is that projected image can not follow the movement of tremblinghand. In the ideal system, projected image is recalculated in a moment when the projector move. Because actually the refresh rate
is limited, the projected figure of the operator is blurred until projected image is recalculated. Use of the projector which refresh
rate is high is one solution. It is also considered that the projector
is fixed to the head because the movement of the head is less thanthe hand.
This could be caused by two different factors. The first possibility is the difference between the location of the movable camera
unit relative to the operator and the location of the handheld projector relative to the surrogate robot. The movable camera units follow
the position of the projector for the surrogate robot, and capture theimage of the operator for projection. The delay caused by tracking,however, cannot be avoided. To solve this problem, it is necessary
to convert the image, which compensates for the difference of therelative positions assuming the tracking delay. The second possible cause is that the projected image cannot follow the movementof a trembling hand. In an ideal system, the projected image is recalculated the moment the projector moves. However, because therefresh rate is limited, the projected figure of the operator is blurred
until the projected image is recalculated; therefore, the use of a projector with a higher refresh rate is one potential solution. It is also
possible that using a head-mounted projector would contribute to
fixing this problem because the head moves less than hands do.

73With regard to the projector, it is thought that the handheld projector, used to project the image of operator adds stress to the participant because the participant has to hold the handheld projector andgaze at the surrogate robot for communication. In order to solve thisproblem and increase the stability of the projector, we will develop
a head-mounted projector.
5 CONCLUSION
This research aims to enable people to communicate remotely withnumerous participants while moving around freely in public spaces.
In this paper, we propose a remote communication system thattransmits the sense of existence of a remote person to the participant
using retro-reflective projection technology and a handheld projector. It was confirmed that it was possible to observe the figure of theremote person from any viewpoint. However, stereoscopic presentation by motion parallax could not be carried out. An investigation
into a system to solve this problem is currently underway.
ACKNOWLEDGEMENTS
This work was partly supported by SCOPE: Strategic Information
and Communications R&D Promotion Programme.
R EFERENCES[1] Susumu Tachi, Naoki Kawakami, Hideaki Nii, Kouichi Watanabe andKouta Minamizawa. Telesarphone: Mutual telexistence master-slavecommunication system based on retroreflective projection technology.
SICE Journal of Control, Measurement, and System Integration, Vol.1,No.5, pp.335-344, 2008.
[2] Susumu Tachi. TWISTER: Immersive Ominidirectional Autostereoscopic 3D Booth for Mutual Telexistence. ASIAGRAPH TOKYO 2007
PROCEEDINGS, pp1-6, 2007.
[3] Mark Cook and Jacqueline M.C. Smith. The Role of Gaze in Impression Formation. British Journal of Social Clinical Psychology, vol.14,
pp.19-25, 1975.
[4] Knut KW Kampe, Chris D. Frith and Uta Frith. ”Hey John”: Signals
Conveying Communicative Intention toward the Self Activate Brain
Regions Associated with ”Mentalizing,” Regardless of Modality. The
Journal of Neuroscience, Vol.23, No.5, pp.5258-5263, 2003.
[5] Anybots, Inc. http://anybots.com.
[6] Willow Garage, Inc. http://www.willowgarage.com.
[7] Shuichi Nishio, Hiroshi Ishiguro and Norihiro Hagita. Geminoid: Teleoperated Android of an Existing Person. in Humanoid Robots: NewDevelopments, de Pina Filho, A.C. eds., I-Tech Education and Publishing, Vienna, Austria, 2007.
[8] Peter Lincoln, Greg Welch, Andrew Nashel, Adrian Ilie, Andrei Stateand Henry Fuchs. Animatronic Shader Lamps Avatars. 8th IEEE International Symposium on Mixed and Augmented Reality, pp.27-33,
2009.
[9] Susumu Tachi. Augmented Telexistence. Mixed Reality-Merging Realand Virtual Worlds, pp.251-260, 1999.
[10] Daisuke Kondo, Yuichi Shiwaku and Ryugo Kijima. Free Form Projection Display and Application. Proceeding of the 5th International
Workshop on Projector Camera Systems, pp.31-32, 2008.

74