ASIAGRAPH_TOKYO 2007 PROCEEDINGSTWISTER：
相互テレイグジスタンス可能な没入型全周囲裸眼 3 次元ブース
TWISTER:
Immersive Omnidirectional Autostereoscopic 3D Booth for Mutual Telexistence

舘 暲 /東京大学
Susumu TACHI /The University of Tokyotachi@star.t.u-tokyo.ac.jp
TWISTER (Telexistence Wide-angle Immersive STEReoscope) は、遠く離れた人が互いに同じ三次元空間に集う相互テレイグジ
スタンスと呼ばれるフェース・ツー・フェース・テレコミュニケションの概念を実現するために設計された没入型の全周囲フルカラーの裸眼 3D ブースである。各々の円筒形のブースに入ったユーザーは、360 度のフルカラーの全周囲の映像を、特殊な眼鏡などを装着しなくとも立体的に観察でき、かつそのユーザーの映像が任意の方向から立体的に撮像される。従って、複数のブースを用いれば、離れたところにいる人たちがすぐ近くにいるような感覚で集うことができる。この概念は、1996 年に提唱され、この概念に基づいて、現在までに TWISTER I、II、III、IV、V のプロトタイプが試作されている。TWISTER IV では、左眼用と右眼用の LED アレイとパララクスバリアからなり 1.7rps という高速で回転する 36 個の提示ユニットを用いて両眼ステレオ映像を提示する。また同時に、回転シリンダに組み込まれたカメラが、ブースのなかにいるユーザーの三次元像を取得する。このようにして 2 台の TWISTER 内のユーザーは、互いのユーザーの 3D 映像を実時間で見ることができる。本論文では、相互テレイグジスタンスのための TWISTER の概念と原理を説明し、現在までに開発されたプロトタイプシステムについて述
べ、TWISTER の様々な応用を概観する。
TWISTER (Telexistence Wide-angle Immersive STEReoscope) is an immersive omnidirectional full-color autostereoscopic 3D booth,designed for a face-to-face telecommunication concept called "mutual telexistence," where people at distant locations can communicate as if
they were present in the same three-dimensional space. Each human user resides in a cylindrical booth, which displays live full-color 360
degree panoramic and autostereoscopic images to the user without the need for any eyewear; at the same time, the booth captures images of
the user from every angle.
face.

With multiple booths, people at remote locations can interact with each other as if they were interacting face-to-This concept was proposed in 1996, and prototype models TWISTER I, II, III, IV, and V, were constructed based on the proposedconcept.

TWISTER IV displays binocular stereo images using 36 display units, each consisting of two LED arrays for the left and righteyes and a parallax barrier that rotates at a high speed (1.7 rps).

This system simultaneously captures 3D images of the observer byemploying cameras installed on the rotating cylinder. Thus, the observers inside two TWISTERs can view 3D images of each other in realtime.

In this paper, the concept and principles of TWISTER for mutual telexistence is explained, and the prototype systems developed andpossible applications of TWISTER are described.

1. Introduction
Face-to-face communication is an essential and inevitable component of aperson's daily life. However, in situations where people are far apart, aface-to-face telecommunication system proves useful. One of the mostessential elements of such a system is the sense of presence. Immersion
being one of the essential factors for the sense of presence, it is desirable
that the visual display of such a system provide an immersive stereoscopic
view. Two typical conventional immersive systems include an HMD(head mounted display) and CAVE (CAVE automatic virtual environment).
In such conventional systems, however, the mechanisms for binocularstereopsis, such as headgears for HMD and shutter-glasses for CAVE, notFig. 1. Face-to Face Telecommunication using TWISTER

only leave the observer with a sense of discomfort but also hinder face-to-that can be observed without the use of any special eyewear. Moreover,face communication since they obscure an observer's face, particularly

we can place cameras between the display units. By rotating the cameras

around the eyes.

simultaneously, the booth can display stereoscopic images and capture thefigure of the user simultaneously. It is also possible to place cameras

Autostereoscopic 3D displays are effective for face-to face communication

outside the booth and view the user from outside. Once these images areas they enable the user to view three-dimensional images without the use ofobtained from various viewpoints around the user, a new view from anany special headgear or eyeglasses. The most common autostereoscopic

arbitrary viewpoint can be constructed using an image-based rendering

3D displays employ lenticular lenses or parallax barriers.

technique.

For suchdisplays, if a viewer positions his/her head in a certain manner, he or she
will perceive different images with each eye, which gives rise to a stereo

With several booths, every user in a remote place can view the three-image. However, when lenticular lenses or parallax barriers were used, adimensional figures of other users working in real time in the mutual virtual

system that provides a 360-degree panoramic field of view could not beenvironment. In such a situation, the booth works as the medium between

realized. Integral imaging using fly's eye lenses is a promising technology.

the real space and the virtual communication space.

Nevertheless, it has not been employed in any 360-degree integral imaging
autostereoscopic 3D display.

In this paper, the concept, principles, developed prototypes, and possible
applications of TWISTER are described.

The conventional parallax barriers, lenticular lenses, or fly's eye lenses haveanother limitation: it is considerably difficult to take a picture of a user's face
while he or she is observing 3D images. This limitation stems from thefact that the screen is too densely packed with micro lenses and LCDs tolocate cameras in between. Since recording a user's face in real time at theeye level is essential for face-to-face telecommunication, the cameras
should be placed at the level of a human user's eyes. However, such a setup leads to sparse arrangement of micro lenses, which adversely affects theperformance of the display.
In order to resolve these abovementioned problems, we propose
TWISTER-Telexistence Wide-angle Immersive STEReoscope- as a boothfor face-to-face telecommunication. By adopting the moving parallax

2. Concept, Principle and Development
The first idea of mutual telexistence using cylindrical booths that can beused both for a panoramic display and for a real-time camera system thattakes pictures of a human user from any direction was proposed in 1996 [1].
The constructed handmade experimental system, in which the functions of
the input device and display device were separated for the sake of simplicity,
demonstrated the feasibility of the idea [1]. Fundamental studies for therealization of the two functions-input and display- were conducted,
following which the first handmade prototype system with the display
function, dubbed TWISTER I, was constructed in 2000, and the results
were reported in 2001 [2].

barrier method, TWISTER can display panoramic stereoscopic images

Fig. 2. Concept of Mutual Telexistence using TWISTERThe CREST project on Telexistence Communication Systems led by theMovable parallax barriers constitute the key device for autostereopsis.

author and funded by JST (Japan Science and Technology Agency)

Figure 3 shows the principle of the movable parallax barriers developed.

commenced in November 2000 and ended in March 2006 [3, 4]. We

Each display unit contains two LED arrays-one for the left eye, and thehave applied the concept of TWISTER to a tangible real system that can beother for the right eye. The units rotate at a speed of approximately 1.7

experienced.

revolutions per second (rps), and the controller synchronizes the display
update to create an effective frame rate of 60 frames per second (fps).

TWISTER II was constructed on the basis of TWISTER I and reported in2001 [5].

TWISTER II successfully realized the world's firstomnidirectional autostereoscopic full color display. We demonstrated it
worldwide at SIGGRAPH in 2002 using TWISTER III [6, 7]. TWISTER
IV is a system that has both functions of display and video input [8]. It
was constructed in 2004. An almost similar system called TWISTER V
was constructed in 2005 and it is placed at the National Museum ofEmerging Science and Innovation (Miraikan).

Communication

experiments are being conducted using TWISTER IV at the University ofTokyo and TWISTER V at Miraikan.
Figure 2 shows the concept of teleexistence in a mutual virtual environment
using TWISTER. Each human user resides inside a booth that has avirtual cylindrical wall that can be used as a display of the virtualenvironment in which all users are projected as well as an input camera
device to obtain the three-dimensional moving pictures of the user inside
the booth. The booth is surrounded by (1) a circular mechanism that turns
continuously at a relatively high speed carrying display units consisting of aFig. 3. Principle of Movable (Rotating) Parallax Barrier Method

pair of LED arrays and a parallax barrier and (2) video cameras.
Since the parallax barrier obscures LED emission from the opposite side,
Video cameras, which are arranged such that they cover the user

different images are shown to the left and right eyes. At one moment, onlycompletely, take pictures of the user from virtually every angle as they turnimages for the left eye and right eye are observed in areas L and R,

around the user inside the booth. Using this method, stereo information

respectively. Both images are observed in area B, while neither image ispertaining to the user can be obtained from virtually any angle. When theobserved in area N. When the display units with barriers rotate around thesystem is used by several users, each user's line of sight and relative position

user, these areas also rotate, as shown in Fig. 4.

are measured and are used to select the appropriate picture of other users.

stereopsis is maintained as the display units rotate around the user. The

The images of other persons thus selected are appropriately arranged three-angle of view of a stereoscopic display depends on the direction anddimensionally in a mutual environment and displayed to all the users.

position of the observer. If the head of the observer is fixed in one position,

The condition ofit exceeds 140° in an ideal condition. On the other hand, if the observer
Virtual transparent boards (alpha plane) are placed in a virtual environment,always faces the center of the region of interest of the image, the angle ofand the pictures taken from two positions (left and right) are texture mappedview can be 360°.

on the boards. The other parts of the scenery are not texture mapped, andthey remain transparent. The corresponding points of the texture-mapped

With moving parallax barriers, the crosstalk between the left eye image andhuman figures on the virtual boards produce the three-dimensional figure of

the right eye image is reduced to almost zero, which gives this system athe user.

powerful advantage over other stereoscopic vision systems. Moreover, theFig. 4. Parallax Barrier rotates maintaining stereoscopic condition

barrier per se cannot be observed by the observer because it rotates at aThe stereo-video picture of a human user inside a booth is captured by 36

speed of 600 degrees per second, whereas the human eye can only track arotating cameras, and an image from an assigned angle is reproduced fromrevolution of approximately 500 degrees per second. This assures athe information captured by appropriate cameras. Figure 7 shows acontinuous 360-degree panoramic stereo view.

schematic diagram of the system.

Two modes are used in rendering for TWISTER. In Mode I, we assume
that we can track the direction in which the head of the user is facing andrender the left and right images on a display cylinder perspective
transformed from the left eye and right eye, respectively. In this mode, aprecise image is obtained when the head is fixed in one position. The
condition is kept if we render in real time according to the measured headdirection.
In Mode II, rendering with concentric mosaics is used. In this mode,
irrespective of the direction the user turns in, correct light information will
be obtained at least from the front. The light information from otherdirections is just an approximation. However, it is not necessary to trackthe direction in which the head of the human user faces.
Fig. 7. Schematic Diagram of Human Stereo-video Picture Capturing SystemFigures 5 and 6 show the typical panoramic displayed images in TWISTER.
Table 1 shows the specifications of TWISTER III and TWISTER IV.
Incidentally, the specification for TWISTER V is the same as that ofTWISTER IV. Minor modification of LED arrangements on a display
unit was conducted for TWISTER V, and TWISTER V can provide LED
display without a diffuser, which was used for TWISTER IV.

Fig. 5. 360 degree Panoramic Image displayed in TWISTER II

Table 1. Specifications of TWISTER III and TWISTER IV

Although it was proved that the proposed method can capture a humanuser's stereo-video picture from an arbitrary direction employing rotatingcameras inside a booth [8] using TWISTER IV and V, it became apparent
that the use of this method is impractical at present because it requires(1) an expensive high-speed large-capacity rotary joint and(2) precise alignment and calibration of video cameras.
Thus, we propose an alternative method of image capturing from anarbitrary direction. As clearly seen from Figures 8 and 9, one of the salient
characteristics of TWISTER is that the human user inside a booth can be
Fig. 6. 360 degree Panoramic Image displayed in TWISTER IV

seen from outside of the booth.

placed in the virtual environment according to the relative position andorientation. Thus, the users have the feeling as if they were near at hand in
the virtual environment.

Fig. 8. Human user observed from outside of TWISTER II booth

Fig. 10. Method of Capturing Human Stereo-video Picture from outside

Mutual telexistence is one of the most important technologies for therealization of networked telexistence, because users "telexisting" in a robot
must know whom they are working with over the network. Figure 12shows how TWISTER is used as a cockpit for mutual telexistence.

Fig. 9. Human user observed from outside of TWISTER IV boothThis implies that the human user's picture can be taken from any direction
from outside of the booth. Figure 10 shows a schematic diagram of the

Human user A can observe remote environment B using omnistereo
camera [9] on a surrogate robot A'. This provides A the panoramic stereo
view of the remote environment displayed on TWISTER booth. A' iscontrolled by A by using the telexistence master-slave control method [3, 4].

proposed method. A concentric rail is arranged outside of the booth, on
which a movable stereo camera is set.

The camera is controlledaccording to the angle from which the human picture should be taken.
Stereo cameras can be used when several people observe the user inside.
When it is necessary to cross two cameras, they can virtually be crossedby changing the roll of both cameras.
Consider the situation shown in Figure 11. Each of the three humanusers is inside a TWISTER booth. Each booth has two stereo cameras.
Although the human users are far apart, they can virtually meet together
in a virtual environment using the TWISTER booths.
First, 360° 3D virtual environment is displayed on each of the booths.
Next, the two stereo cameras of each booth capture the pictures of theuser inside TWISTER from the angles calculated based on the relative
position and orientation of the three users in the virtual environment.
Then, the captured images are transmitted using communication links to theother TWISTERs. At each booth, the human figures of other users areFig.12. Face-to face Communication using TWISTERs

Cameras B' and C' mounted on the booth are controlled by the position and4. Acknowledgement

orientation of Humans B and C relative to A', respectively.

B and C canThis study on TWISTER has been conducted in collaboration with many ofobserve A projected on A' by using HMP (Head Mounted Projector). A'

the former and present students and staff members of Tachi Laboratory ofis covered with a retro-reflective material, which makes it possible tothe University of Tokyo, including Yutaka Kunita, Kenji Tanaka, Junya

project both images on the same robot and still both the images areHayashi, Masaaki Koyanagi, Hiroki Yokoyama, Naoko Ogawa, Atsushi

separately observed by B and C.

Sakuma, Taro Maeda, Yasuyuki Yanagida, Masahiko Inami, Naoki
Kawakami, Dairoku Sekiguchi, Ichiro Kawabuchi and Hideaki Nii.

This method of mutual telexistence using RPT (Retro-reflective Projection
Technology) was proposed and proved to be quite effective [11].

5. Reference

Figure 13 shows an example of the projection of human figure on a robot.

1) Susumu Tachi, Taro Maeda, Yasuyuki Yanagida, Masaaki Koyanagi andHiroki Yokoyama: A Method of Mutual Tele-Existence in a Virtual

TWISTER will be widely used for several applications in 3C’s and 3E’s,Environment, Proceedings of The 6th International Conference on Artificiali.e., Communication, Control, Creation, Education, Entertainment andReality and Tele-Existence, (ICAT'96), pp.9-18, Makuhari, Chiba, Japan

Elucidation.

(1996.11)
2) Yutaka Kunita, Naoko Ogawa, Atsushi Sakuma, Masahiko Inami, TaroMaeda, and Susumu Tachi: Immersive Autostereoscopic Display for
Mutual Telexistence: TWISTER I (Telexistence Widw-Angle Immersive
STEReoscope Model I), Proceedings of IEEE Virtual Reality 2001,Yokohama, Japan, pp. 31-36 (2001.3)
3) Susumu Tachi: Two Ways of Mutual Telexistence: TELESAR andTWISTER, in Telecommunication, Teleimmersion and Telexistence (S.
Tachi ed.), pp. 3-24, IOS Press, ISBN 1-58603-338-7 (2003)
4) Susumu Tachi: Telexistence: Next-Generation Networked Robotics, inTelecommunication, Teleimmersion and Telexistence II (S. Tachi ed.), pp.
3-38, IOS Press, ISBN 1-58603-519-3 (2005)
5) Kenji Tanaka, Junya Hayashi, Masahiko Inami and Susumu Tachi: TheDesign and Development of TWISTER II: Immersive Full-color
thAutostereoscopic Display, Proceedings of the ICAT2001 (11 InternationalFig. 12

Mutual Telexistence using TWISTER as a Cockpit

Conference on Artificial Reality and Telexistence), Tokyo, Japan, pp. 56-63(2001.12)
6) Kenji Tanaka, Junya Hayashi, Masahiko Inami, Yutaka Kunita, Taro Maedaand Susumu Tachi : TWISTER: A Media Booth, Emerging Technologies,
SIGGRAPH 2002, San Antonio, USA (2002.7)
http://www.siggraph.org/s2002/conference/etech/twister.html
7) Kenji Tanaka, Junya Hayashi, Masahiko Inami and Susumu Tachi:
TWISTER: An Immersive Autostereoscopic Display, Proceedings of IEEE
Virtual Reality 2004, Chicago, USA (2004.3)

Fig. 13

An example of Projection of Human Figure on TELESAR 2 robot8) Junya Hayashi, Kenji Tanaka, Masahiko Inami, Dairoku Sekiguchi,Naoki
Kawakami and Susumu Tachi: Issues in image-capture system forTWISTER, Proceedings of the ICAT2002(12th International Conference

3. Conclusion
Technology based on the concept of networked telexistence enables users
to meet and talk as if they share the same space and time, even if they are
far apart from each other. This is the goal of the development of mutual
telexistence communication systems, as a natural progression fromtelephone to telexistence-videophone. TWISTER (Telexistence Wideangle Immersive STERoscope) was developed and its efficacy wasdemonstrated and proved. It will be widely used as a telexistence boothfor communication and control with a sensation of presence.

on Artificial Reality and Tele-Existence), Tokyo, Japan, pp. 44-51 (2002.12)
9) Kenji Tanaka and Susumu Tachi: TORNADO: Omnistereo Video Imagingwith Rotating Optics, IEEE Transactions on Visualization and Computer
Graphics, Vol. 11, No. 6, pp. 614-625 (2005.11)
10) Susumu Tachi, Naoki Kawakami, Masahiko Inami and Yoshitaka Zaitsu:
Mutual Telexistence System Using Retro-reflective Projection Technology,International Journal of Humanoid Robotics, Vol. 1, No. 1, pp. 45-64(2004.3)