ICAT 2002
December 4-6, Tokyo, JAPAN

Issues in an Image Capture System for TWISTER
Junya Hayashi, Kenji Tanaka, Masahiko Inami, Dairoku Sekiguchi, Naoki
Kawakami, Susumu Tachi
Graduate School of Information Science and Technology, The University of Tokyo
7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8656, JAPAN{junya, tanaken, minami, dairoku, kawakami, tachi}@star.t.u-tokyo.ac.jp
Abstract
In this paper we report the development of TWISTER[3][8](TelexistenceWide-angle
Immersive
STEReoscope), which realize the concept of “Mutual
Telexistence”. In contrast to conventional telexistence,
where people in remote places can be appeared with arealistic sense of presence, in mutual telexistence, thefigure of the observer him/herself can be displayed toremote users and all of the users can communicate as if
they were in the same three-dimensional space.
TWISTER is a cylindrical booth designed for mutual
telexistence. It enables the user to view immersive andautostereoscopic images, and at the same time, it
captures and reconstructs the image of the user insidewith cameras positioned in-between the display unit. In
previous reports on the development of TWISTER, wehave introduced the development of display part [1]. In
this report we discuss some methods to capture imagesof the user and render them in real time.
Key words: Mutual Telexistence, Virtual Reality,
Multiple Cameras, Image Based Rendering

1. Introduction
There are a number of researches about face-to-facecommunication systems among the people in distantlocations [1]. Some of them adopted displays, which canprovide immersive stereoscopic images. These include
IPT (Immersive Projection Technology) [2][4][5] andHMD (Head Mounted Display) based systems. Thesesystems require special eyewear that hides the users’
faces and at the same time makes it difficult to shootthem. In contrast to these conventional systems, where
the images of people in remote locations can be seen
only to the user, in mutual telexistence, the figure of theobserver him/herself can also be displayed to remoteusers and all of the users can communicate as if they
were in the same three-dimensional space (Fig. 1). We
have designed TWISTER especially for this feature ofmutual telexistence. It displays three-dimensional imagesof other people to a user inside the booth, while
capturing the image of the user.
In order to realize the display of three-dimensional
images without special eyewear, “Autostereopsis” inother words, we have adopted a “Rotating Parallax

Fig 1: The Concept of TWISTER
Barrier” method [8]. The current prototype, TWISTER
III [19], can display immersive full-color stereoscopic
motion pictures at the frame rate of 30fps (frames per
second). We can also put image sensors in-between
display units, because the gaps between two display
units, consisted of only two LED arrays, are wide
enough. In the future we are going to put 30 cameras
shooting inwards on the corner of the cylinder.
Because one of the major elements for the sense ofpresence is the motion parallax, we have to design thesystem to be able to generate the images of the user
inside the booth according to the user’s eye position. We
also have to generate the image in real time using limited
number of cameras. We have already shown andexperimented the method of obtaining the images fromdesired viewpoint with multiple cameras arranged on aline. The approach is based on an IBR (Image Based
Rendering) method. By selecting necessary scan line inadvance using analog circuit, we have achieved the realtime image generation [9]In this paper, we extend this method to an image
capture system with circumferentially arranged cameras,
based on discussion and experiments, and show themethod to obtain images seen from arbitrary viewer’s
position of two degrees of freedom in one horizontal
plane.

2. BackgroundAs a prototype experiment of the image capture system
for TWISTER, Kunita has built the system with 12
cameras arranged on a linear actuator. In thisexperiment, he generated the images seen from anarbitrary virtual viewpoint in real time. He selected thenecessary light information from 12 cameras byscanning line, before digital processing by PC. Since
necessary image data is selected in early stage, the realtime rendering of images from arbitrary points has beenrealized.

90°

Fig 2. Image Capture Prototype System
With TWISTER, we have to extend this method in order
to fit the cylindrical arrangement of the cameras of
TWISTER.

3. Principle
Considering a certain closed surface, if we can obtain theinformation of all light rays from all the points on thesurface, we can reconstruct the image of inside theclosed surface seen from arbitrary viewpoints byassuming the going-straight nature of light and no
attenuation of light. We will try to reconstruct theimages of inside the TWISTER using this principle byarranging multiple cameras looking inside from thecylindrical surface.
First, we consider the case where the infinite number ofcameras is everywhere located horizontally andperpendicularly on the cylinder. We can reconstruct theimages seen from arbitrary viewpoints by using theprinciple directly. Next, considering the state where theinfinite number of cameras is horizontally located.
There, the picture seen from a virtual viewpoint, whichis in the same level surface as cameras, can bereconstructed in approximation. However, when theobject position is not known correctly, since there is an
estimate error of the depth direction, a perpendicularerror comes out on reconstructed images.
Considering TWISTER, we plan to put limited numberof cameras on the cylindrical device horizontally. As thepresent TWISTER is not equipped with the mechanism
for measuring the object’s position and form, it isnecessary to assume an object’s geometric form. And theestimate error of the depth distance to the object will
come out both horizontally and perpendicularly inreconstructed images. Therefore, it is necessary toconsider how to assume object's position and shape, andfrom which camera we should get the surface
information of the assumed object. From now on, we
will call the assumed position and shape of an object“Virtual Screen”, as it is a projection plane of eachcamera’s output. We will suggest how to decide thevirtual screen, and how to choose camera projecting on
the virtual screen.

4. Theory
4-1. Issues
Limited number of cameras is arranged horizontally at
equal intervals on the wall of the cylindrical booth, andthey take the images of an object in the booth. Now, we
will consider the method of reconstructing the picture of
the object seen from an arbitrary viewpoint, which isoutside the booth and on the same plane as the cameras.
At this time, position and shape of the object is notknown, and we have to reconstruct the desired image as
sufficient accuracy as possible. Specifically, as describedin Section 3., the problems are (1) how to set a virtual
screen, and (2) how to choose camera output for the
texture of the virtual screen.
In addition, since the object’s position is restricted in thecylinder, the area where the object can exist is limited
and we don’t need to think about background images.
This is because we plan to put the captured object on a
virtual environment, in which the background in actual
environment does not have a meaning.
4-2. How to choose camera for each light ray
When an image capture system is composed of a limited
number of cameras, it is necessary to approximate theincident light to a virtual viewpoint with the outputs ofeach camera. At this time, for each incident light to thevirtual viewpoint, it is advantageous to choose thecamera so that the angle of the incident light vector to
the virtual viewpoint and the actual light ray to thecamera can be as small as possible. The reason is that bychoosing the camera in this way, we can minimize theerror effect to the virtual screen, and we can also
reconstruct the object’s color with fine accuracy
including the reflective characteristic.
Fig. 3 shows how to choose camera to approximate theincident light ray to the virtual viewpoint. The positionsof two adjacent cameras are C1 and C2, the position of avirtual viewpoint is P, and the intersection of the light
ray and the virtual screen is Q in the figure.
We can divide space to some areas by choosing thecamera that approximates each light ray to virtual
viewpoint. The boundary of these areas is described as
the locus of Q, where PQ is the bisector of ∠C1C2Q.
We will obtain this locus here.
We set the position of 2 cameras C1(-1, 0) and C2(1, 0),
the position of virtual viewpoint P(p, q), and theintersection of the light ray and the virtual screen Q(x,y). When Q is on the boundary of the areas, ∠C1QP isequal to ∠ C2QP. Therefore by using complex plane
method,From the figure, we see that the obtained locus
approaches to the straight line PM as the distance fromcameras to Q becomes large, and the angle between PM
and C1C2 approaches 90 degrees. Besides, although it isFig. 3 Camera Selection for Approximationarg( ( x( +x +yiyi) −)−( q( +−1pi) ) ) = arg( ( x +(yix +) −yi()q−+1pi ) )(1)

It can be described asRe[(( x + yi ) − ( q + pi )( x + yi ) − ( −1)) (( x + yi ) −1
( x + yi ) − ( q + pi )Fig. 4 The locus of Point Q)] > 0Im[( ( x( +x +yiyi) −) −( q( +−1pi) ) ) ( ( x +(yix +) −yi()q−+1pi ) )] = 0( 2)

an obvious solution, in the case of the angle between PM
and C1C2 is equal to 90 degrees, the locus corresponds tothe straight line PM.

And it can be solved as below.

Apply to the TWISTER Systemf ( x, y ) = − qx 3 + px 2 y − qxy 2 + py 3

Considering TWISTER system, since the object isrestricted to the inside of TWISTER, the angle between
PC and C1C2 is close to 90 degrees, and the distance
between adjacent cameras can be set small enough,compared to distance between the camera and the object.
Thus, in the case of our image capture system we canapproximate the boundary of areas, where each camera
approximates the light, by straight line PM.

+ pqx 2 − pqy 2 + (q 2 − p 2 − 1) xy+ qx + py − pq = 0(3)

g ( x, y ) = x 4 + 2 x 2 y 2 + y 4−2 px 3 − 2qx 2 y − 2 pxy 2 − 2qy 3+( p 2 − q 2 − 1) x 2 − 4 pqxy − ( p 2 − q 2 − 1) y 2+2 px − 2qy − p 2 + x 2 > 0( 4)(In addition, when f (x, y)=0 and g (x, y)<0, PQ is thebisector of an exterior angle of ∠C1QC2. )Fig. 4 shows the locus of Equation (3) when (p, q) is(-20, -20), (-20, -5), (-5, -20) and (-5, -5). Two circles in
the figure are described as reference of the body of
TWISTER and the safety shield (objects can only exists
inside the shield). The radius of TWISTER’s body andthe distance from each camera to the center of
TWISTER are both set to 600mm and the distance
between adjacent cameras is set to 200mm. The dashedlines in the figure represent the straight line PM (M isthe middle point of C1C2).

By using the straight line PM as the boundary of theseareas, we can choose the camera, which approximate
each light ray to the virtual viewpoint, without
considering the position and the shape of the virtual
screen (Fig. 5).
4-3. Error Evaluation of Virtual Screen
When we reconstruct the image of the object from thearbitrary viewpoint based on the outputs from a limited
number of cameras, the more the virtual screen
approximates the actual object well, the more an exact
picture can be obtained. However, if there is an error
between the object's position and the virtual screen, anerror appears in output images. We will evaluate theerror here.

φ=tan −1 @γsin θ
D
1 + γcos θ

π
4
φ

0
0

0.2

0.4
γ 0.6

π
6

0.8

π
3
θ

π
2

10

Fig. 7 Angle Error with Approximation of Virtual
Screenequation (6). From this graph, it turns out that the error
can be made small by making the virtual screen close to
the object or making the angle between PQ and CQ as
small as possible.

Fig. 5 Areas where each camera
approximates light rays
As shown in Fig. 6, the position of the camera is C, theposition of the virtual viewpoint is P, one of the points
on the virtual screen projected by the camera Q, theangle between PQ and CQ is θ , and the error between
the position of the object and the virtual screen is δ .

From now on, Taylor series expansion of the equation(6) is used to analyze in detail,φ = γ sin θ − γ 2 sin θ cosθ + ο[γ 3 ](7)Here, if θ is small enough and γ is the order of 0.1 orso, the error φ of the angle direction isφ = γ sin θ(8)Moreover, if we reconstruct the image of the object, with45 degrees of field angle, seen from the virtual
viewpoint and output it on a VGA (640 x 480) display,
the number of error pixels is described as below (sin isapproximated with angle in following calculation).

640 ÷

π
4

⋅φ =

2560

π

γ sin θ [ px](9)πFor example, if θ = 30
, the error is 42.6γ[px] (If theFig. 6 Modeling Error
At this time, the amount of the angle direction error φ isdescribed as below.

PQ sin φ = δ sin(θ − φ )(5)

It is be solved as,

γ sin θφ = tan ()(6)
1 + γ cosθδ
where γ = PQ
, and γ is the relative length of the error−1compared to the distance between the virtual viewpoint
and the virtual screen. Fig. 7 shows the graph of

position of the virtual screen is assumed at the center of
TWISTER and the number of cameras is 30, the value ofθ is 30π when Q is on the boundary of the areas.).
In an output image, this error clearly appears on theboundary of the each camera’s output area in thefollowing two situations:・when the virtual screen is front rather than the actual
object’s position, the same point will appear twice on ascreen.
・when the virtual screen is back rather than the actual
object’s position, some point will disappear on a screen.

Hence if we want to prevent the error to appear on thescreen, we should make the error on the boundary of thearea under 1px. Taking account of that the error pixels
on the boundary of the area are described as the sum of
the approximation error of adjoining area,

42.6γ + 42.6γ ' < 1⇔⇔γ + γ ' < 0.0117 = 1.2%
γ < 0.6%(10)

Therefore, for example, when the distance between theobject and the virtual viewpoint is 2m, if we set thevirtual screen within 1.2 cm from the object, the error
from the approximation of the object’s position can beπ).
ignored in practice (in this case, θ = 30
In addition, the value of γ is inversely proportional tosin θ , and almost directly proportional to the number ofcameras. For example, if the number of a camera isincreased 10 times, the maximum of γ can be madeabout 6%.
4-4. How to make Virtual Screen for TWISTER
Considering TWISTER, since we don’t have themechanism for measuring the object’s position, we use
the model as virtual screen. Although the shape of aplane, a cylinder, etc. can be considered as the shape ofthe model, we adopted a N-sided prism as a model’s
shape this time (N is the number of cameras attached on
TWISTER at equal intervals) because of the easiness of
the rendering procedures. In this way, each camera’s
virtual projection screen, which is separated by theboundary as mentioned in Section 4-2, is set as a plane
parallel to the image surface of the camera and theoutput clipped from the camera by scanning line can bestuck on a virtual screen only by expansion andcontraction. It makes rendering process very easy.
Here we describe in detail about the method of making
such a N-sided prism virtual screen. In the following
explanation, we explain by using the upper surface
figure of the system. For example, an expression“triangular prism ABC” means a prism that is displayed
as a triangle ABC in an upper surface figure.
First, we set the center of TWISTER O, virtual
viewpoint P, the radius of TWISTER R, N cameras
located on TWISTER at equal intervals Ci (i=0, … N-1)
and the middle point of CiCi+1 Mi (CN=C0). And, we set
the virtual screen as a N-sided prism M0’M1’M2’…MNwhich is obtained from a N-sided prism
1’,
M0M1M2…MN-1 by a central similarity of center P. The
ratio belongs to the value of δ , which is the distance
between the N-sided prism virtual screen and the centerof TWISTER. At this time, when the virtual screen
moves with movement of virtual viewpoint’s position P,
the movement follows an envelope curve, a pillar withradius δ and center O. The area on the virtual screen

Fig 8: Cylindrical Virtual Screen of TWISTER
which camera Ci takes charge of is Mi’Mi+1’, which is theplane parallel to the image plane of Camera Ci.
4-5. Necessary Field Angle for Camera
In TWISTER, the area where the object can exist islimited in a safety shield, and cameras are arranged on
the cylinder outside the shield. Here we set the distancefrom the center of TWISTER to each camera R1 and aradius of the safety shield R2. At this time, we need
2 sin −1 ( R 2 / R1 ) for each camera’s field angle to capture
the whole images in the shield. In other words, if theeach camera’s field angle α is given, the area captured
from all the cameras is the cylinder with radius
R1 sin(α / 2) .
Additionally, considering TWISTER II, since the radiusof the safety shield is 400mm and the distance from thecenter of TWISTER to each camera is 600mm, we needsin −1 (400 / 600) = 83.6[deg] for the field angle ofcameras to capture the whole images in the shield.
4-6. Summary of Theory
We have proposed the methods to determine the camera
for each light ray incident to the arbitrary virtual
viewpoints, and the method to make virtual screen, andhave evaluated the error between the virtual screen andthe object. As described in Section 4-2, when we adopt“the bisector model” for the camera selection, we canreduce the effects on the virtual screen from theapproximation error of modeling. At the same time byadopting “the middle point model”, which is theapproximation of “the bisector model”, we can select thecamera independently of the determination of the virtual
screen. Although, we have set the virtual screen as a Nsided prism for easiness of rendering, this idea can beapplied to any virtual screen model.

5-2. Virtual Screen
In this experiment, we set the virtual screen as a 30sided prism using the method described in Section 4-4.
5-3. Result
The picture obtained in the NTSC selection stage isshown in Fig. 11. These images represent the necessary
NTSC scanning lines for reconstructing the image of theobject seen from a virtual viewpoint, which position is
1800mm distant from the center of the booth, using theoutput of 5 cameras of 12 sets. In addition, since there
exists some overlap areas, where two adjacent cameras
require the same number of scanning line, we arranged
two switching circuit and divided the output of the evennumber cameras and the odd-number cameras.

Fig. 9 Necessary Field Angle for Camera

5. Experiment
The experiment was conducted based on the above
theory. We have selected the required light informationby scanning line here. The rendering process is plannedin the near future.
5-1. System Architecture
To imitate the situation that 30 cameras are arranged ona cylindrical device (TWISTER), 12 small CCD cameras(Toshiba IK-C40) were put in order at intervals of 12
degrees (Fig. 10). The synchronization is taken by thecommon Genlock input, and each camera is rotated 90[deg], so that the scanning lines of a camera become
direction of the perpendicular direction. This
arrangement enables us to select line images from analog
signals in a scanning line unit.

Fig. 11 Output Images Selected by Scanning Lines
As a next step, we plan to input these obtained images tothe rendering PC, texture-map on a 30-sided prism
virtual screen, and generate an object’s image seen fromthe virtual viewpoint in real time (Fig. 12).

Since the output of 12 cameras has the huge amount ofdata and it is difficult to deal it with a single PC, we
select required NTSC scanning lines in an early stage
using switching circuits.

Fig. 12 Texture mapping and rendering(Future work)

6. Future Works
6-1. Object Clipping from a BackgroundFig. 10 Image Capture System

In TWISTER, since only the object is clipped fromobtained images and arranged to the virtual environment,

an actual background does not have a meaning. Hence,
we should consider the method of clipping the object
image from the background image. Basically, we plan touse luminance key. Only the object is lighted, whilesurroundings are made dark, and the object is clippedfrom black background.
Since TWISTER system contains not only image capturecomponent but also display component, we have todelete the image of bright LED. There are some methods
to delete the image of LED.

2.

C. Cruz-Neira, D. J. Sandin and T. A. DeFanti:
Surround-Screen Projection-Based Virtual Reality:
The Design and
Implementation of the CAVE,
Proceedings of ACM SIGGRAPH '93, pp. 135-142(1993)

3.

S. Tachi, T. Maeda, Y. Yanagida, M. Koyanagi and
Y. Yokoyama: A method of mutual tele-existence in a
virtual environment, Proceedings of ICAT '96, pp. 9-18, 1996.

4.

Jason Leigh, Thomas A. DeFanti, Andrew E.
Johnson, Maxine D. Brown, and Daniel J. Sandin:
Global tele-immersion: Better than being there, In
ICAT97, pp. 10-17, 1997

5.

M. Hirose, T. Ogi, S. Ishikawa and T. Yamada: A
Development of Immersive Multiscreen Display(CABIN), Proceedings of the Virtual Reality Societyof Japan Second Annual Conference, pp. 137-140(1997)

6.

F. Okuno, H. Hoshino, J. Arai and I. Yuyama:
Real-time Pickup Method for a Three-dimensional
Image based on
Integral
Photography,
Applied Optics, Vol.36, No.7, pp. 1598-1603 (1997)

7.

Y. Kunita, M. Inami, T. Maeda, S. Tachi: Design ofVisualDisplay for Mutual Tele-Existence,
Proceedings of the Virtual Reality Society of Japan
Second Annual Conference, pp. 352-353 (1997)

8.

Y. Kunita, M. Inami, T. Maeda and S. Tachi:
Prototype System of Mutual Tele-existence, ACM
SIGGRAPH '99 Conference Abstracts andApplications, p. 267 (1999)

9.

Y. Kunita, M. Inami, T. Maeda and S. Tachi: Realtime rendering system of moving objects, Proceedingsof the 1999 IEEE Workshop on Multi-View Modeling\& Analysis of Visual Scenes (MVIEW'99), pp. 81-88(1999)

・ Synchronize the image capture cycle with LED’s
flashing cycle.
・ Calculate the position of LEDs and overwrite
black there
Much more consideration is necessary about this issue.
6-2. Rotation of Cameras
As evaluated in Section 4-3, it is difficult to obtain theaccurate images by using only 30 cameras or so. We
considers image-capture component to rotate around theobject with display component to increase the number ofcameras virtually,
By rotating high-speed camera around the user while
getting many images in high time frequency, we canobtain images from much more positions than thenumber of cameras (We call this camera “Virtual
Camera”). Furthermore, since high-speed movements ofcameras are hard to recognize for the user, the cameras
can capture the user’s image without interrupting user’sviewing action.

7. Summary
We have descried the theory of image-capture system of
TWISTER, the outline of the system design and theexperiment for it. From this consideration, it turns outthat how to choose the camera reconstructing each lightinformation and the setting method of a virtual screen
can be considered independently. Furthermore, we haveevaluated the error between virtual screen and theposition of the object, and suggested the N-sided prism
virtual screen for TWISTER.
We will advance the experiment. We plan to generate
the object’s image from arbitrary virtual positions using
this system, and display the images on TWISTER’s
display component or another devices in the future.

References
1.

F. Kishino: Human Communication -the VirtualSpace Teleconferencing system-: Proceedings of The
Institute of Image Information and Television
Engineers, Vol. 46, No. 6, pp. 698-702, 1992.

10. Y. Kunita, M. Inami, T. Maeda and S. Tachi: RealTime Rendering System of 3D Moving Objects,
Proceedings of the 3D Image Conference '99, p. 13-18(1999)
11. Y. Kunita, M. Inami, T. Maeda and S. Tachi: A
Real-time Projection System of Photo-Realistic
Human Figures intoThree-Dimensional
VirtualEnvironments, Proceedings of the 5th Symposium onIntelligent Information Media (IIM 1999), pp. 207212 (1999)
12. Y. Kunita, N. Ogawa, A. Sakuma, M. Inami, T.
Maeda and S. Tachi: Immersive Autostereoscopic
Display for Mutual Telexistence: TWISTER I(Telexistence Wide-angle Immersive STEReoscopemodel I), Proceedings of the 3D Image Conference
2000, p. 111-114 (2000)
13. Y. Kunita, M. Inami, T. Maeda and S. Tachi:
Immersive Autostereoscopic Display for Mutual
Telexistence (II):Design and Development of

TWISTER II, Proceedings of the Virtual Reality
Society of Japan 5th Annual
Conference, p. 369370 (2000)
14. T. Endo, A. Katayama, H. Tamura, M. Hirose: An
Omnidirectional Image Construction Method from
Multi-viewpoint Images, Proceedings of the Virtual
Reality Society of Japan 5th Annual Conference, pp.
415-418 (2000).
15. Y. Kunita, N. Ogawa, A. Sakuma, M. Inami, T.
Maeda
and
SusumuTachi:
Immersive
Autostereoscopic Display for Mutual Telexistence:
TWISTER I (Telexistence Wide-angle Immersive
STEReoscope model I), Proceedings of the IEEE VR
2001, pp.31-36 (2001)
16. J. Tago, T. Nitta, T. Naemura, and H. Harashima:
Video-Based Rendering with Dynamic Layered
Representation, Proceedings of the 3D Image
Conference ‘01, pp.33-36 (2001).
17. K.Tanaka, J. Hayashi, Y. Kunita, M. Inami, T.
Maeda, S. Tachi: The Design and Development of
TWISTER II: Immersive Full-color Autostereoscopic
Display, Proceedings of ICAT '01, pp. 56-63 (2001).
18. J. Hayashi, K. Tanaka, M. Inami, and S. Tachi: An
Immersive Full Color Autostereoscopic Display,
Proceedings of the 3D Image Conference '02, p. 93-96(2002)
19. K. Tanaka, J. Hayashi, I. Kawabuchi, D. Sekiguchi,
M. Inami, N. Kawakami and S. Tachi: The
Design and Development of TWISTER III, an
immersive full-color autostereoscopic display formotion pictures, Proceedings of the Virtual Reality
Society of Japan 7th Annual
Conference, p. 411412 (2002)