                 
Tomoya Ishikawa Kazumasa Yamazawa†
Naokazu Yokoya ‡Graduate School of Information Science, Nara Institute of Science and Technology

(a)

(b)

(c)

(d)

Figure 1: Novel view telepresence: (a, b) remote and user’s sites and (c, d) generated omni-directional and planar perspective images.

 With high-speed network and high-performance PC, networked
telepresence, which allows a user to see a virtualized real scene in aremote place, has been investigated. Uyttendaele et al.[2004] haveproposed a system which enables a user to change the viewpoint
and view-direction interactively like walk-through in a real scene.
However, the viewpoint is restricted onto the camera path. Kanadeet al.[1997] have proposed a method which generates novel view
images in the 3D dome. Their system needs the assumption thatthe positions of objects are limited within the area covered by theoutside-in cameras. Therefore, it is suitable for 3D modeling, rather
than telepresence. We propose a novel view telepresence system
which enables a user to control the viewpoint and view-direction
freely by virtualizing real dynamic environments. Our system uses
multiple omni-directional cameras for virtualizing both inside-out
and outside-in observations from camera positions in a real scene
simultaneously. Furthermore, it can generate novel view images inreal-time from omni-directional live videos transferred on network.
The example is shown in Figure 1.

          
Figure 2 illustrates a configuration of a prototype telepresence system as well as the flow diagram. The prototype system enables auser to see a virtualized remote scene at arbitrary viewpoint and inarbitrary view-direction interactively.
The novel view generation is based on using multiple omnidirectional cameras for obtaining both inside-out and outside-in observations from camera positions in a real scene simultaneously.
Furthermore, for supporting dynamic scenes, the acquired images
are segmented into static and dynamic regions by using a robust
background subtraction technique. Novel view images are generated for each region in real-time. See [Ishikawa et al. 2005] for thedetail of novel view generation for static and dynamic regions.
We have implemented the proposed method and incorporated it into
a server-client system. We use three omni-directional cameras connected to a PC for transmission of multiple video streams. ThePC as a video stream server transfers not only the captured video
streams but also the binary silhouette images made by backgroundsubtraction. The background subtraction process does not dependon user’s viewpoints, so the server executes this process before
transmission. The acquired video streams and the silhouette images are transferred to another PC as a novel view generator in theuser’s site via network.
The novel view generator is connected with a magnetic sensor formeasuring user’s viewpoint and view-direction and an HMD for
displaying novel view images. The magnetic sensor measures the e-mail: tomoya-i@is.naist.jp†

e-mail: yamazawa@is.naist.jp‡ e-mail: yokoya@is.naist.jp

Remote site

User’s siteTransmitter

Omni-directional cameras

Video stream server
Acquisition of multiple
omni-directional images

HMD with receiver

Magnetic sensor

NetworkUser

Novel view generator
Reception of multiple
videos and mask imagesStatic region

Dynamic region

Segmentation into
static and dynamic regions

Novel view generation Novel view generation
using Morphingusing Visual Hull

Transmission of multiple
videos and mask images

Synthesis of two novel view images
Display of synthesized novel view image

Figure 2: Configuration of prototype system and flow diagram.
receiver’s 6DOF data. The novel view generator computes a novelview image according to the user’s viewpoint and view-direction
and display the image on the HMD.

    

Figure 1 shows appearances of both remote and user’s sites andgenerated omni-directional novel view image as well as planar perspective images. Planar perspective image is presented to an HMD
mounted on a user. The user can change the viewpoint and viewdirection freely and sense the feeling of being in a remote site.
Each frame of novel view is generated in about 200[ms] by a PC
(Pentium4-3.2GHz); that is, the frame rate is 5[fps]. Note that, thecalculation time is not constant because the time required depends
on the size of dynamic objects. When the user’s viewpoint is near
the cameras, the quality of novel view becomes high.
We have confirmed that our system can provide a user with the feeling of being in a remote site. The system can be applied to virtualsightseeing, remote conference, and so on. In future work, we will
improve the generated image quality, especially one of dynamic regions by improvement of background subtraction and so on.

    I SHIKAWA , T., YAMAZAWA , K., AND YOKOYA, N. 2005. Novel
view generation from multiple omni-directional videos. In Proceedings of IEEE and ACM Int. Symp. on Mixed and Augmented
Reality (ISMAR05), 166–169.
K ANADE, T., R ANDER, P., AND NARAYANAN , P. J. 1997. Virtualized reality: Constructing virtual worlds from real scenes.
IEEE MultiMedia Magazine 1, 1, 34–47.
U YTTENDAELE , M., C RIMINISI , A., K ANG , S. B., W INDER , S.,S ZELISKI , R., AND H ARTLEY, R. 2004. Image-based interactive exploration of real-world environments. IEEE Computer
Graphics and Applications 24, 3, 52–63.