AN EFFECTIVE INTERACTION TOOL FOR PERFORMANCE IN THE VIRTUAL STUDIO-INVISIBLE LIGHT PROJECTION SYSTEM

Virtual studios are now used for various programs such as news, weather forecasts, and educational programs. However, the performers in a virtual studio cannot directly see the surrounding virtual background or their virtual co-performers (i.e., virtual subjects). Therefore, performers must check the position of virtual subjects by watching a monitor, causing them to look away from the virtual subject. Viewers see an unnatural image in which the performer seems to be looking in the wrong direction. To overcome this problem, we have developed an interaction support system called the Invisible Light Projection System. By using projector images, we made virtual subjects visible within the studio but invisible to the camera. This system has been used in actual NHK studio production. In this paper, we describe the system and how it can make virtual-studio production more effective. INTRODUCTION The virtual studio, introduced in the mid-1990s, is a video composite technique that uses images of performers shot in front of a chroma-key blue background (i.e., a blue screen) and virtual images created through real-time computer graphics (CG). Composite images made using the chroma-key technique create an effect by which performers seem to be inside the virtual images. The advantages of this technique are that the virtual images enable visual effects that cannot be created using an actual set, storage space is unnecessary, and changes in the virtual images can be made relatively easily. However, some of the differences from standard studio video production can create problems in directing and shooting. For example: a. To obtain key images of the performers, a blue screen must be used for shooting. b. In blue-screen shooting, flat lighting is necessary, making it difficult to produce shadows of the performers in the composite picture. c. Performers cannot directly see virtual subjects with which they need to interact. Technical solutions to some of these problems have been developed. For example, the AxiVision (developed by NHK) and Zcam (developed by 3DV Systems) systems can detect the camera distance to the subject while shooting, enabling key signals of the performers to be cut out without using a blue screen. There is also a technique by which the key image of the performer is processed and composited as a shadow. However, a technique that allows performers to see virtual subjects is still needed. Otherwise, performers will have to continue watching virtual subjects through monitor images, which often causes them to look in an inappropriate direction. This is particularly obvious in HDTV productions, because of the Figure 1 – Performer checking the CG composite title by looking at a studio floor monitor image clarity. To overcome this problem, we have developed the Invisible Light Projection System. In this paper, we describe the composition and application of this system. VIRTUAL SUBJECTS IN THE STUDIO Current checking methods and their disadvantages The usual method used to allow performers to check the position of virtual images has been to mark positions with tape on floors or walls, or, as shown in Fig. 1, to look at the image composite on a studio floor monitor. The disadvantage of markers is that they cannot be used with moving virtual subjects, and that the use of too many markers can confuse performers. Also, a performer lacks a direct feel of what they are supposedly interacting with. The alternative is for the performer to guess the relative distance between himself and the virtual subject by looking at a studio floor monitor that shows the composite image of the performer and the virtual image. This causes the performer to look at the monitor rather than at the virtual subject. With either method, the need to act naturally while being unable to see the virtual subject places a heavy burden on a performer’s acting skills. As a result, thorough rehearsal prior to shooting is necessary, and poor performance that necessitates additional re-takes often prolongs the production time. Actors in movies and commercials must also look at CG co-actors. In these situations, the actors are guided by another actor who substitutes for the CG image, or by a marker ball attached to a pole. The actor can see the substitute for the CG, enabling him to maintain proper eye contact, so he can concentrate on his own performance. These substitutes are erased from the final composite image during post-production. While this method can be used in television production, its cost makes it impractical and the need for post-production work makes it unsuitable for live broadcasts. Under these circumstances, there is a definite need for an interaction support system designed for use in television production. Our solution To show performers the position of virtual subjects, we used projected light as an interface. Projected light can be used to mark the position of a moving virtual subject, and can be turned off when not in use. Further, by applying time-division to the projected light, we have created an interface that is not detected by television cameras. The NTSC video signal is a continuous image of 59.94 fields. Under normal shooting conditions, one field is approximately 16.7 ms. The CCD camera, which is the standard camera for television production, has an electronic shutter function. When the shutter is set to 1/100, the exposure time is 10 ms per field, leaving the remaining 6.7 ms as unexposed time. By controlling the projected light so that it is intermittent and turned on only during the non-exposure time, we made it “invisible” to the camera. To confirm the validity of this principle, we did an experiment using camera-synchronized electronic control and time division applied to laser light. We adjusted the laser light so that it was emitted only during the non-exposure time of a CCD camera whose shutter speed we set to 1/100 (Fig. 2). As expected, we found that the projected laser light did not appear in the CCD camera images. Since this interface is visible to the human eye, but invisible to the camera, we named it Invisible Light. Figure 2 – Timing of Invisible Light We then used Invisible Light in a demonstration to indicate the position of a moving virtual subject to a performer. The performer was able to follow an Invisible Light image that was projected onto the studio floor and the performer's movements were synchronized with the corresponding CG image. Because the Invisible Light did not appear in the television camera image, conventional image compositing could be applied. The composite image showed the performer chasing a randomly moving CG ball while apparently maintaining steady eye contact with the ball. These results confirmed that Invisible Light was an effective means of establishing the position of virtual subjects. However, since laser light could be dangerous to the performer, we had to use a different type of projected light for positioning. Also, to create positioning forms other than a single point – for example, characters and complex images – we had to develop a new projection system. Figure 4 High-speed LC shutter THE INVISIBLE LIGHT PROJECTION SYSTEM To display video images with Invisible Light, we decided to control the projector light by using liquid crystal shutters. System Structure The system configuration of the Invisible Light Projection System is shown in Fig. 3. A newly developed high-speed liquid crystal shutter (LC shutter) is placed in front of the projector lens. The projector displays the virtual subject. As shown in Fig. 4, the LC shutter can be adjusted to become transparent, and it controls the projected light to make it invisible to the camera. The “invisible” projected image can be adjusted so that it will match the size and position of the virtual subject or the composite image. By watching this projected image, performers can orient themselves with the virtual subject. Fig. 3 System configuration Development of the High-Speed LC Shutter Initially, we considered a control method that would use a mechanical shutter. However, the equipment would be bulky, and shutter noise would be a problem. We also required a steady operation system that could be synchronized with the non-exposure time of the camera, and realized that a mechanical shutter would not be suitable for studio production. We then developed an electronically controlled LC shutter (Fig. 4) that can be synchronized with a CCD camera shutter. The shutter, made of a composite film of nematic LC and polymer, can modulate incident light at a very high speed – within the time of one field. This high-speed LC shutter has a higher maximum transmittance than that of conventional LC shutters with polarizing plates. Through high voltage control, we achieved a high-speed response with rise and decay times of under 2 ms. By installing this high-speed LC shutter in front of the projector lens, we can display projector images in Invisible Light. Inside the Studio Camera Image Composite Image High-speed LC shutter High-luminance projector Studio Camera Image Compositing Unit PC for CG rendering CG Image LC shutter control unit Performer