Abstract:
A system and method for enhancing the realism of a displayed image may include an image display subsystem and a shadow projector. When an image is displayed, a shadow is projected near the image so that the shadow appears to be cast by the image. Shadows and lighting effects may also be reflected within a displayed image by rendering shadows and lighting corresponding to the viewer&#39;s environment within the image before or during display.

Description:
FIELD OF THE INVENTION 
     This invention relates to image displays and more particularly to enhancing the realism of images by creating an illusion of environmental lighting effects from and on the images. 
     BACKGROUND OF THE INVENTION 
     The present invention relates to the field of optics, and this application makes use of several terms of art used throughout the field. The inventors have adopted and use the following definitions, and they are therefore included in this patent. A ‘real image’ is an optical image formed of real foci (Merriam-Webster&#39;s Online Collegiate Dictionary). In other words, real images are made by light rays that are brought to a focus at the image position, and therefore can be made visible on a surface, such as a screen or smoke. See Warren J. Smith, Modem Optical Engineering, 8 (McGraw-Hill 1966). A ‘virtual image,’ on the other hand, is an image formed of points from which divergent light rays seem to emanate without actually doing so (Merriam-Webster&#39;s Online Collegiate Dictionary) . For example, the virtual image in a plane mirror is at some distance behind the mirror. Virtual images cannot be made visible on a screen because they are made by light rays that do not actually come from the location at which the image appears. See Warren J. Smith, Modem Optical Engineering, 9 (McGraw-Hill 1966). As used throughout the specification, a ‘negative image’ is a negative of an image, or an image having the light and dark parts in approximately inverse order to those of the original subject (Merriam-Webster&#39;s Online Collegiate Dictionary). A negative image may be created by reversing the polarity of an image, digital signal processing, or forming a two-dimensional ‘gobo,’ which is an opaque cut-out resembling the original subject to be placed near a light source. A ‘focal plane’ is a plane that is perpendicular to the axis of a lens or mirror and passes through the focus of the lens or mirror (Merriam-Webster&#39;s Online Collegiate Dictionary). 
     Movies, slides, video, and other displayed images have long been a major source of audience entertainment. Much effort has been expended to make images appear more like objects in three-dimensional space in order to make them more compelling to the viewer, and thereby increase the entertainment value of such images. A variety of techniques have been employed to cause displayed images to appear more like objects in three-dimensional space. One technique is to create “aerial” images, which are virtual images that are projected into free space, producing the illusion of a three-dimensional hologram. Such virtual images are typically created using concave mirrors and positive lenses that form images of, for example, physical objects, film projections or video screens. Examples of such a technique can be found in Krause et al., U.S. Pat. No. 5,944,403 and Orlandi, European Patent No. EP-B1-0 410 968. 
     Another technique to enhance the illusion of realism of displayed images is to clearly establish the distance between the viewer and the image to create an illusion of image depth. One method for creating such an illusion is to place physical objects near the plane of focus of the image in order to establish the relative depth from the image and the object to the viewer. Another method is to place a semi-transparent scrim between the optical objective and the image, and to project an image on the scrim that emphasizes the position of the image relative to the scrim. 
     Although these techniques succeed to one degree or another in making images more compelling, the images still appear unrealistic because the lighting in and around the images does not match the surrounding environment. In particular, images do not cast a shadow on their surroundings, nor do they contain shadows or reflect lighting effects from light sources in their environments. 
     SUMMARY OF THE INVENTION 
     To overcome the limitations in the prior art described above, the present invention discloses a system and method for enhancing the realism of displayed images by reflecting lighting effects from the displayed image&#39;s environment in and around the image. 
     Specifically, in one embodiment of the invention, the system comprises an image display subsystem and a shadow projector. The image display subsystem may be, for example, a virtual or real image projector or video display, and it may display an aerial image or a real image. The shadow projector may be, for example, a slide, film, or video projector or a theatrical lighting fixture with gobo. The shadow projector projects a second image, which may be from a photographic negative or an inversion of the signal of the displayed image, near the displayed image such that the second image appears to be a shadow of the displayed image. Since real shadows are created only by physical objects, the displayed image appears more like an object in three-dimensional space. 
     The “shadow” projected by the shadow projector can be made to correspond with the shape, alignment, movement, and perspective of an object or image relative to a real or virtual light source. By a “virtual light source,” the inventors mean a light source within an image. For example, a virtual light source may be a sun projected on a movie screen. With a second image source comprising fixed media, for example, a gobo or slide, the source can be changed in discrete steps with the movement of a light source or the image. When the second image source comprises continuous media, for example, video or film, the image correspondence may take place continuously. By corresponding the shadow image to the displayed image, the image appears to cast a realistic shadow, further enhancing the illusion of realism. 
     In another embodiment of the invention, the displayed image contains a virtual light source. The shadow projector projects a shadow near either an object image within the displayed image or a real object near the displayed image, such that the “shadow” appears to be cast from the virtual light source. This creates an illusion that the virtual light source is a real light source, and further enhances the realism of the image. 
     An alternative embodiment comprises an image display subsystem and a first image source. The lighting and shadows from light sources and physical objects in the viewer&#39;s environment are integrated into the image. When displayed, the image contains realistic lighting and shadows from the viewer&#39;s environment. This enhances the realism of the display by creating an illusion that the image is lighted by the environmental light sources or shadowed by real objects in its environment. 
     Other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the invention. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 illustrates one embodiment of the invention, wherein a virtual image projector projects an image of an object and a shadow projector casts the negative of the image in the direction of the image such that it appears to be a shadow of the image; 
     FIG. 2 illustrates an alternative embodiment of the system shown in FIG. 1, wherein the apparent shadow moves as the shadow projector changes its projection angle; 
     FIG. 3 illustrates another embodiment of the invention, wherein both the image display projected by the virtual image projector and an inverter are driven by the same video storage device; 
     FIG. 4 illustrates another embodiment, wherein a real object is placed near an image containing a virtual light source. A shadow projector projects a shadow of the object that appears to be cast from the virtual light source; 
     FIG. 5 illustrates another embodiment of the invention, wherein the projected image appears to contain a shadow cast by the shadow-casting object from an environmental light source; 
     FIG. 6 illustrates an alternative embodiment of the system in FIG. 5, wherein the object, shadow, and virtual light source are rendered on an electronic display device by a video source before being projected; and 
     FIG. 7 illustrates an alternative embodiment of the system of FIG. 6, wherein the virtual light source is rendered to correspond with a moving environmental light source. 
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The invention summarized above and defined by the enumerated claims may be better understood by referring to the following detailed description, which should be read in conjunction with the accompanying drawings. This detailed description of particular preferred embodiments, set out below to enable one to practice the invention, is not intended to limit the enumerated claims, but to serve as particular examples thereof. It is to be understood that other embodiments may be utilized as changes may be made without departing from the scope of the present invention. 
     FIG. 1 illustrates one embodiment of the present invention. An aerial image  10  of a real object  11  is displayed by an image display subsystem  12 . In this embodiment, the image display subsystem  12  is a virtual image projector, using a concave mirror. A shadow projector  14  has a focal plane containing a negative image  16  of the aerial image  10 . The shadow projector  14  is positioned in the proximity of the aerial image  10  such that projection of the negative image  16  appears to create a shadow  18  of the image  10 . In this embodiment, the shadow projector  14  is a film slide projector. The shape and size of the negative image  16  and resulting projected image or shadow  18  are configured to be appropriate for the geometrical relationship of the shadow projector  14  and the image  10 . A baffle or scrim  20  is added as shown to shield the object  11  from view. The object  11  may be replaced by a real image displayed or projected on a screen. The image  10  may alternatively be a real image. The image display subsystem  12  may alternatively be a virtual image projector using a positive lens, or a real image display, for example, a film slide projector, motion picture film projector, video projector, or a display from an electronic display device. The shadow projector  14  can also be, for example, a theatrical lighting fixture with a gobo, motion picture film projector, or video projector displaying an electronically stored negative image. 
     FIG. 2 illustrates the movement of the shadow projector  14  shown in FIG. 1 to a new position and alignment. The corresponding change in shape and orientation of the projected shadow  19  is also shown. As the shadow projector  14  moves, the negative image  15  changes as needed to maintain proper perspective. If the shadow projector  14  uses fixed media, such as a transparency slide or gobo, then the transformation of the negative image  15  can be accomplished in discrete steps by substituting different fixed negative images ( 15 ,  17 ) as necessary at each new position of the shadow projector  14 . If the negative image  15  is continuously variable, as with motion picture film or video, then the motion of the shadow projector  14  and the shadow  19  can be accomplished continuously. By changing the shape of the negative image  15  as the shadow projector  14  is moved, such that the contour of the shadow  19  accurately represents the perspective of the geometry of the shadow projector and image, the realism of the aerial image can be further enhanced. 
     FIG. 3 illustrates another embodiment of the invention. The image display subsystem  12  displays an aerial image  9  of a real image  13 , for example, a moving picture displayed on a screen. A video source  22  drives both the real image  13  display and an inverter  24 . The inverter  24  reverses the polarity of the video source  22  using digital signal processing, and outputs the signal to the shadow projector  21 . The shadow projector  21  projects the inverted signal of the real image  13  as a shadow  23  in the proximity of the displayed image  9 , such that the shadow  23  appears to be cast by the displayed image  9 . By using the video source  22  as a source for both the displayed image  9  and the shadow  23 , moving pictures and their shadows can be easily synchronized. The simplicity of the system is also increased, avoiding the need for a separate image source  25  for the shadow projector  21 . 
     Synchronization may also be accomplished without using an inverter  24 . When both the image display subsystem  12  and the shadow projector  21  are film or slide projectors, the negative of the film used for the image display can be created and used as the source  25  for the shadow projector  21 . The image  9  may alternatively be a real image. The image display subsystem  12  may alternatively be a virtual image projector using a positive lens, or a real image display, for example, a film slide projector, motion picture film projector, video projector, or a display from an electronic display device. 
     FIG. 4 illustrates another embodiment of the invention. An image display subsystem  26  displays a real image  30  containing a virtual light source  32 . An object  34  is placed near the real image  30 . The shadow projector  14  projects a shadow  27  near the object  34 , such that the shadow appears to be cast by the object  34  from the virtual light source  32 . A shadow  27  may also be projected near an image of an object that is also contained in the real image  30 . This system and method allow both real objects and images to appear to cast shadows from a virtual light source. As discussed above, the shadow projector  14  may also be a motion picture or video projector displaying a negative  16  of the film or inverted digital signal used to display the first image  28 . 
     FIG. 5 illustrates an embodiment of the invention comprising an image display subsystem  12 , an image source  37 , and an aerial image  41 . A real shadow  36  is cast onto the image source  37  by a real object  38  from a real light source  40 . The shadow  36  is aligned on the image source  37  such that, when displayed, the image  41  appears to be shadowed by a second real object  42  that is near the displayed image  41 . The light source  40  is positioned so as to illuminate the image source  37  such that, when displayed, the image  41  appears to be lighted by a second light source  44  that is in the environment of the displayed image  41 . The light source  40  also corresponds with the movement, brightness and color of the environmental light source  44 . By reflecting environmental light sources and shadows from real objects near the displayed image in the image itself, the illusion of realism is further enhanced. The displayed image  41  may appear to be a physical object, capable of reflecting the lighting and shading of its environment. 
     FIG. 6 illustrates that the image source  37  of FIG. 5 may be the output  39  of a video source  22 . A virtual shadow  36  is rendered on a real image display according to the perspective of a virtual light source  40  and an object image  38 . The virtual light source  40  and the object image  38  are rendered on the display so as to correspond with the real light source  44  and real object  42  when the image  41  is displayed. The image source  39  may be film, slides, or an electrical signal. Rendering may be accomplished by recording images of real or computer generated objects with similar lighting on film, slides, or video storage media such as a video disc or tape, so that when the image is played back through the image display subsystem  12 , the environmental lighting and shading are reflected in the image  41 . Rendering may also be accomplished by generating and displaying the images in real-time. Rendering the shadows and lighting effects within an image, rather than using real objects, allows moving and more complicated images to be easily displayed. 
     In FIG. 7, a light source position sensor  46  senses the movement of the environmental light source  44 , and outputs the position data to a light source position processor  48 . The processor  48  then outputs the data to the video source  22 , which renders a corresponding change in position for the virtual light source  40 . Where a digital storage medium, such as a disc or solid state memory is employed, the position of the light source  44  can be used as an index to retrieve and display a prerecorded image  39  that is appropriate for the lighting geometry in the environment. Sensors may also be used to sense and output color and brightness data to the processor  48 . By dynamically changing the illumination of the displayed image  41  to correspond to the environmental lighting and shading, the realism of the image is further enhanced. 
     In embodiments where a real-time three-dimensional computer image generator is employed as the video source  22 , the position data from the light source  44  can control the position of the virtual light source  40  used to render the three-dimensional object  36 , which is displayed on the display  39  and imaged by the image display subsystem  12  as an aerial image  41 . The image display subsystem  12  may alternatively be a real image display, such as a display of an electronic device, slides, film or video projected on a screen, or a television screen. Such real-time image rendering enables the image  41  to reflect lighting changes as they occur, without the need of prerecording those changes. 
     In any of the above embodiments, the image display subsystem  12  may be a film or slide projector and screen, a television, video, or liquid crystal display screen, or a virtual image projector, projecting aerial images of the real image displays described above or of real objects.