Abstract:
The perception of a displayed image is altered for viewers moving relative to the position of the display system screen, thereby imparting a sense of three-dimensional immersion in the scene being displayed. A display generator generates a scene having foreground and background elements, and a display screen displaying the scene. A sensor detects the position of a viewer relative to the display screen, and a processor is operative to shift the relative position of the foreground and background elements in the displayed scene as a function of viewer position, such that the viewer&#39;s perspective of the scene changes as the viewer moves relative to the display screen. The foreground and background elements may be presented in the form of multiple superimposed graphics planes, and/or a camera may be used to record the scene through panning at sequential angles. The system may be used to implement virtual windows, virtual mirrors and other effects without requiring viewers or users to modify behavior or wear glasses, beacons, etc.

Description:
REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims priority from U.S. Provisional Patent Application Ser. No. 60/957,845, filed Aug. 24, 2007, the entire content of which is incorporated herein by reference. 
     
    
     FIELD OF THE INVENTION 
       [0002]    This invention relates generally to display systems and, more particularly, to systems that alter perspective, synthesize depth perception and provide other capabilities, thereby enhancing the viewing experience. 
       BACKGROUND OF THE INVENTION 
       [0003]    Flat panel displays are growing in size and falling in price. At this time, non-projection, true HD (i.e.—1080 p) 50″ displays are available in the $1,000 to $2,000 price range, and new models are introduced on a regular basis. When edge-lit liquid-crystal display (LCD) panels are replaced with back-lit white light-emitting diodes, another leap in technology will occur. When organic LED panels become viable, flexible, affordable wall-sized displays, including wrap-around configurations, should be possible. 
         [0004]    As large displays proliferate, users are finding uses for them beyond just “watching TV.” In some cases, for example, users are displaying pictures or rotating sequences of pictures on these displays, thereby creating, in effect, large picture frames. However, existing systems for video display are restricted either in their utility or in their realism, due to the inherent limitations in a two-dimensional presentation unit. Of these, the most important impact to the perception of a scene may be its lack of visual perspective as the viewer changes his position relative to the display unit. 
         [0005]    The use of 3-D glasses or other paraphernalia is oppressive in those circumstances where a casual effect is desired, or where no user intervention is to be required. Thus, prior-art systems relying on still or even video images, or various photographic or video projection techniques, cannot achieve the level of perception of reality that is desired for many residential or commercial display applications. 
       SUMMARY OF THE INVENTION 
       [0006]    This invention resides in apparatus and methods providing a unique experience for the viewer of a display, particularly large wall-mounted panels. In the preferred embodiments, the perception of a displayed image is altered for viewers moving relative to the position of the display system screen, thereby imparting a sense of three-dimensional immersion in the scene being displayed. 
         [0007]    A perspective-altering display system according to the invention comprises a display generator for generating a scene having foreground and background elements, and a display screen displaying the scene. A sensor detects the position of a viewer relative to the display screen, and a processor is operative to shift the relative position of the foreground and background elements in the displayed scene as a function of viewer position, such that the viewer&#39;s perspective of the scene changes as the viewer moves relative to the display screen. 
         [0008]    The foreground and background elements may be presented in the form of multiple superimposed graphics planes. At least one of the graphics planes may include prerecorded material or material received through a transmission medium or camera. A camera may be used to record the scene through panning at sequential angles, with a memory being used to store the images obtained at the sequential angles for later recall as a function of user movement. An interpolator may be used to “fill in” visual gaps in the scene. 
         [0009]    The display screen may be mounted on a wall having a backside, and a camera may be mounted on the backside of the wall which pans as a viewer moves, thereby imaging a scene representative of the display being a virtual window through the wall. A camera with a zoom capability may be used for recording the scene. With the sensor being operative to detect the viewer&#39;s distance from the display screen, the processor is further operative to zoom in the camera as the viewer moves closer to the display screen, and zoom out the camera as the viewer moves away from the display screen. 
         [0010]    The sensor may be operative to detect the viewer&#39;s distance from the display screen, with the processor being operative to increase the resolution of the scene as the viewer moves closer to the display screen and decrease the resolution of the scene as the viewer moves away from the display screen. A camera with a tilt capability may be used for recording the scene, with the processor being further operative to tilt the camera in response to the viewer&#39;s up/down movement. The camera may have a field of view which includes a viewer of the display screen, enabling the display to function a virtual mirror. 
         [0011]    A plurality of cameras may be used for capturing the scene, with the processor being further operative to construct a three-dimensional image for display on the screen. A user control may be provided enabling a viewer to select a specific camera or cameras to see how others would view the user from different perspectives. 
         [0012]    The sensor may include an infrared CCD (charge-coupled device) camera. Regardless, the camera may have an image sensor, such that an image of a person in front of the display may be focused onto the image sensor as a spot or group of pixels, allowing the movements of the person to be tracked with no moving parts. 
         [0013]    The sensor may be a camera having at least a pan mount that tracks the movement of a viewer, with the processor being operative to shift the relative position of the foreground and background elements as a function of the tracking. The sensor may be a camera having at least a pan/tilt mount that tracks the movement of a viewer, with the processor being operative to shift the relative position of the foreground and background elements as a function of the tracking. The sensor may be a camera having an auto-focus capability. 
         [0014]    The sensor may be operative to sense a plurality of individuals in front of the display screen. The processor may be operative to shift the relative position of the foreground and background elements by favoring larger clusters of individuals as opposed to smaller clusters or single individuals. Alternatively, moving individuals may be favored over stationary individuals, or individuals actually looking at the display screen may be favored over those who are not. 
         [0015]    One or more transducers may be provided for producing sounds associated with the scene, with the processor operative to alter the reproduction of the sounds as a function of viewer movement. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0016]      FIG. 1A  shows a viewer having a field of view walking from the right towards the left relative to a display screen (shown as viewed from above); 
           [0017]      FIG. 1B  is a simplified representation of what the person might see on the display screen according to the invention from the position shown in  FIG. 1A ; 
           [0018]      FIG. 2A  shows the person moving to the left, closer to the center of the display screen; 
           [0019]      FIG. 2B  shows how closer objects have shifted laterally to the greatest degree, as opposed to more distant objects, much as a train passenger would experience while looking out the window of the train as the train moves; 
           [0020]      FIG. 3A  illustrates how the viewer has moved to the left-most portion of the display; 
           [0021]      FIG. 3B  shows how close objects have shifted to the extent that they now partially overlap with an object further away; 
           [0022]      FIG. 4A  depicts the image of a foreground object being gathered by a camera and recorded by a recorder; 
           [0023]      FIG. 4B  shows mid-range objects being recorded; 
           [0024]      FIG. 4C  shows distant or background objects are being recorded; 
           [0025]      FIG. 5  depicts an alternative technique for implementing perspective alteration according to the invention; 
           [0026]      FIG. 6A  shows how an infrared CCD (charge-coupled device) camera, preferably with a wide-angle lens, may be used as a sensor according to the invention; 
           [0027]      FIG. 6B  shows the use of a panning camera; 
           [0028]      FIG. 6C  depicts three persons generating a composite thermal field on an image sensor; 
           [0029]      FIG. 7A  illustrates a “virtual picture window” embodiment, wherein the movement of a viewer causes an outdoor camera to pan back/forth, thereby allowing the viewer to visualize the outdoor scene as if the display were a hole in the wall; 
           [0030]      FIG. 7B  shows how the invention enables virtual windows on insides walls, which may be useful in homes and businesses such as restaurants, bars and nightclubs; 
           [0031]      FIG. 7C  shows how the display can function as a “virtual mirror”; and 
           [0032]      FIG. 8  depicts how the virtual mirror embodiment of the invention may be used in bathrooms and dressing rooms. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0033]    This invention employs a variety of techniques to provide a unique experience for the viewer of a display, particularly large wall-mounted panels. In the preferred embodiments, the perception of a displayed image is altered for viewers moving relative to the position of the display system screen, thereby imparting a sense of three-dimensional immersion in the scene being displayed. 
         [0034]      FIG. 1A  shows a viewer  102  having a field of view  104  walking from the right towards the left relative to a display screen  106 . One or more sensors  108 , which may use visible-light, infrared, ultrasonic, or other modalities described in further detail below, are used to track at least the lateral position of individual  102 . 
         [0035]      FIG. 1B  is a simplified representation of what the person  102  might see on the display screen  106  according to the invention from the position shown in  FIG. 1A . Relatively close objects are shown at  115 ,  116 . Less close objects are seen at  114 . Somewhat distant objects are shown at  112 , and distant objects are shown at  110 . Although four relative distances are mentioned, the invention is not limited in this regard, and is applicable to more or fewer such relative distances. 
         [0036]    In  FIG. 2A , the person  102  has moved to the left, closer to the center of the screen  106 . The sensor(s)  108  have detected this movement and, in response, the perspective of the scene has been altered. As shown in  FIG. 2B , closer objects have shifted laterally to the greatest degree, followed by objects  114  and  112  in order, much as a train passenger would experience while looking out the window of the train as the train moves. Object  110 , being significantly distant, would shift little, if at all. 
         [0037]    This process continues in  FIG. 3A , where the viewer has moved to the left-most portion of the display. Close object  116  has shifted to the extent that it now partially overlaps with an object further away, and object  115  has begun to move off the screen  106 , as shown in  FIG. 3B . The ways in which the invention makes this possible will now be described in further detail. 
         [0038]    In the preferred embodiment, the invention employs a technique similar to that utilized in animation films: multiple superimposed graphics planes. In  FIG. 4A , the image of a foreground object  412  is gathered by camera  402  and recorded by recorder  410 . Depending upon the circumstances, a blue or other solid-color background  414  may be used for chroma-keying. 
         [0039]    In  FIG. 4B , mid range objects  420  are being recorded whereas, in  FIG. 4C , distant or background objects are being recorded. Commercially available software packages, or customized software tuned to specific program content, can be utilized to derive the desired material from among multiple subjects representing different focal points, and, once identified, can track the subjects as they change their position and even their orientation. In an alternative approach, multiple cameras, positioned to capture three-dimensional information, may be utilized to derive a three-dimensional array for each frame of motion, thereby allowing the producer to select “slices” which can be captured as graphics plane images for manipulation by the graphics processor of the instant invention. 
         [0040]    The camera(s) may record moving video images for the foreground graphics plane(s), the background graphics plane(s), or any combination thereof. For example, the background graphic plane may be based upon a still picture, while the foreground cameras record motion imagery. In this embodiment, position sensor(s)  108  detect the location of the viewer relative to the screen, and reposition the foreground graphics plane(s) as the viewer moves, thereby conveying to the viewer the impression, for example, that he is looking through a window at an outdoor scene, with, perhaps, a nearer image, such as a tree branch, that the viewer can see around by simply shifting his position relative to the display screen. The tree branch (or other object) may also be moving, as it would in a breeze, for example. 
         [0041]    The video source for these graphics planes may include prerecorded material supplied by playback from any recording devices. Other sources include broadcast, satellite, cable, or other programming sources, material delivered over broadband or other telecommunication links, privately recorded material, live video from cameras (including security and monitoring cameras), computer-generated graphics and the like, or any other source of image material. Graphics planes displaying text information may be superimposed over, or under, other graphics planes. 
         [0042]      FIG. 5  depicts an alternative technique for implementing perspective alteration according to the invention. Here a camera  502  having at least a pan mount  506  records an actual scene  512  at incremental angles suggested by arrow  508 . The view at each angle is recorded by unit  510  for later replay. The number of increments depends upon the desired resolution, room dimensions, and other factors. For example, at a very high resolution, single-degree increments may be recorded through a full 180 degrees. At a lesser resolution, single-degree increments may be recorded across a smaller angle of view, or larger-degree increments may be used at angle of view with or without interpolation to fill in any ‘gaps’ during replay. 
         [0043]    Thus, a variety of novel video applications are enabled by virtue of the invention, such as a simulated window which has a view that is not possible from that position if there were an actual window in that position (i.e., viewing a sunset from an eastern exposure), or a view which is not possible at all (i.e., a winter scene during the summer, or a scene from a different country). These represent a clear improvement over any of the typical “light box” with photographic transparency or other attempt to simulate a window view with conventional means. 
         [0044]      FIG. 7A  illustrates a “virtual picture window” embodiment, wherein the movement of viewer  102 , detected by sensors  108 , cause outdoor camera  704  on mount  706  to pan back/forth, thereby allowing the viewer  102  to “see” the outdoor scene  720  “through” the wall  702  on display  106  using graphics processing system  710 . Depending upon the movement of the viewer toward/away from the screen  106 , in this and other applicable embodiments the camera  704  may zoom as the viewer comes closer, pan when the user moves away and tilt if the viewer goes up/down, as might be the case on a staircase, for example. 
         [0045]    Through the use of distance detection, the display system can provide a three-dimensional effect, by applying modifications to an image as the viewer changes his position. Thus, for example, an image of a painting might capture the artist&#39;s intentions when viewed at a distance, while a close-up examination of the video display by the viewer could display the details of the brush strokes as the viewer changed his position relative to the screen. 
         [0046]    The invention is not limited to virtual windows through outside walls. As shown in  FIG. 7B , the invention enables virtual windows on insides walls, which may be useful in homes and businesses such as restaurants, bars and nightclubs. Similar to the embodiment depicted in  FIG. 7A , as user  102  moves relative to screen  106 , camera  704  pans (or tilts or zooms), enabling the user  102  to see people  730  or other objects through the wall  702 . 
         [0047]    In still another aspect, the display system can function as a “virtual mirror.” Such an embodiment is depicted in  FIG. 7C , wherein graphics processing system  710  directs the camera  704  to at least pan at an angle “A” substantially equal to angle “B” formed by the location of the user and line  722  perpendicular to the plane of display  106 . This allows the viewer to see a synthesized reflection depicted by broken line  720 , typically including the subject him/herself. 
         [0048]    The mirror embodiment of the invention may be used in bathrooms and dressing rooms, as illustrated in  FIG. 8 . Multiple cameras  802 ,  804 ,  806 ,  808  disposed around the display screen  106  allow the processor unit  812  to construct a three-dimensional image for display on the screen, and the proximity sensing devices may be utilized to create an on-screen image which is representative of what a person should see as he re-positions himself, in three dimensions, about the display screen. It is a simple matter to provide a left-to-right reversed-image, in keeping with the mirror aspect of the invention. A user control  810  allows the user to select a specific camera or cameras to see how others would view them from different perspectives. 
         [0049]    In the dressing room/bathroom embodiments of the invention, the camera(s) capturing the image of the viewer may be placed behind a semi-transparent screen allowing better visualization such as eye-level contact to be maintained. One or more video cameras may be provided on a bendable tether—or wireless hookup—enabling a user to view hard-to-reach places such as cars, nose, mouth etc. Variable degrees of magnification may be provided, based upon detected distance from a surface being viewed, for example. 
         [0050]    In terms of position sensing, the preferred embodiment uses an infrared CCD (charge-coupled device) camera, preferably with a wide-angle lens  206  shown in  FIG. 6A . The user  202  generates a thermal image  204 , which is focused onto camera array  210  as a spot or group of pixels, allowing the system  220  to know where a person is with no moving parts. Processor  106  can then cause the perspective, depth perception, or other characteristics to change accordingly. If no tilt or zoom functions are provided, a linear sensor may be substituted for a 2D sensor. 
         [0051]    As an alternative to a fixed camera with sufficiently wide-angle lens, a panning camera may be used, as shown in  FIG. 6B . Here the camera  230  is trained on a subject and, as that subject moves positional information is sensed by pan mount  234  and communicated to alter screen  106  through electronics unit  240 . Again, if tilt/zoom functions are provided, the camera  230  may use tilt and/or auto focus to determine other positional aspects of the viewer. 
         [0052]    The invention may handle multiple viewers in different ways. These solutions include (1) favoring clusters of potential viewers over singular viewers; (2) favoring moving viewers over stationary viewers; and (3) favoring viewers actually looking at the screen over those looking away. One advantage of the sensor system of  FIG. 6A  is that clustering is naturally accommodated. In  FIG. 6C , three persons  242 ,  244 ,  246  generate a composite thermal field  240 , which generates a relatively large imprint  251  on sensor  210 . In contrast, the narrower thermal field  250  of single person  248  results in a smaller spot  241  on array  210  such that, in this embodiment of the invention, the perspective of persons in the group would be favored. 
         [0053]    The approach of  FIG. 6A  also naturally addresses the favoring of moving viewers over stationary viewers. Referring again to  FIG. 6C , if the group consisting of persons  242 ,  244 ,  246  were stationary, it would be a straightforward processing task to detect that an individual is moving. In this embodiment of the invention, the moving individual  248  would be preferred over the stationary group. 
         [0054]    In all embodiments of the invention, sensing in the visible region of the spectrum may be used instead of—or in concert with—IR sensing. This presents advantages and disadvantages. In terms of advantages, visible light sensing may allow a single camera (or cameras) to detect the image and position of the viewer (as in the mirror embodiments). Another advantage is that recognition techniques may be used to determine if a particular person is actually looking at the display in which case that person may be favored over individuals looking away. If multiple persons are looking at the display, other techniques such as clustering and motion favoring may also be used. Perhaps the only disadvantage is that image recognition and other operations require additional processing power, however, that is easily accommodated with modern processors. 
         [0055]    In each case, specialized graphics processing provides the management of the graphics planes and any audio material, while processing rules (for example—“take image modification instructions from the position of the closest viewer only”) ensure that the system will not be misdirected by movement of viewers that are on the opposite side of the room. Where multiple display systems are in use (as for simulating an array of “structural” windows), an overall system for management of the displays is utilized, thus providing an integrated, coordinated system of imaging displays. For example—an overall image, larger than the entire display system, may be utilized, or alternative schemes, in which image planes or other data may “flow” from one display screen to another display screen next to it. 
         [0056]    While described herein with reference to flat-panel displays (LED, LCD, Plasma, etc.) the principles disclosed may be applied with suitable results to any number of display technologies currently available or in development today (CRT-type, front or rear projection, Electroluminescence, OLEDs, etc.). Furthermore, adjustments may be applied to the image data to correct for any geometric distortions introduced due to the position of the camera(s) or display unit(s). In addition, alternative embodiments may utilize additional graphics planes to enhance the effect of the display. 
         [0057]    In all cases, audio may be included, representing material that may or may not be related to the video images presented on the screen. Thus, the system can serve the function of an enhanced video display terminal, a television viewing screen, a security monitoring system, a video entertainment system, or any other system for which display of graphics material is of value to the viewer. 
         [0058]    With further regard to audio, if the system is provided with stereo or surround sound, the sound reproduction may be altered as a function with user position, with or without a change in visual perspective. For example, as an individual walks past the display screen the sound of elements in the scene (i.e., birds, vehicles, etc.) may be varied whether or not the individual is looking at the screen. If the user moves toward the screen, sounds may be enhanced or attenuated. For example, if a viewer moves toward a frog or a bird in the scene, the sounds of that creature may be enhanced, or diminished as the user moves away.