Abstract:
A system for displaying a wide field of view video image of a location. A series of location cameras disposed at the location captures the wide filed of view video image as a series of individual video images covering the desired field of view. A distance sensor unit senses distances of closest objects in one or more overlap areas between field of views of the location cameras from the two or more location cameras covering each respective overlap area. A display unit displays the series of individual video images to a user for creating a visual experience of the location. A processor unit determines a horizontal span of each individual video image displayed by the display unit based on the sensed distances of the closest objects.

Description:
FIELD OF THE PRESENT INVENTION  
       [0001]     The present invention relates to wide field imaging.  
       BACKGROUND OF THE PRESENT INVENTION  
       [0002]     Systems that seek to acquire very wide field of view (e.g. 360°) images are significant for a number of applications in both still and motion picture capturing and display. One such system employs a camera rotating around the horizontal of its focal plane used for capturing still panoramic photographs. Many images are taken, ranging from of the order of about 7 for consumer photographs to more than 100 for professional panoramas. The camera is typically mounted on a tripod, with the pan axis centered on the horizontal center of the focal plane. Parallax errors are reduced as the number of images is increased.  
         [0003]     Sophisticated software has been recently developed that allows a consumer-quality panorama to be built from a series of handheld images. However, such techniques are based on panoramic still images, and cannot produce video panoramas since the product of exposure time and the number of images that must be taken exceeds the time per frame of a typical video sequence.  
         [0004]     In other systems, a single camera coupled with a domed, spherical, or toroidal section mirror is used. The camera is usually mounted in such systems above the mirror so that the camera can see a e.g. 360° surround band around the mirror. The mirror may be placed on a conference room table to provide a view of everyone sitting around a meeting table or may be placed on a tripod for panoramic landscape pictures.  
         [0005]     In yet other systems, multiple mirrors and cameras are utilized to acquire the panoramic views. One such system uses 6 cameras looking down on mirrors arranged in a ring. The mirrors for this system consist of a 6-side pyramid, with different cameras looking down on each of the mirrors. The ring of cameras is supported by a post at the center of the pyramid of mirrors.  
       SUMMARY OF THE PRESENT INVENTION  
       [0006]     In one aspect of the present invention provides a system for displaying a wide field of view video image of a location. The system comprises a plurality of location cameras for placement at the location. The location cameras capture the wide field of view video image as a plurality of individual video images that together cover the desired field of view. The system also comprises a distance sensor unit. The distance sensor unit senses distances of closest objects in one or more overlap areas between field of views of the neighboring location cameras. The system further comprises a display unit for displaying the plurality of individual video images to a user for creating a visual experience of the location based on the sensed distances to the closest object. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0007]      FIG. 1A  is a schematic drawing illustrating a front view of a surrogate device in an embodiment of the present invention;  
         [0008]      FIG. 1B  is a schematic drawing illustrating a top view of the surrogate device of  FIG. 1A ;  
         [0009]      FIG. 1C  is a schematic drawing illustrating a detail of  FIG. 1B  for one object configuration;  
         [0010]      FIG. 1D  is a schematic drawing illustrating a detail of  FIG. 1B  for another object configuration;  
         [0011]      FIG. 2 . is a schematic drawing illustrating a display system in an embodiment of the present invention;  
         [0012]      FIG. 3  is a schematic drawing illustrating a parallax artifact;  
         [0013]      FIG. 4  is a schematic drawing illustrating a parallax artifact;  
         [0014]      FIG. 5  is a flowchart illustrating a method of displaying a wide field of view video image in an embodiment of the present invention. 
     
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS  
       [0015]     The system of the example embodiment consists of two main parts, a display system at the user&#39;s location and a remotely located surrogate device.  
         [0016]     The surrogate device is connected to the display system via high-speed networking in the example embodiment, and stands in the place of the user at the remote location.  
         [0017]      FIG. 1A  shows a schematic drawing of the surrogate device  100  of the example embodiment. The surrogate device  100  has a “head”  102  made from four display panels, e.g.  104  in a square, outwardly facing arrangement. Live video of the user&#39;s face  106  is displayed on all four sides, namely on all four LCD panels of the surrogate device  100 .  
         [0018]     Four cameras e.g.  108 ,  110  in the corners of the surrogate devices head  102  together capture 360° surround live video from the remote location at which the surrogate device  100  is located. The cameras e.g.  108 ,  110  are positioned near the mid-point of the height of the displays e.g.  104  in the example embodiment, to achieve a proximity to the eye-level of the displayed user face  106 .  
         [0019]     Four directional microphones e.g.  112 ,  114  are also provided in the corners of the surrogate devices head  102  for capturing the remote sound field. The surrogate device head  102  also comprises speakers e.g.  116 ,  118  in the corners of the surrogate device&#39;s head  102  for transmitting a sound field from the user&#39;s location at the remote location.  
         [0020]     The surrogate device  100  further comprises an internal computer unit  120  for connecting to the cameras and microphones, processing data, and sending it to the display system. The computer unit  120  also receives live video of the user&#39;s head  106  and displays it on the four display screens e.g.  104 . The surrogate device also receives audio signals from the user, and then processes and outputs the signals to the speakers e.g.  116 ,  118 .  
         [0021]     The video data from the four cameras, e.g.  108 ,  110  is compressed and transmitted over a high-speed network (not shown) to computers at the location of the display system. An antenna  121  is utilized in the example embodiment for wireless connection to the network.  
         [0022]      FIG. 1B  shows a schematic drawing of a top view of the surrogate device  100 . This drawing illustrates how the four cameras, e.g.  108 ,  110  together capture a 360° field of view around the surrogate devices head  102 . Objects are located around the surrogate  100 , including objects  124  to  127  at different distances in overlap regions  128  to  131  of field of views of adjacent cameras.  
         [0023]      FIG. 2  is a schematic drawing of one example of a display system  200  that can be used in connection with the surrogate device  100 . The video from the surrogate device  100  (see  FIG. 1A ) is projected on the walls/screens e.g.  202  of a display cube  204  by four projectors e.g.  206 . The projectors e.g.  206  are housed in “hush boxes” e.g.  208  to make them virtually inaudible in the example embodiment. Speakers e.g.  210  are mounted above and below each projection screen e.g.  202  of the display cube  204 .  
         [0024]     The hush boxes e.g.  208  in the example embodiment are built using sound isolation techniques familiar to a person skilled in the art. Each box e.g.  208  has a double-pane window  212  for the projector light beam to shine out of the box  208  and onto the screen  202 . Each box e.g.  208  also has measures for adequately cooling the projectors  206 .  
         [0025]     Computers  214 ,  215  are placed in an adjacent room  216  for sound isolation purposes, and drive the projectors e.g.  206  and the speakers e.g.  210  with video and audio information transmitted from the surrogate device  100  (see  FIG. 1 ) at the remote location via network  217 . The network  217  of the present embodiment includes an antenna  218  to enable wireless communication with the surrogate  100 , via its respective antenna  121 .  
         [0026]     Anechoic foams are placed on the walls of the room  218  in the example embodiment, for eliminating local reflections. Stray reflection of the light is reduced and the contrast of the display screens e.g.  202  is increased by using black anechoic foams.  
         [0027]     The user  220  is located inside the display cube  204 , which includes a suitable entrance arrangement (not shown) in at least one of the walls e.g.  202 . The images on the projection screens e.g.  202  are presented in life size in the example embodiment. This means that the angle sub-tended by objects on the screen is about the same angle as if the user  220  was actually at the remote location viewing the objects themselves. It will be appreciated by a person skilled in the art that the reproduction is not exact unless the user&#39;s head is centered in the display cube  204 . Life size presentation of both local and remote participants in the example embodiment is advantageous for preserving the gaze of the participants.  
         [0028]     Cameras e.g.  222  are provided in each corner of the display cube  204  which take video images of the user  220  for display on the display panels e.g.  104  of the surrogate device  100  (see  FIG. 1A ). The user  220  wears a wireless lapel microphone (not shown). A receiver  224  connected to one of the computers  214  receives the audio data from the lapel microphone in the example embodiment. The video and audio data from the cameras e.g.  222  and the wireless microphone is transmitted over the high-speed network  217  via a wireless communications channel to the surrogate device  100  (see  FIG. 1A ).  
         [0029]     In the following, it will be described how parallax artifacts are automatically corrected in the example embodiment. Because the cameras e.g.  108 ,  110  in the surrogate device&#39;s head  102  (see  FIG. 1 ) cannot have a common optical center at the same level as the display of the user&#39;s eyes on the display panels e.g.  104  (see  FIG. 1A ), parallax results. This parallax causes visibility gaps and/or duplication of imagery between adjacent images. For example, consider a geometry  300  as illustrated in  FIG. 3 , where the cameras  302 ,  304  have a 90-degree field of view. If a person  306  at the remote location stands between the two cameras  302 ,  304 , the user cannot see them.  
         [0030]     Consider instead a geometry  400  shown in  FIG. 4 , where the field of view of the cameras  402 ,  404  is widened to 143°. Here, the user will see two copies of a person  406  standing in the overlap region  408  of the field of views of the cameras  402 ,  404 , one projected on each of adjacent display wall at the user&#39;s location. While this is an improvement over not seeing the person at all, it is still less than desirable in most situations. For example, if the two copies of the person are presented, this introduces errors in preserving the gaze of both the user and the remote person.  
         [0031]     It has been recognized by the inventors that if an object at the remote location is located at exactly the point where the outermost rays captured by the cameras intersect, the user will see only one copy of that object, projected in the area where adjacent projection screens come together.  
         [0032]     Thus, partial automatic correction of parallax artifacts can be enabled in an example embodiment of the present invention, by adjusting the horizontal span of the displayed images from each camera depending on the nearest distance to an object at the edges of each camera&#39;s field of view. One copy of the closest object or person is projected for the user independent of the distance of the closest object or person from the surrogate device in the example embodiment.  
         [0033]     Returning now to  FIG. 1B , this is accomplished in the example embodiment by measuring the distance from the surrogate device  100  to objects e.g.  124  in the overlap area  128  of the vertical edges of the fields of view of cameras  108 ,  110  and modifying the horizontal field of view of the projected images so that they intersect at the distance of the closest object in that overlap area  128 .  
         [0034]     It will be appreciated that the distance to objects in the overlap area of two adjacent cameras can be acquired using a number of methods, one of which will be discussed below for the example embodiment.  
         [0035]     The example embodiment utilizes infrared sensors e.g.  132  of a type that sends out a narrow beam  134  and based on the distance of the object it reflects from, the beam illuminates different pixels in a linear array of photocells (not shown). The corresponding distance can be read out over a serial bus by the computer unit  120  (see  FIG. 1A ).  
         [0036]     A pair of infrared sensors e.g.  132 ,  136  is used in the example embodiment for each camera e.g.  108 , which send out narrow beams  134 ,  138  along respective field of view boundary lines  140 ,  142 . The beams  134 ,  138  are directed downwardly to substantially follow the bottom left and right corners of the extending field of view of the camera  108 .  
         [0037]     Four additional infrared sensors e.g.  144  are provided in the example embodiment and are positioned centrally below the displays e.g.  104  of the surrogate device  100  (see  FIG. 1A ). Each of the four additional sensors e.g.  144  sends out a narrow beam  146  which extends horizontally and centrally through the overlap area  128  of two cameras  108 ,  110 . The distance to the nearest object is computed as the minimum distance reported by all the sensors monitoring the area between a pair of cameras.  
         [0038]     Once the distance to the nearest object in the overlap area is known, the horizontal field of view of a display of images from the cameras adjacent to the overlap area can be modified.  FIGS. 1C and 1D  illustrate two different scenarios for adjusting the displayed horizontal field of view depending on closest objects in the overlap region  131  between adjacent cameras  148 ,  150 .  
         [0039]     Turning initially to  FIG. 1C , the closest object  126  in that scenario is located on the left side of the overlap region  131 . In this scenario, vertical edges  152 ,  154  of the displayed images from cameras  148  and  150  respectively are chosen such that they intersect at point  156  at the measured object distance on a line extending centrally through the overlap region  131 , which coincides with the narrow beam  158  from one of the distance sensors. It will be appreciated that thus, the object  126  will be visible substantially only on the image originating from camera  148 , but not on the image originating from camera  150 .  
         [0040]     In the scenario shown in  FIG. 1D , the object  126  is centrally located within the overlap region  131 . Here, the vertical edges  152   b  and  154   b  of the displayed images originating from cameras  148  and  150  respectively are chosen to intersect at point  156   b.  In this geometry, the shortest distance to the object  126  is detected by the narrow beam  156  of the central sensor, which can be used directly to determine the intersection point  156   b.    
         [0041]     It will be appreciated that respective “halves” of the object  126  will be visible in the images from cameras  148  and  150  respectively, thus avoiding undesired duplication of the object  126  in the displayed wide field of view video image.  
         [0042]     There are four cameras and a 360-degree surround view is synthesized in the example embodiment. The vertical field of view displayed should therefore preferably be the same in all four images. If the vertical field of view were not the same throughout, there would be discontinuities where parts of an object appeared taller or shorter than other parts.  
         [0043]     Having a uniform vertical field of view displayed means that any changes to the horizontal field of view displayed must be made without affecting the vertical field of view—i.e., anamorphically. Also, the distance to the nearest object in each of the four overlap areas  128  to  131  around the surrogate device  100  will usually be different from each other. The modification of each projected image is preferably a function of the nearest distance in both of its adjacent overlap areas.  
         [0044]     As distances to the closest objects change during the capture of the individual video images, the horizontal scaling may be continually adjusted in real-time, if desired. It will be appreciated that a number of different approaches to modifying the horizontal scale can be employed. Once approach will be described below for the example embodiment. The pixel column of the desired left and right edges of the imagery on the screen is calculated, and a single horizontal scale is computed. In this approach scale transition artifacts would be limited to the seam between screens, which already has a number of artifacts due to the screen discontinuity. Any artifacts from a transition in a horizontal scale should be less evident there. The desired left and right edges of the projected image can be found from trigonometry.  
         [0045]      FIG. 5  shows a flowchart  500  illustrating a method for displaying a wide field of view video image embodying the present invention. At step  502 , a plurality of cameras  504 ,  506  are used to take individual video images  505 ,  507  of a location  508 , together covering a desired wide field of view.  
         [0046]     In step  510 , distances of the nearest object  512  at the overlap area the adjacent edges of the individual field of views of the cameras  504 ,  506  to each of the cameras  504 ,  506  is measured.  
         [0047]     In step  514 , the distance information is utilized to anamorphically adjust the horizontal span of the individual displayed images  516 ,  518  to form a composite wide field of view image  520 . In the composite image  520 , one half of the object  508  is displayed in each partial image  516 ,  518  respectively.  
         [0048]     It will be appreciated by the person skilled in the art that numerous modifications and/or variations may be made to the present invention as shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. The present embodiments are, therefore, to be considered in all respects to be illustrative and not restrictive.  
         [0049]     For example, while infrared sensors have been used in the described embodiment to sense the distances of the nearest objects, it will be appreciated that the present invention can be implemented utilizing a variety of different sensing techniques, including e.g. ultrasonic sensors or stereo vision technique sensor implementations.  
         [0050]     Furthermore, while the adjusting of the horizontal span of the displayed images in the example embodiment includes computing a single horizontal scale, in different embodiments the horizontal scale could be modified by e.g. dividing the screen in half and modifying the horizontal scaling of each half screen. In the region between the half screens, the horizontal scale could e.g. be gradually changed, to reduce scale transition artifacts at the seam between the half screens.  
         [0051]     It will also be appreciated that the present invention is not limited to the use of four cameras to capture the desired wide field of view video image. Rather, the number of cameras can be chosen to suit a particular desired coverage of the wide field of view, both horizontally and vertically, and also in terms of depth coverage between adjacent field of views of the individual cameras.  
         [0052]     Similarly, the number and configuration of distance sensor elements can be chosen in different embodiments to achieve a desired coverage and depth resolution.