Patent Publication Number: US-7593042-B2

Title: Maintenance of panoramic camera orientation

Description:
CROSS-REFERENCE(S) TO RELATED APPLICATION(S) 
   This application is a continuation-in-part of U.S. patent application Ser. No. 10/902,675, entitled “Omni-Directional Camera With Calibration And Up Look Angle Improvements”, filed Jul. 28, 2004 by the present inventor and assigned to Microsoft Corp., the assignee of the present application. Priority is claimed to said application which is hereby incorporated by reference for all that it teaches and discloses. 

   TECHNICAL FIELD 
   The following description relates generally to image processing. More particularly, the following description relates to maintaining orientation of a panoramic camera system. 
   BACKGROUND 
   Panoramic video camera devices are configured to image wide angles of up to three hundred and sixty degrees (360°). These camera devices are especially useful in a conference room scenario, where meeting participants can be imaged no matter where they are located in the conference room. Special adjustments are made to provide a better image in certain conference room configurations, such as where meeting participants are seated around a rectangular table. If a camera device is moved after adjustments are made, the adjustments are rendered incorrect and the image becomes distorted. 

   
     BRIEF DESCRIPTION OF THE DRAWINGS 
     The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein: 
       FIG. 1  is a diagram of a prior art conference room video configuration. 
       FIG. 2  is a depiction of an exemplary cylindrical panoramic image 
       FIG. 3  is a depiction of a non-cylindrical panoramic image 
       FIG. 4  is a block diagram of an exemplary panoramic camera device. 
       FIG. 5  is a block diagram of an exemplary panoramic camera configuration. 
       FIG. 6  is an exemplary process diagram for maintaining orientation in a panoramic camera. 
   

   DETAILED DESCRIPTION 
   The present description describes one or more techniques and devices that maintain orientation of a panoramic camera device. Panoramic camera devices can provide cylindrical panoramic images or non-cylindrical panoramic images. A cylindrical panoramic image is an image that has uniform resolution across a span of the panorama. A non-cylindrical panoramic image is a panoramic image that has varying degrees of resolution across the span of the panorama. 
   When a panoramic camera device is configured to record a particular non-cylindrical configuration, a resultant image is distorted if the panoramic camera is moved from its initial orientation. In the following description, devices and techniques are described that record an initial orientation of a panoramic camera. If the panoramic camera is moved, the adjusted orientation of the camera is determined and the panoramic camera is configured to adjust a rendering of a panoramic image to account for the change in orientation. 
   Exemplary Configuration 
     FIG. 1  is a diagram of a prior art conference room video configuration  100 . The conference room video configuration includes a conference table  102  and a panoramic camera  104  situated atop the conference table  102 . The panoramic camera  104  may be configured to record a non-cylindrical panoramic image. 
   Exemplary Cylindrical Panoramic Image 
     FIG. 2  is a depiction of an exemplary cylindrical panoramic image  200 . The cylindrical panoramic image  200  is taken from a panoramic camera device located in the center of a rectangular table that is ten feet long and five feet wide (10′×5′). Since resolution is the same across the cylindrical panoramic image  200 , an image of a face  202  of a person seated at the short side of the table (i.e. two and one-half feet (2.5°) from the camera) appears about twice as large as an image of a face  204  of a person seated at a long end of the table (i.e. five feet (5′) from the camera). 
   Exemplary Non-Cylindrical Panoramic Image 
     FIG. 3  is a depiction of a non-cylindrical panoramic image  300 . Like the image shown in  FIG. 2 , the non-cylindrical panoramic image  300  is taken from a panoramic camera device located in the center of a rectangular table that is ten feet long and five feet wide (10′×5′). In generating the non-cylindrical panoramic image  300 , the panoramic camera normalizes the size of face images  302  so that they appear about equal. 
   Cylindrical panoramic cameras are radially symmetric and, therefore, the orientation of the camera with respect to a table on which is sits is not taken into account in processing a panoramic image. However, non-cylindrical panoramic cameras must be able to determine the orientation of the panoramic camera with respect to the table so that the different fields of resolution taken by the panoramic camera can be processed correctly. 
   For example, the non-cylindrical panoramic image  300  is oriented so that resolution is increased for far ends of the table. If a camera taking such an image was rotated ninety degrees (90°), then resolution of the near sides of the table is increased and resolution is decreased for far ends of the table. As a result, faces on the short sides of the table would appear approximately four times larger than faces on the long ends of the table. 
   However, if the camera was configured to track the ninety degree (90°) rotation and account for the rotation in processing the panoramic image, then the normal appearance of the panoramic image would be maintained. Such a technique is described herein. 
   Exemplary Panoramic Camera Device 
     FIG. 4  is a block diagram of an exemplary panoramic camera device  400  in accordance with the present description. The exemplary panoramic camera device  400  is shown having certain elements and particular functionality is described and allocated among the certain elements. It is noted, however, that the exemplary panoramic camera device  400  may include more or fewer elements than shown and that functionality attributed to a particular element may be allocated to one or more other elements in one or more other configurations. Also, any element and/or function shown and described with respect to  FIG. 4  may be implemented in hardware, software, firmware or a combination thereof regardless of how such an element and/or function is shown and described herein. 
   The exemplary panoramic camera device  400  includes a processor  402  and memory  404 . One or more individual cameras  406  are also included in the exemplary panoramic camera device  400 . The one or more individual cameras  406  are configured to image a wide-angle area (i.e. a panoramic area) if up to three hundred and sixty degrees (360°). In the following discussion, a configuration of multiple individual cameras  406  is assumed, each of the individual cameras  406  being configured to image a portion of a panoramic area. Resultant images  408  from the individual cameras  406  are stored in the memory  404  and are stitched together by an image stitching module  410  to produce a single panoramic image  412 . The images may also be processed by a warping module  414  that applies a warping function to the panoramic image  412  that minimizes distortion of the panoramic image  412 . 
   Image stitching and warping as applied to panoramic images is described in U.S. patent application Ser. No. 10/262,292 filed Sep. 30, 2003, entitled “Foveated Wide-Angle Systems and Method for Capturing and Viewing Wide-Angle Images in Real Time,” by Zicheng Liu and Michael Cohen. Said application is assigned to Microsoft Corp. and is incorporated herein by reference for all that it discloses and teaches. 
   The exemplary panoramic camera device  400  also includes an input/output (I/O) module  416  for communicating with one or more remote devices (not shown). A user interface  418 , a microphone  420  and a speaker  422  are also included in the exemplary panoramic camera device  400  to send and receive information to and from one or more users. The user interface  418  includes several interface devices (not shown) such as buttons, slider, display screens, touch screens, light emitting diodes (LEDs), and the like. 
   The user interface  418  also includes a table size selector  424  and an orientation initialization unit  426 . The table size selector  424  is used to identify multiple table shapes and/or sizes, such as round, small rectangular (5′×10′), large rectangular (16′×5′), etc. A user selects a table size with the table size selector  424  to correspond to a table with which the exemplary panoramic camera device  400  is used. If “round” is a table shape option, it is noted that a cylindrical panorama process may be used and any computations described below for conversion to a non-cylindrical panorama may be omitted. 
   The table size selector  424  may also indicate a nominal orientation for each table configuration. For example, the table size selector  424  can include a diagram indicating each table size and include an identifier in the diagram that indicates how the exemplary panoramic camera device  400  should be initially oriented with respect to the table. For a rectangular table, the table size selector  424  may indicate that a front (not shown) of the camera should face a far end of the table. 
   The exemplary panoramic camera device  400  includes a magnetometer  427  that can detect and orientation of the exemplary panoramic camera device  400  with respect to the earth&#39;s magnetic field. Although the exemplary panoramic camera device  400  is shown having a magnetometer  427 , it is noted that any device may be used that can be configured to detect a relative orientation of the exemplary panoramic camera device  400 . 
   The memory  404  of the exemplary panoramic camera device  400  stores an operating system  428  that controls general functionality required to operate the exemplary panoramic camera device  400  and its components. A remapping table  430  and an orientation module  432  are also stored in the memory  404 . The remapping table  430  contains values that are used to remap the individual images  408  to the panoramic image  412 . 
   Remapping tables and functionality are described in U.S. patent application Ser. No. 10/177,315 entitled “A System and Method for Camera Color Calibration and Image Stitching”, filed Jun. 21, 2002 by the present inventor and assigned to Microsoft Corp., the assignee of the present application. Said application which is hereby incorporated by reference for all that it teaches and discloses. 
   The orientation module  432  includes an initial heading  434 , a detected heading  436  and a rotation module  438 . The initial heading  434  is a magnetometer  427  reading that is obtained and stored when the orientation initialization unit  426  is actuated. When a user sets up the exemplary panoramic camera device  400 , the user orients the camera appropriately and actuates the orientation initialization unit  426 . A present heading of the exemplary panoramic camera device  400  is taken and stored at that time. 
   The detected heading  436  is a magnetometer  427  reading at any given time. The rotation module  438  is configured to determine a difference between the initial heading  434  and the detected heading  436  and to apply appropriate corrections to the values contained in the remapping table  430 . The corrections may be applied in real time as the individual images  408  are remapped to the panoramic image  412  or the corrections may be applied to the values in the remapping table  430  to derive new values which are then stored in the remapping table. 
   The rotation module  438  may also be configured to apply corrections only when a threshold change in orientation has been detected. For example, if the exemplary panoramic camera device  400  is moved only say a degree or two, correction may not be desired. If the exemplary panoramic camera device  400  is rotated a significant amount—for example, ten degrees or more—then the rotation module  438  may detect the magnitude of the rotation and determine that corrections should be applied to the remapping table  430 . 
   The orientation maintenance process is described in greater detail below, with respect to the process diagram of  FIG. 6 . 
     FIG. 5  is a block diagram of an alternative exemplary panoramic camera configuration  500 . The exemplary configuration  500  shows a first location (Location A  502 ) and a second location (Location B  504 ). Location A  502  includes a panoramic camera  506  with a magnetometer  507  and a client device  508 . Unlike the exemplary panoramic camera device  400  shown in  FIG. 4 , certain elements and functionality shown and described in  FIG. 4  are included in and performed by the client device  508 . The panoramic camera  506  functions to image Location A  502  and the client device  508  is configured to handle the data processing associated with the techniques described herein. 
   Location B  504  also includes a panoramic camera  510  with a magnetometer  511  and a client device  512 . The client device  506  of Location A  502  and the client device  512  of Location B  504  communicate with each other over a communication line  514 , such as a telephone line or a network such as the Internet or a local area network (LAN). 
   Although some elements and functionality of the panoramic camera  400  shown in  FIG. 4  may be located and performed in the client devices  508 ,  512  of  FIG. 5 , it is noted that the magnetometer  427  must be included in a portion of the panoramic camera that receives object images for processing. 
   Exemplary Process Diagram 
     FIG. 6  is depicts an exemplary process diagram  600  for maintaining orientation in a panoramic camera. In the following description of the exemplary process diagram  600 , continuing reference is made to elements and reference numerals shown in previous figures. 
   At block  602 , the magnetometer  427  provides the detected heading (B)  436  to the rotation module  438  and the rotation module  438  retrieves the initial heading (A)  434  (block  608 ). At block  606 , the rotation module  438  determines if and how much the panoramic camera has been rotated since it was initially set up. If a rotation has occurred, the rotation module  438  applies a remapping function (T c ) to values in the remapping table  430  (block  604 ). 
   T c  is an image remapping function that maps the images from the individual cameras to the cylindrical panoramic image. If the magnetometer reading is B, then T c  can be rotated by (A−B) degrees to correct for changes in the panoramic camera orientation, yielding T′ c . 
   A non-cylindrical warping function (warping module  414 ) is applied to T′ c  at block  610 , yielding a non-cylindrical warping function T N . The image stitching module  410  stitches the individual images  408  from the cameras  408  (block  614 ) according to the corrected values at block  612 . The panoramic image  412  is created at block  616 . 
   By applying the orientation maintenance techniques described above, a panoramic image will remain in the same orientation even after a panoramic camera creating the panoramic image is rotated. The image rotation process shifts the panoramic image so that the margins of the panoramic image remain the same. For example, if a viewer is viewing the panoramic image and the panoramic camera is rotated ninety degrees, the viewer will not notice a difference since the image from the rotated camera will be rotated in an amount to offset the rotation of the camera. 
   Conclusion 
   While one or more exemplary implementations have been illustrated and described, it will be appreciated that various changes can be made therein without departing from the spirit and scope of the claims appended hereto.