Abstract:
An assembly includes a pair of image capture devices that capture 360-degree, stereo cubemap representation images of a scene. A controller generates a representation of the scene by correcting errors caused by placement of the image capture devices relative to each other in the assembly. The controller rotates an image from the image capture device to align objects in the image with objects in an image from the additional image capture device. Additionally, the controller replaces portions of an image from the image capture device including the additional image capture device with portions of an image from the additional image capture device. Additionally, the controller uses optical flow to cancel horizontal disparity and vertical disparity between images captured by the image capture device and by the additional image capture device.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims priority to U.S. Provisional Patent Application No. 62/357,918, filed on Jul. 1, 2016, which is incorporated by reference herein in its entirety. 
     
    
     BACKGROUND 
       [0002]    This disclosure generally relates to virtual reality systems, and more specifically to capturing full-stereo 360-degree images for presentation by a virtual reality system. 
         [0003]    An increasing number of devices allow viewing of full-spherical 360 by 180-degree content, such as virtual reality content. However, it remains difficult to capture full stereoscopic (“stereo”) 360 by 180-degree video for subsequent presentation. Existing and announced devices and techniques for capturing full stereo 360-degree video have significant hardware costs and processing time, making capture of full stereo video impractical for many applications. 
       SUMMARY 
       [0004]    An assembly includes a pair of image capture devices each configured to capture full spherical images, represented as a 360-degree, stereo cubemap representation of a scene surrounding the image capture devices. Each of the image capture devices is coupled to a controller that receives images captured by each of the image capture devices. The controller generates the representation of the scene by correcting errors from placement of the image capture devices relative to each other in the assembly. The controller rotates an image from an image capture device to align objects in the image with objects in an image from the other image capture device of the pair. Additionally, the controller also identifies portions of an image of the scene from an image capture device occluded by the other image capture device of the pair, and replaces the identified portions with portions of an image of the scene captured by the other image capture device. In various embodiments, the controller uses optical flow to cancel horizontal disparity and vertical disparity between images captured by each of the image capture devices. Alternatively, images captured by each of the image capture devices are transformed into a transverse equirect format or into a cubemap representation to cancel horizontal and vertical disparity between images captured by different image capture devices. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  is an example system for capturing stereo images, in accordance with an embodiment. 
           [0006]      FIG. 2  shows an example of misalignment between the image capture device and the additional image capture device, in accordance with an embodiment. 
           [0007]      FIG. 3  an example imaging geometry for points along a circle on a ground plane, in accordance with an embodiment. 
           [0008]      FIG. 4  shows changes in horizontal disparity and changes in vertical disparity along a plane at a specific downward angle from a horizontal plane including the image capture device and the additional image capture device, in accordance with an embodiment. 
           [0009]      FIG. 5  shows a range of horizontal azimuth angles where an image captured by an image capture device includes an additional image capture device, in accordance with an embodiment. 
           [0010]      FIG. 6  is a conceptual diagram of correcting horizontal disparity from an image captured by the image capture device to generate a modified image, in accordance with an embodiment. 
       
    
    
       [0011]    The figures depict various embodiments of the present invention for purposes of illustration only. One skilled in the art will readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles of the invention described herein. 
       DETAILED DESCRIPTION 
     System Architecture 
       [0012]      FIG. 1  shows an example system  100  comprising an image capture device  110  and an additional image capture device  120 . In various embodiments, the image capture device  110  and the additional image capture device  120  are cameras, video cameras, or other devices configured to capture image data. While  FIG. 1  shows a system  100  including two image capture devices, in other embodiments, the system  100  may include a greater number of image capture devices (e.g., 4 image capture devices, 8 image capture devices, etc.). 
         [0013]    In the embodiment shown by  FIG. 1 , the image capture device  110  and the additional image capture device  120  are each coupled to a connecting member  130  so the image capture device  110  and the additional image capture device  120  are in a common plane parallel to the connecting member  130 . However, in other embodiments, the image capture device  110  and the additional image capture device  120  are configured to be in a common plane using any suitable orientation or configuration. Both the image capture device  110  and the additional image capture device  120  include at multiple wide angle or fisheye lenses that together cover the full 360 degree field of view of the image capture device  110  and of the additional image capture device  120 . 
         [0014]    Images captured by the image capture device  110  and by the additional image capture device  120  are communicated to a controller  140  that combines an image from the image capture device  110  and an additional image from the additional image capture device  120  into an equirectangular image. As further described below, the controller  140  is also configured to process images from the image capture device  110  and from the image capture device  120  when combining an image from the image capture device  110  with an image from the additional image capture device  120 . Processing of an image from the image capture device  110  and an additional image from the additional image capture device  120  by the controller  140  may include one or more of: geometrically and photometrically aligning the image and the additional image, removing vertical parallax between the image and the additional image, and equalizing horizontal disparity between the image and the additional image. Additionally, processing by the controller  140  may exchange regions of the image and the additional image, e.g., to account for reversal of a user&#39;s eyes when looking backwards or to remove the image capture device  110  and the additional image capture device  120  from the additional image and from the image, respectively. 
       Geometric Alignment 
       [0015]    Misalignment where the image capture device  110  or the additional image capture device  120  are in different planes (e.g., misalignment in coupling the image capture device  110  and the additional image capture device  120  to the connecting member  130 ) or misalignment in the image capture device  110  or in the additional image capture device  120  themselves makes stereo viewing of images captured by the image capture device  110  and by the additional image capture device  120  difficult. Relatively small misalignment in the image capture device  110  or in the additional image capture device  120  makes stereo viewing of a combination of an image from the image capture device  110  and an additional image from the additional image capture device  120  difficult.  FIG. 2  shows an example of misalignment between the image capture device  110  and the additional image capture device  120 . In the example of  FIG. 2 , misalignment between the image capture device  110  and the additional image capture device  120  causes a position  210 A of an object  220  in an image captured by the image capture device  110  to differ from a position  210 B of the object  220  in an additional image captured by the additional image capture device  120 . To compensate for this misalignment, the controller  140  determines a rotation to apply to the image captured by the image capture device  110  so a rotated version of the image captured by the image capture device  110  is in a parallel coordinate system with the additional image captured by the additional image capture device  120 . For example, if the image capture device  110  is a camera positioned on the left of the connecting member  130  and the additional image capture device  120  is an additional camera positioned on the right of the connecting member  130 , the controller  140  determines a rotation to be applied to the image captured by the camera positioned on the left of the connecting member  130  so an image captured by the camera positioned on the left of the connecting member  130  (i.e., the image capture device  110 ) is in a parallel coordinate system with an image captured by the camera positioned on the right of the connecting member  130  (i.e., the additional image capture device  120 ). In other embodiments, the controller  140  determines a rotation to be applied to the additional image captured by the additional image capture device  120 , and rotates the additional image so a rotated version of the additional image is in a parallel coordinate system with the image captured by the image capture device  110 . 
         [0016]    To determine the rotation to be applied to the image (or to the additional image), the controller  140  maps the image into a cubemap and maps the additional image into an additional cubemap. For example, the image and the additional image are equirectangular images, and the controller  140  maps the equirectangular image into a cubemap and maps the equirectangular additional image into an additional cubemap. Mapping the image and the additional image into a cubemap and an additional cubemap, respectively, reduces local distortion. 
         [0017]    The controller  140  subsequently identifies features, such as Harris Corners, from the cubemap, which corresponds to the image from the image capture device  110 , and determines locations in the additional cubemap, which corresponds to the additional image from the additional image capture device  120 , corresponding to the identified features from the cubemap. For example, the controller  140  identifies a location in the additional cubemap within a threshold distance of a location in the cubemap of a feature identified from the cubemap by performing a coarse to fine Lucas-Kanade search, or by performing any other suitable search. Based on locations of features identified in the cubemap and corresponding locations in the additional cubemap (determined using any suitable search), the controller  140  determines a transformation, which includes one or more of a rotation and a translation, explaining a maximum amount (e.g., a maximum number, a maximum percentage) of locations of features in the identified cubemap and their corresponding locations in the additional cubemap. For example, the controller  140  uses RANSAC to identify the transformation explaining the maximum amount of locations of features in the identified cubemap and their corresponding locations in the additional cubemap. When the identified transformation includes a translation, the controller  140  discards the translation from the identified transformation and applies the rotation from the identified transformation to the image from the image capture device  110 , causing the rotated image from the image capture device  110  to align with the additional image from the additional image capture device  120 . In other embodiments, the controller  140  similarly determines a translation and/or rotation applied to the additional image rather than applied to the image. 
       Photometric Alignment 
       [0018]    Exposure differences or unequal responses by the image capture device  110  and the additional image capture device  120  cause the image capture device  110  and additional image capture device  120  to capture an image of a scene and an additional image of a scene, respectively, having different exposures or gains. After geometrically aligning the image and the additional image as further described above, the controller  140  blurs the image and the additional image with a low-pass filter, such as a 255 pixel wide low pass filter, which mitigates small effects from disparities and from remaining geometric misalignments. The controller  140  then modifies each color channel of the image captured by the image capture device  110  by multiplying each color channel of each pixel by a ratio of that color channel in the filtered image to the same color channel in the filtered additional image. In embodiments where the controller  140  rotates the additional image rather than the image, the controller  140  modifies each color channel of the additional image captured by the additional image capture device  120  by multiplying each pixel in a color channel by a ratio of the filtered additional image in the color channel to the filtered image in the color channel. 
       Correcting Horizontal and Vertical Disparity 
       [0019]    After geometrically aligning the image from the image capture device  110  and the additional image from the additional image capture device  120 , horizontal and vertical disparity remains between the image and the additional image because of the offset between the image capture device  110  and the additional image capture device  120 . Horizontal or vertical disparity between the image and the additional image is inversely proportional to a distance from the image capture device  110  or the additional image capture device  120  to a point in the scene imaged by the image capture device  110  or the additional image capture device  120  and the apparent interocular distance between the image capture device  110  and the additional image capture device  120 . The apparent interocular distance changes with a cosine of a horizontal azimuth angle away from a normal to a line between the image capture device  110  and the additional image capture device  120 . The apparent interocular distance also varies with a vertical altitude angle above or below the horizon. 
         [0020]      FIG. 3  shows an example imaging geometry for points along a circle  300  on a ground plane. As shown in  FIG. 3 , images of a point  305  directly in front of the image capture device  110  and the additional image capture device  120  have no vertical disparity, as a focal length of the image capture device  110  equals a focal length of the additional image capture device  120  for the point  305 . However, the example of  FIG. 3  shows a point  310  on the circle  300  at a horizontal azimuth angle  315  of 90 degrees lying on a vertical plane through a line connecting the image capture device  110  and the additional image capture device  120  where focal lengths of the image capture device  110  and of the additional image capture device  120  differ. Further,  FIG. 3  shows an example where, at a vertical altitude angle of 45 degrees, if a radius  325  of the circle  300  equals a height of the image capture device  110  and a height of the additional image capture device  120  above the ground plane including the circle  300 , a vertical altitude angle  320 A of the image capture device  110  and a vertical altitude angle  320 B differ by slightly more than 1 degree, causing vertical disparity between the image and the additional image that a user&#39;s eyes is unable to bring into alignment. 
         [0021]      FIG. 4  shows changes in horizontal disparity  410  of the image capture device  110  along a plane at a specific downward angle from a horizontal plane including the image capture device  110 , changes in horizontal disparity  420  of the image capture device  110  and of the additional image capture device  120  along a plane at a specific downward angle from a horizontal plane including the image capture device  110  and the additional image capture device  120 , and changes in vertical disparity  430  of the image capture device  110  and of the additional image capture device  120  along a plane at a specific downward angle from a horizontal plane including the image capture device  110  and the additional image capture device  120 . Having horizontal disparity remain at a constant value for a fixed depth allows users to more readily obtain depth cues from the horizontal disparity. As shown in  FIG. 4 , for a specific altitude angle, the horizontal disparity has a desired value  415  at points directly in front of the image capture device  110  and the additional image capture device  120 , falls to zero at 90 degrees to the left and right of the image capture device  110  and of the additional image capture device  120 , and has an inverse  417  of the desired value  415  at points directly behind a reference plane of the image capture device  110  and the additional image capture device  120 . 
         [0022]    To have the desired value  415  of horizontal disparity for locations behind the image capture device  110  and behind the additional image capture device  120 , the controller  140  exchanges portions of the image captured by the image capture device  110  with portions of the additional image captured by the additional image capture device  120 . As shown in  FIG. 4 , for locations behind the image capture device  110 , the horizontal disparity is the inverse  417  of the desired value  415  for the image capture device  110 . The horizontal disparity of images of locations in front of the reference plane of the additional image capture device  120  is also the inverse of the desired value for the image capture device  120 . Accordingly, the horizontal disparity of images of locations behind the reference plane of the additional image capture device  120  have the desired value  415  of the horizontal disparity of the image capture device  110 , so exchanging images of locations behind the reference plane of the image capture device  110  with additional images of locations behind the reference plane of the additional image capture device  120  causes the images of locations behind the reference plane of the image capture device  110  to have the desired value  415  of horizontal disparity for the image capture device  110 , or to more closely approximate the desired value of horizontal disparity for the image capture device  110 . The preceding exchange also causes the images of locations behind the reference plane of the additional image capture device  120  to have the desired value of horizontal disparity for the additional image capture device  120 , or to more closely approximate the desired value of horizontal disparity for the additional image capture device  120 . 
         [0023]    The controller  140  may further correct horizontal and vertical disparity between the image from the image capture device  110  and the additional image from the additional image capture device  120  by computing dense optical flow from the image to the additional image and from the additional image to the image. To flow the image and the additional image without leaving gaps, the controller  140  iteratively searches for flow vectors to each pixel in the image and in the additional image (i.e., performing flow mapping from destination to source). In various embodiments, optical flow is determined for the image and for the additional image; hence, after determining how far to flow each pixel, the controller  140  modifies the image and the additional image using a half-length flow. 
         [0024]    To further correct for horizontal disparity between an image from the image capture device  110  and an additional image from the additional image capture device  120 , for a specific depth, the controller  140  initially flows the image to increase the horizontal disparity by a factor that is a ratio of a constant to an absolute value of cosine of a horizontal azimuth angle away from a normal to a line between the image capture device  110  and the additional image capture device  120  for different locations at the specific depth. For example, if θ represents the horizontal azimuth angle away from a normal to a line between the image capture device  110  and the additional image capture device  120 , the controller  140  flows the image to increase the horizontal disparity between the image and the additional image by a factor of 1/|cos(θ)|. After increasing the horizontal disparity, the controller  140  exchanges portions of the image corresponding to locations behind the reference plane of the image capture device  110  with portions of the additional image corresponding to locations behind the reference plane of the additional image capture device  120 . The controller  140  may perform similar operations on the additional image to correct for horizontal disparity between the image and the additional image. 
         [0025]    To offset vertical distortion between the image and the additional image, the controller  140  divides the vertical components of the optical flow maps for the image and for the additional image in half. The controller  140  uses the half of the optical flow map for the image when searching for flow vectors to pixels in the image. Similarly, the controller  140  uses the reverse of half of the optical flow map for the additional image when searching for flow vectors to pixels in the additional image. Such compensation for vertical distortion may reduce eyestrain in users subsequently viewing stereoscopic content from the image and the additional image when looking towards the ground of the content, while also making areas of the stereoscopic content closer to the horizon with smaller vertical disparity easier to view. 
         [0026]    At regions where the horizontal azimuth angle away from a normal to the line between the image capture device  110  and the additional image capture device  120  is approximately 90 degrees or −90 degrees, images captured by the image capture device  110  include the additional image capture device  120  or images captured by the additional image capture device  120  include the image capture device  110 . Further, at regions where the horizontal azimuth angle away from a normal to a line between the image capture device  110  and the additional image capture device  120  is approximately 90 degrees or −90 degrees, horizontal disparity falls to zero (although the factor based on the ratio of a constant to the cosine of the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  tends toward infinity to compensate for the decrease in horizontal disparity).  FIG. 5  shows a range  510  of horizontal azimuth angles away from a normal to the line between the image capture device  110  and the additional image capture device  120  where images captured by the image capture device  110  include the additional image capture device  120 . Similarly,  FIG. 5  also shows an additional range  520  of horizontal azimuth angles away from a normal to the line between the image capture device  110  and the additional image capture device  120  where images captured by the additional image capture device  120  include the image capture device  110 . 
         [0027]    To compensate for both of the above, the controller  140  replaces image data captured by the additional image capture device  120  with image data captured by the image capture device  110  for a region where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is within a range. For example, the controller  140  replaces image data captured by the additional image capture device  120  with image data captured by the image capture device  110  where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is greater than −100 degrees and less than −80 degrees. Similarly, for a region where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is within an alternative range, the controller  140  replaces image data captured by the image capture device  110  with image data captured by the additional image capture device  120 . For example, the controller  140  replaces image data captured by the image capture device  110  with image data captured by the additional image capture device  120  where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is greater than 80 degrees and less than 100 degrees. Additionally, the controller  140  may linearly interpolate horizontal disparities across the region where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is within the range (e.g., greater than −100 degrees and less than −80 degrees) by sampling values of the horizontal disparity at locations where the horizontal azimuth angle is a threshold amount outside of the range (e.g., less than −100 degrees by a threshold amount and greater than −80 degrees by the threshold amount). Similarly, the controller  140  linearly interpolates horizontal disparities across the region where the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  is within the alternative range (e.g., greater than 80 degrees and less than 100 degrees) by sampling values of the horizontal disparity at locations where the horizontal azimuth angle is the threshold amount outside of the range (e.g., less than 80 degrees by the threshold amount and greater than 100 degrees by the threshold amount). 
         [0028]      FIG. 6  is a conceptual diagram of correcting horizontal disparity and removing the additional image capture device  120  from an image  610  captured by the image capture device  110  to generate a modified image  630 . In the example of  FIG. 6 , region  615  in the modified image  630  corresponds to a range of the horizontal azimuth angle away from the normal to the line between the image capture device  110  and the additional image capture device  120  between −80 and −100 degrees, so the controller  140  generates the modified image  630  by replacing image data captured by the additional image capture device  120  for the horizontal azimuth angle between −80 and −100 with image data captured by the image capture device  110 . 
         [0029]    To generate the modified image, the controller  140  identifies a starting location in the additional image  620  captured by the additional image capture device  120  that corresponds to a location in the modified image  630  having a horizontal azimuth angle a specified amount less than −100 degrees by inverting the previously described disparity mapping to look up the corresponding flow at that location. The controller  140  then determines a location in the image  610  corresponding to the identified location in the additional image  620  using an optical flow map from the additional image  620  to the image  610 , as further described above. Additionally, the controller  140  identifies an ending location in the image  610  that corresponds to an additional location in the modified image  630  with a horizontal azimuth angle a specified amount greater than −80 degrees by inverting the previously described disparity mapping. The controller  140  subsequently resamples pixels between the location in the image  610  corresponding to the identified location in the additional image  620  and the ending location in the image  610  and fills pixels in the modified image  630  along a line between the location in the modified image  630  and the additional location in the modified image  630 . 
         [0030]    In other embodiments, the controller  140  compensates for horizontal and vertical disparity by transforming an image from the image capture device  110  and an additional image from the additional image capture device  120  into a “transverse equirect” format where epipolar curves indicating how pixels move between the image and the additional image based on depth are mapped to form horizontal lines. In the transverse equirect format, the controller  140  applies one or more stereo matching methods to the image and the additional image. In some embodiments, the controller  140  also applies a weighted push-pull algorithm with weights equal to the stereo matching confidence, resulting in inverse depth. Alternatively, the controller  140  receives the image and the additional image in an equirect format and converts the image and the additional image into cubemap representations. 
       Summary 
       [0031]    The foregoing description of the embodiments has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the patent rights to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure. 
         [0032]    Some portions of this description describe the embodiments in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof. 
         [0033]    Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described. 
         [0034]    Embodiments may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability. 
         [0035]    Embodiments of the invention may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein. 
         [0036]    Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the patent rights be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the patent rights identified by claims that issue on an application that is based on the preceding disclosure.