Method for calibrating an augmented reality device

A method for calibrating a device having a first sensor and a second sensor. The method includes capturing sensor data using the first sensor and the second sensor. The device maintains a calibration profile including a translation parameter and a rotation parameter to model a spatial relationship between the first sensor and the second sensor. The method also includes determining a calibration level associated with the calibration profile at a first time. The method further includes determining, based on the calibration level, to perform a calibration process. The method further includes performing the calibration process at the first time by generating one or both of a calibrated translation parameter and a calibrated rotation parameter and replacing one or both of the translation parameter and the rotation parameter with one or both of the calibrated translation parameter and the calibrated rotation parameter.

BACKGROUND OF THE INVENTION

Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” or “augmented reality” experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A virtual reality, or “VR,” scenario typically involves presentation of digital or virtual image information without transparency to other actual real-world visual input; an augmented reality, or “AR,” scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the actual world around the user.

Despite the progress made in these display technologies, there is a need in the art for improved methods, systems, and devices related to augmented reality systems, particularly, display systems.

SUMMARY OF THE INVENTION

The present disclosure relates generally to methods and systems related to calibration of an augmented reality (AR) device. More particularly, embodiments of the present disclosure provide methods and systems for calibrating an AR device while the device is powered on and in use by adjusting one or more parameters of a calibration profile. Although the present invention is described in reference to an AR device, the disclosure is applicable to a variety of applications in computer vision and image display systems.

In accordance with a first aspect of the present invention, a method for calibrating a device having a first sensor and a second sensor is provided. The method includes capturing sensor data using the first sensor and the second sensor. In some embodiments, the device maintains a calibration profile to model a spatial relationship between the first sensor and the second sensor. In some embodiments, the calibration profile includes a translation parameter and a rotation parameter. The method may also include determining a calibration level associated with the calibration profile at a first time. The method may further include determining, based on the calibration level, whether to perform a calibration process. The method may further include performing the calibration process at the first time by generating one or both of a calibrated translation parameter and a calibrated rotation parameter and replacing one or both of the translation parameter and the rotation parameter with one or both of the calibrated translation parameter and the calibrated rotation parameter.

In some embodiments, performing the calibration process at the first time includes replacing only the rotation parameter with the calibrated rotation parameter. In some embodiments, performing the calibration process at the first time includes generating both the calibrated translation parameter and the calibrated rotation parameter and replacing both the translation parameter and the rotation parameter with the calibrated translation parameter and the calibrated rotation parameter. In some embodiments, the method further includes determining a second calibration level associated with the calibration profile at a second time, determining, based on the second calibration level, to perform a second calibration process, and performing the second calibration process at the second time by generating a second calibrated translation parameter and a second calibrated rotation parameter and replacing the translation parameter and the rotation parameter with the second calibrated translation parameter and the second calibrated rotation parameter. In some embodiments, the calibration level is a first calibration level, the calibration process is a first calibration process, and the rotation parameter is a first calibrated rotation parameter.

In some embodiments, the sensor data includes one or more first images captured using the first sensor and one or more second images captured using the second sensor. In some embodiments, one or both of the calibrated translation parameter and the calibrated rotation parameter are generated using the sensor data. In some embodiments, the calibration level is determined based on the sensor data. In some embodiments, the method further includes capturing additional sensor data using an additional sensor that is separate from the first sensor and the second sensor. In some embodiments, the calibration level is determined based on the additional sensor data. In some embodiments, determining, based on the first calibration level, to perform the first calibration process includes determining that the first calibration level is greater than a calibration threshold and determining, based on the second calibration level, to perform the second calibration process includes determining that the second calibration level is less than the calibration threshold.

In accordance with a second aspect of the present invention, a device is provided. The device may include a first sensor and a second sensor configured to capture sensor data. The device may also include a memory device configured to store a calibration profile modeling a spatial relationship between the first sensor and the second sensor, the calibration profile including a translation parameter and a rotation parameter. The device may further include a processor coupled to the first sensor, the second sensor, and the memory device. In some embodiments, the processor is configured to perform operations including determining a calibration level associated with the calibration profile at a first time. The operations may also include determining, based on the calibration level, to perform a calibration process. The operations may further include performing the calibration process at the first time by generating one or both of a calibrated translation parameter and a calibrated rotation parameter and replacing one or both of the translation parameter and the rotation parameter with one or both of the calibrated translation parameter and the calibrated rotation parameter.

In some embodiments, performing the calibration process at the first time includes replacing only the rotation parameter with the calibrated rotation parameter. In some embodiments, performing the calibration process at the first time includes generating both the calibrated translation parameter and the calibrated rotation parameter and replacing both the translation parameter and the rotation parameter with the calibrated translation parameter and the calibrated rotation parameter. In some embodiments, the operations further include determining a second calibration level associated with the calibration profile at a second time, determining, based on the second calibration level, to perform a second calibration process, and performing the second calibration process at the second time by generating a second calibrated translation parameter and a second calibrated rotation parameter and replacing the translation parameter and the rotation parameter with the second calibrated translation parameter and the second calibrated rotation parameter. In some embodiments, the calibration level is a first calibration level, the calibration process is a first calibration process, and the rotation parameter is a first calibrated rotation parameter.

In some embodiments, the sensor data includes one or more first images captured using the first sensor and one or more second images captured using the second sensor. In some embodiments, one or both of the calibrated translation parameter and the calibrated rotation parameter are generated using the sensor data. In some embodiments, the calibration level is determined based on the sensor data. In some embodiments, the device further includes an additional sensor configured to capture additional sensor data. In some embodiments, the additional sensor is separate from the first sensor and the second sensor. In some embodiments, the calibration level is determined based on the additional sensor data. In some embodiments, determining, based on the first calibration level, to perform the first calibration process includes determining that the first calibration level is greater than a calibration threshold and determining, based on the second calibration level, to perform the second calibration process includes determining that the second calibration level is less than the calibration threshold.

In accordance with a third aspect of the present invention, a non-transitory computer-readable medium for calibrating a device having a first sensor and a second sensor is provided. The non-transitory computer readable medium may include instructions that, when executed by a processor, cause the processor to perform operations. The operations may include the method described in accordance with the first aspect of the present invention.

In accordance with a fourth aspect of the present invention, a method for calibrating an augmented reality device is provided. The method may include accessing a calibration profile including at least one translation parameter and at least one rotation parameter. The method may also include capturing, using a first camera of the augmented reality device, one or more images from the first camera of a first field of view. The method may further include capturing, using a second camera of the augmented reality device, one or more images from the second camera of a second field of view. In some embodiments, the second field of view at least partially overlaps the first field of view. The method may further include comparing at least one of the one or more images from the first camera to at least one of the one or more images from the second camera. The method may further include determining a deformation amount between a first position of the first camera in relation to a second position of the second camera based on the comparison. The method may further include determining whether the deformation amount is greater than a deformation threshold. The method may further include in response to determining that the deformation amount is greater than the deformation threshold: identifying a plurality of matched features present in the one or more images from the first camera and the one or more images from the second camera, partitioning the one or more images from the first camera and the one or more images from the second camera into a plurality of bins, determining, for each bin of the plurality of bins, a quantity of the plurality of matched features located within each bin of the plurality of bins, determining, for each bin of the plurality of bins, that the quantity is greater than a feature threshold, performing a first calibration process by minimizing a first error equation that is a function of a first calibrated rotation parameter to generate the first calibrated rotation parameter; and replacing the at least one rotation parameter in the calibration profile with the first calibrated rotation parameter.

In some embodiments, the method further includes determining whether the deformation amount is less than the deformation threshold and in response to determining that the deformation amount is less than the deformation threshold: capturing, using a plurality of cameras of the augmented reality device including the first camera and the second camera, a plurality of map points, generating a sparse map, the sparse map including a group of map points as seen from a plurality of camera pose positions of the first camera and the second camera, aligning the group of map points of the sparse map, determining, based on the sparse map, that an online calibration trigger is satisfied, performing a second calibration process by minimizing a second error equation that is a function of a second calibrated translation parameter and a second calibrated rotation parameter to generate the second calibrated translation parameter and the second calibrated rotation parameter, and replacing the at least one rotation parameter in the calibration profile with the second calibrated rotation parameter and the at least one translation parameter in the calibration profile with the second calibrated translation parameter. In some embodiments, determining whether the deformation amount is greater than the deformation threshold occurs at a first time and determining whether the deformation amount is less than the deformation threshold occurs at a second time. In some embodiments, the first time precedes the second time. In some embodiments, the second time precedes the first time. In some embodiments, the first time is simultaneous with the second time. In some embodiments, the first time is concurrent with the second time.

In accordance with a fifth aspect of the present invention, an augmented reality device having a calibration profile including a translation parameter and a rotation parameter is provided. The augmented reality device may include a first camera configured to capture one or more first images. The augmented reality device may also include a second camera configured to capture one or more second images. The augmented reality device may further include a processor coupled to the first camera and the second camera. In some embodiments, the processor is configured to perform operations including determining, based on a deformation amount of the first camera in relation to the second camera, that the augmented reality device is deformed at a first time and in response to determining that the augmented reality device is deformed at the first time: performing a first calibration process to generate a first calibrated rotation parameter and replacing the rotation parameter in the calibration profile with the first calibrated rotation parameter.

Numerous benefits are achieved by way of the present invention over conventional techniques. For example, conventional techniques may require a user to repeatedly bring an AR device back to the factory for recalibration. Factory calibration may include making physical measurements on the device using precise instruments, which is time-consuming and expensive for the user of the AR device. In contrast, the present invention allows calibration while the AR device is powered on and in use, providing real-time calibration that is responsive to a particular strain placed on the device based on the particular use of the device. For example, when an AR device is used at warmer temperatures, heat may cause the device to partially warp or expand, thereby rendering any factory calibration inaccurate for the current use. Furthermore, because calibration according to the present invention may be based on captured camera images, it may provide better overall performance in comparison to factory calibration if deformation of the AR device has occurred by providing better alignment of virtual images which are generated based in part on the captured camera images. The method of calibration presented herein is also beneficial in that only rotation corrections are made to the calibration profile when the AR device is deformed beyond some threshold. Under such high deformation circumstances, translation corrections are found to be highly erratic and may result in poor performance of the AR device. Accordingly, the method of calibration provides a “routing”-like functionality in which one of two different process paths is selected based on a deformation amount of the AR device. Other benefits of the present disclosure will be readily apparent to those skilled in the art.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Although optical devices, particularly those with head-mounted displays, may be calibrated with highly sophisticated instruments while in the factory, during use such devices may quickly become deformed due to heat, use, and various forms of wear and tear, causing the factory calibration to become inaccurate. One possible solution is for a user to repeatedly bring the optical device back to the factory for recalibration. To avoid the obvious costs of such a solution, embodiments described herein allow for an accurate and robust run-time calibration while the device is in use, eliminating the need for factory recalibration. Embodiments look to a current calibration level of the device to determine which of two types of calibration processes to perform. A first calibration process limited to rotation corrections is performed when the device is significantly out of calibration, and a second calibration including rotation and translation corrections is performed under slight miscalibration. Embodiments described herein are useful not only for optical devices, but for any device having two sensors with a spatial relationship that is modeled by a translation component and a rotation component.

FIG. 1is a drawing illustrating an augmented reality (AR) scene as viewed through a wearable AR device according to an embodiment described herein. Referring toFIG. 1, an augmented reality scene100is depicted wherein a user of an AR technology sees a real-world park-like setting106featuring people, trees, buildings in the background, and a concrete platform120. In addition to these items, the user of the AR technology also perceives that he “sees” a robot statue110standing upon the real-world platform120, and a cartoon-like avatar character102flying by, which seems to be a personification of a bumble bee, even though these elements (character102and statue110) do not exist in the real world. Due to the extreme complexity of the human visual perception and nervous system, it is challenging to produce a virtual reality (VR) or AR technology that facilitates a comfortable, natural-feeling, rich presentation of virtual image elements amongst other virtual or real-world imagery elements.

FIG. 2illustrates a schematic view of a wearable AR device200, according to some embodiments of the present invention. AR device200may include a left eyepiece202A as part of a left optical stack and a right eyepiece202B as part of a right optical stack. In some embodiments, AR device200includes one or more sensors including, but not limited to: a left front-facing world sensor206A attached directly to or near left eyepiece202A, a right front-facing world sensor206B attached directly to or near right eyepiece202B, a left side-facing world sensor206C attached directly to or near left eyepiece202A, and a right side-facing world sensor206D attached directly to or near right eyepiece202B. The positions of one or more of sensors206may vary from the illustrated embodiment, and may include various backward-, forward-, upward-, downward-, inward-, and/or outward-facing configurations. Sensors206A,206B,206C,206D may be configured to generate, detect, and/or capture sensor data220A,220B,220C,220D, respectively, which may be electronic data corresponding to a physical property of the environment surrounding AR device200, such as motion, light, temperature, sound, humidity, vibration, pressure, and the like.

In some embodiments, one or more of sensors206may be cameras and one or more of sensor data220may be camera images. For example, sensor data220may include a single image, a pair of images, a video comprising a stream of images, a video comprising a stream of paired images, and the like. In some embodiments, one or more of sensors206may be depth sensors and one or more of sensor data220may be depth images/maps. For example, one of sensors206may include a time-of-flight imaging system configured to transmit light pulses to illuminate target objects and to determine distances to the target objects based on received optical signals. One example of such a system is described in reference to U.S. patent application Ser. No. 15/721,640 titled “REAL TIME CALIBRATION FOR TIME-OF-FLIGHT DEPTH MEASUREMENT” filed on Sep. 29, 2017, the entire disclosure of which is hereby incorporated by reference, for all purposes, as if fully set forth herein. Additional examples of sensors206may include any type of motion sensor, depth sensor, light sensor, mechanical sensor, temperature sensor, sound sensor, humidity sensor, vibration sensor, pressure sensor, and the like.

AR device200may include an additional sensor207separate from sensors206. Additional sensor207may be configured to generate, detect, and/or capture additional sensor data221. Additional sensor207may be any type of sensor described above in reference to sensors206and additional sensor data221may be any type of sensor data described above in reference to sensor data220. In some embodiments, additional sensor data221is used to determine a calibration level associated with sensors206(i.e., associated with calibration profile254), as is described in further detail below. In one example, additional sensor207is a strain gauge positioned over a portion of AR device200(e.g., extending between two of sensors206) for determining the strain to AR device200. In another example, additional sensor207is a mechanical sensor positioned along the frame of AR device200(e.g., at a center point between eyepieces202) for measuring the bend, angle, torsion, etc., of a portion of the frame of AR device200. Further examples of additional sensor207are provided in U.S. Provisional Patent Application No. 62/698,015 filed Jul. 13, 2018, titled “SYSTEMS AND METHODS FOR DISPLAY BINOCULAR DEFORMATION COMPENSATION”, the entire disclosure of which is hereby incorporated by reference, for all purposes, as if fully set forth herein.

In some embodiments, AR device200includes one or more image projection devices such as a left projector214A that is optically linked to left eyepiece202A and a right projector214B that is optically linked to right eyepiece202B. Projectors214may inject light associated with virtual content onto one or more waveguides of eyepieces202in a manner that a user perceives virtual content as being positioned at a particular distance. Eyepieces202A,202B may comprise transparent or semi-transparent waveguides configured to direct and outcouple light received from projectors214A,214B, respectively. During operation, a processing module250may cause left projector214A to output left virtual image light222A onto left eyepiece202A, and may cause right projector214B to output right virtual image light222B onto right eyepiece202B. In some embodiments, each of eyepieces202may comprise a plurality of waveguides corresponding to different colors and/or different depth planes.

Some or all of the components of AR device200may be head mounted such that projected images may be viewed by a user. In one particular implementation, all of the components of AR device200shown inFIG. 2are mounted onto a single device (e.g., a single headset) wearable by a user. In another implementation, one or more components of a processing module250are physically separate from and communicatively coupled to the other components of AR device200by one or more wired and/or wireless connections. For example, processing module250may include a local module on the head mounted portion of AR device200and a remote module physically separate from and communicatively linked to the local module. The remote module may be mounted in a variety of configurations, such as fixedly attached to a frame, fixedly attached to a helmet or hat worn by a user, embedded in headphones, or otherwise removably attached to a user (e.g., in a backpack-style configuration, in a belt-coupling style configuration, etc.).

Processing module250may include a processor252and an associated digital memory, such as non-volatile memory (e.g., flash memory), both of which may be utilized to assist in the processing, caching, and storage of data, such as sensor data220. For example, processing module250may receive left front image(s) (i.e., sensor data220A) from a left front-facing camera (i.e., sensor206A), right front image(s) (i.e., sensor data220B) from a right front-facing world camera (i.e., sensor206B), left side image(s) (i.e., sensor data220C) from a left side-facing world camera (i.e., sensor206C), and right side image(s) (i.e., sensor data220D) from a right side-facing world camera (i.e., sensor206D). Sensor data220may be periodically generated and sent to processing module250while AR device200is powered on, or may be generated in response to an instruction sent by processing module250to one or more of the cameras. As another example, processing module250may receive ambient light information (i.e., sensor data220) from an ambient light sensor (i.e., sensor206).

When implemented as cameras, sensors206A,206B may be positioned to capture images that substantially overlap with the field of view of a user's left and right eyes, respectively. Accordingly, placement of sensors206may be near a user's eyes, but not so near as to obscure the user's field of view. Alternatively or additionally, sensors206A,206B may be positioned so as to substantially align with the incoupling locations of virtual image light222A,222B, respectively. When implemented as cameras, sensors206C,206D may be positioned to capture images to the side of a user, e.g., in a user's peripheral vision or outside the user's peripheral vision. Images captured using sensors206C,206D need not necessarily overlap with images captured using sensors206A,206B.

During operation of AR device200, processing module250may use one or more parameters from a calibration profile254to account for the spacing and orientation differences between sensors206so that sensor data220may be correctly analyzed. Calibration profile254may additionally be used when generating virtual image light222to account for the spacing and orientation differences between eyepieces202such that a user may view virtual image elements comfortably and in proper alignment. To accomplish this, processor252may repeatedly access calibration profile254to ensure that the parameters being used reflect the most updated and accurate parameters that are available. In some instances, processor252may retrieve parameters from calibration profile254immediately after a calibration process is performed. In one particular implementation, calibration profile254is stored in a non-volatile memory such that processor252may retrieve the last used parameters upon powering on AR device200. Alternatively, it may be desirable to access a stored factory calibration at startup of AR device200when AR device200may not have significant deformation resulting from, for example, thermal expansion of the device caused by running onboard electronic components.

In some embodiments, calibration profile254is maintained by processor252to model a spatial relationship between a first sensor and a second sensor of sensors206(e.g., sensors206A,206B). According to some embodiments of the present invention, calibration profile254includes a translation parameter T corresponding to the relative distance between the first sensor and the second sensor, and a rotation parameter R corresponding to the relative angular orientation between the first sensor and the second sensor. Each of translation parameter T and rotation parameter R may take on a wide range of data types. For example, translation parameter T may be a single quantity (e.g., 0.1 meters), a one-dimensional matrix (e.g., [0.1; 0; 0] meters), a multi-dimensional matrix (e.g., [[0.1; 0; 0][0; 0; 0][0; 0; 0]] meters), an array, a vector, or any other possible representation of single or multiple quantities. Similarly, rotation parameter R may be a single quantity (e.g., 0.5 degrees), a one-dimensional matrix (e.g., [0.5; 0; 0] degrees), a multi-dimensional matrix (e.g., [[0.5; 0; 0][0; 0; 0][0; 0; 0]] degrees), an array, a vector, or any other possible representation of single or multiple quantities.

Under ideal conditions, translation parameter T and rotation parameter R are calibrated in the factory immediately after manufacture of AR device200, and remain accurate indications of the spatial relationship between the first sensor and the second sensor throughout the life of the device. Under actual conditions, AR device200becomes deformed due to heat, use, and various forms of wear and tear, causing the factory calibrated values of translation parameter T and rotation parameter R to become inaccurate. One possible solution is for a user to repeatedly bring AR device200back to the factory for recalibration. Alternatively, a run-time calibration method may be employed for calibrating translation parameter T and rotation parameter R while AR device200is powered on and in use by a user.

In some instances, a calibration level associated with calibration profile254is periodically determined. Based on the calibration level, processing module250may cause one of several types of calibrations to occur. For example, when the calibration level is below a first calibration threshold, processing module250may cause a first calibration process to be performed, and when the calibration level is above the first calibration threshold, processing module250may cause a second calibration process to be performed. In some instances, neither calibration process may be performed when the calibration level is above the first calibration threshold and a second calibration threshold, indicating that calibration profile254is accurate. As used herein, the term “calibration level” may correspond to the level of accuracy of calibration profile254in modeling the actual spatial relationship between the first sensor and the second sensor (e.g., sensors206A,206B). Accordingly, higher calibration levels may correspond to a more accurate modeling of the actual spatial relationship and a lower calibration level may correspond to less accurate modeling of the actual spatial relationship. The process of monitoring a calibration level in connection with calibration of AR device200is described in further detail below.

FIG. 3illustrates a calibration model300of AR device200, according to some embodiments of the present invention. In calibration model300, each of sensors206may be represented using the pinhole camera model as occupying a single point, with sensor206C being offset from sensor206A by a known translation and rotation (modeled by the transformation [TL|RL]) and sensor206D being offset from sensor206B by a known translation and rotation (modeled by the transformation [TR|RR]). A center point302between sensors206A,206B is used to track the position of AR device200in the environment with respect to a world origin and is also used as a baseline for translation and rotation adjustments. In some embodiments, the relative distance between each of sensors206A,206B and center point302may be equal to translation parameter T, where translation parameter T represents a 3×1 matrix corresponding to a three-dimensional (3D) vector (e.g., [0.1 0.2 0.1] meters). In some embodiments, the relative angular orientation between each of sensors206A,206B and center point302may be equal to rotation parameter R, where rotation parameter R represents a 3×3 matrix (referred to, in some embodiments, as a rotation vector). Accordingly, the transformation between sensor206B and center point302may be modeled by the transformation [T|R] and the transformation between sensor206A and center point302may be modeled by the transformation [T|R]−1.

FIGS. 4A, 4B, and 4Cillustrate various steps for determining a calibration level associated with calibration profile254, according to some embodiments of the present invention.

In some embodiments, determining a calibration level associated with calibration profile254includes determining a deformation amount D of AR device200affecting the position and/or orientation of left front-facing world sensor206A in relation to right front-facing world sensor206B. Deformation amount D may be used as a calibration level and may be inversely proportional to the calibration level as described herein. For example, determining whether deformation amount D is greater than a deformation threshold may be tantamount to determining whether the calibration level is less than a calibration threshold. Similarly, determining whether deformation amount D is less than a deformation threshold may be tantamount to determining whether the calibration level is greater than a calibration threshold.

The steps described in reference toFIGS. 4A, 4B, and 4Cmay be performed on a per-frame basis or be performed every N-th frame. The steps may use epipolar geometry, which may require that a pair of corresponding “features” or “points of interest” be observable by each of sensors206A,206B. Deformation amount D may be determined prior to or during performance of methods800,900,1000,1100, described in reference toFIGS. 8, 9, 10, and 11.

In reference toFIG. 4A, a left image402captured by sensor206A at time t1may be compared to a right image404captured by sensor206B at time t1to identify at least one feature, either in its entirety or portions thereof, that appears in both images (using, for example, one or more feature matching techniques). After determining that both images402,404include a feature420(a five-pointed star), an epipolar line422is generated based on left image402and is projected onto right image404. Epipolar line422may be generated based on the vertical/horizontal positioning and/or orientation of feature420as appearing in left image402, and may be projected onto right image404using the most updated version of calibration profile254. Epipolar line422represents a line on which feature420is expected to lie from the perspective of sensor206B if sensors206A,206B are perfectly aligned. Deviation in the position of feature420from epipolar line422indicates calibration error between sensors206A,206B and the magnitude of the deviation corresponds to more or less error.

In some embodiments, a first point405and a second point407are identified within feature420in each of images402,404to facilitate determining the vertical/horizontal positioning and/or orientation of feature420. In the example shown inFIG. 4A, left image402is analyzed to identify first point405along a top left point of feature420and second point407along a top right point of feature420. Next, an intersecting line between first point405and second point407is formed in left image402and the intersecting line is transformed from left image402to right image404using calibration profile254to project epipolar line422onto right image404. Once epipolar line422is projected onto right image404, first point405and second point407within right image404and are compared to epipolar line422. After comparing feature420(i.e., points405and407) in right image404to epipolar line422, deformation amount D (i.e., a calibration level) may be calculated based on the translation offset and the orientation offset between feature420and epipolar line422. Because feature420in the example illustrated inFIG. 4Ais aligned well with epipolar line422, deformation amount D is determined to be low (e.g., equal to 0). Additional features may be analyzed to determine deformation amount D with a higher degree of accuracy. In some embodiments, deformation amount D is expressed in pixels and may, in some embodiments, be equal to the number of pixels separating feature420and epipolar line422. In some embodiments, deformation amount D is referred to as the reprojection error.

In reference toFIG. 4B, a left image406captured by sensor206A at time t2may be compared to a right image408captured by sensor206B at time t2to identify feature420appearing in both images. Images406,408are analyzed to identify points405,407within feature420in each of images406,408. Next, an intersecting line between points405,407is formed in left image406and the intersecting line is transformed from left image406to right image408using the latest updated version of calibration profile254to project epipolar line422onto right image408. Points405,407within right image408are then compared to epipolar line422to determine deformation amount D. Because feature420is not aligned with epipolar line422(the translation offset and the orientation offset are significant as shown by the misalignment between points405and407and epipolar line422), deformation amount D is determined to be higher than the example shown inFIG. 4A(e.g., equal to 26.3).

In reference toFIG. 4C, a left image410captured by sensor206A at time t3may be compared to a right image412captured by sensor206B at time t3to identify feature420appearing in both images. Images410,412are analyzed to identify points405,407within feature420in each of images410,412. Next, an intersecting line between points405,407is formed in left image410and the intersecting line is transformed from left image410to right image412using the latest updated version of calibration profile254to project epipolar line422onto right image412. Points405,407within right image412are then compared to epipolar line422to determine deformation amount D. Because feature420is significantly offset with epipolar line422(the translation offset and the orientation offset are significant as shown by the misalignment between points405and407and epipolar line422), deformation amount D is determined to be higher than the examples shown inFIGS. 4A and 4B(e.g., equal to 84.1).

FIGS. 5A, 5B, and 5Cillustrate various steps for determining a calibration level associated with calibration profile254, according to some embodiments of the present invention. The examples illustrated inFIGS. 5A, 5B, and 5Ccorrespond to the examples illustrated inFIGS. 4A, 4B, and 4C, respectively, and demonstrate an alternative approach of projecting the right image onto the left image to calculate an identical deformation amount D (i.e., calibration level). The steps described in reference toFIGS. 5A, 5B, and 5Cmay be performed on a per-frame basis or be performed every N-th frame.

In reference toFIG. 5A, a left image502captured by sensor206A at time t1may be compared to a right image504captured by sensor206B at time t1to identify feature520appearing in both images. Images502,504are analyzed to identify points505,507within feature520in each of images502,504. Next, an intersecting line between points505,507is formed in right image504and the intersecting line is transformed from right image504to left image502using the latest updated version of calibration profile254to project epipolar line522onto left image502. Points505,507within left image502are then compared to epipolar line522to determine deformation amount D. Because feature520in the example illustrated inFIG. 5Ais aligned well with epipolar line522, deformation amount D is determined to be low (e.g., equal to 0).

In reference toFIG. 5B, a left image506captured by sensor206A at time t2may be compared to a right image508captured by sensor206B at time t2to identify feature520appearing in both images. Images506,508are analyzed to identify points505,507within feature520in each of images506,508. Next, an intersecting line between points505,507is formed in right image508and the intersecting line is transformed from right image508to left image506using the latest updated version of calibration profile254to project epipolar line522onto left image506. Points505,507within left image506are then compared to epipolar line522to determine deformation amount D. Because feature520is not aligned with epipolar line522(the translation offset and the orientation offset are significant as shown by the misalignment between points505and507and epipolar line522), deformation amount D is determined to be higher than the example shown inFIG. 5A(e.g., equal to 26.3).

In reference toFIG. 5C, a left image510captured by sensor206A at time t3may be compared to a right image512captured by sensor206B at time t3to identify feature520appearing in both images. Images510,512are analyzed to identify points505,507within feature520in each of images510,512. Next, an intersecting line between points505,507is formed in right image512and the intersecting line is transformed from right image512to left image510using the latest updated version of calibration profile254to project epipolar line522onto left image510. Points505,507within left image510are then compared to epipolar line522to determine deformation amount D. Because feature520is significantly offset with epipolar line522(the translation offset and the orientation offset are significant as shown by the misalignment between points505and507and epipolar line522), deformation amount D is determined to be higher than the examples shown inFIGS. 5A and 5B(e.g., equal to 84.1).

FIG. 6illustrates an example calculation of deformation amount D (i.e., calibration level) based on two images having a common feature appearing in both images, according to some embodiments of the present invention. First, feature620having points605,607is identified in both a first image602and a second image (not shown). First image602may represent a left image or a right image, among other possibilities. An intersecting line between points605,607is formed in the second image and is transformed from the second image to first image602using the latest updated version of calibration profile254to project epipolar line622onto first image602(as is shown in reference toFIGS. 4A, 4B, 4C, 5A, 5B, and 5C). Points605,607within first image602are then compared to epipolar line622to determine deformation amount D.

In some embodiments, a first offset650is calculated as a vertical distance between point605and epipolar line622and/or a second offset652is calculated as a vertical distance between point607and epipolar line622. The calculated value of deformation amount D may be equal to or related to (e.g., a scaled version of) first offset650, second offset652, an average offset654between first offset650and second offset652, a minimum or maximum of first offset650and second offset652, a ratio between first offset650and second offset652(e.g., first offset650divided by second offset652, second offset652divided by first offset650, etc.), a difference between first offset650and second offset652(e.g., first offset650subtracted from second offset652or second offset652subtracted from first offset650, etc.), among other possibilities.

FIG. 7illustrates an example calculation of deformation amount D (i.e., calibration level) based on two images having a common feature appearing in both images, according to some embodiments of the present invention. First, feature720having points705,707is identified in both a first image702and a second image (not shown). First image702may represent a left image or a right image, among other possibilities. An intersecting line between points705,707is formed in the second image and is transformed from the second image to first image702using the latest updated version of calibration profile254to project epipolar line722onto first image702(as is shown in reference toFIGS. 4A, 4B, 4C, 5A, 5B, and 5C). Points705,707within first image702are then compared to epipolar line722to determine deformation amount D.

In some embodiments, a line756intersecting points705,707is formed in first image702and an angle758between line756and epipolar line722is calculated. Angle758may alternatively or additionally be calculated by determining vertical offsets between points705,707and epipolar line722(similar to first offset650and second offset652described in reference toFIG. 6) and a horizontal offset between points705,707, and using trigonometry to solve for angle758. The calculated value of deformation amount D may be equal to or related to (e.g., a scaled version of) angle758, the sine (function) of angle758, the tangent (function) of angle758, the inverse of angle758, among other possibilities.

In some embodiments, the deformation amount calculated in reference toFIG. 6is a translation deformation amount DTand the deformation amount calculated in reference toFIG. 7is a rotation deformation amount DR. In some embodiments, the calculated value of deformation amount D may be equal to or related to (e.g., a scaled version of) the sum of translation deformation amount DTand rotation deformation amount DR, an average between translation deformation amount DTand rotation deformation amount DR, a minimum or maximum of translation deformation amount DTand rotation deformation amount DR, a ratio between translation deformation amount DTand rotation deformation amount DR(e.g., translation deformation amount DTdivided by rotation deformation amount DR, rotation deformation amount DRdivided by translation deformation amount DT, etc.), a difference between translation deformation amount DTand rotation deformation amount DR(e.g., translation deformation amount DTsubtracted from rotation deformation amount DRor rotation deformation amount DRsubtracted from translation deformation amount DT, etc.), among other possibilities.

FIG. 8illustrates a method800for calibrating AR device200, according to some embodiments of the present invention. Performance of method800may include performing more or fewer steps than those shown inFIG. 8, and steps of method800need not be performed in the order shown. Although method800is described in reference to calibrating an AR device, the method may be used to calibrate any device having two sensors whose spatial relationship is modeled by a calibration profile having a translation parameter and a rotation parameter.

In some embodiments, method800begins at block802in which sensor data220is captured by sensors206. In some embodiments, sensor data220may be captured by a first sensor and a second sensor of sensors206. For example, sensor data220may include one or more first images captured by the first sensor and one or more second images captured by the second sensor. In some embodiments, both the first images and the second images are camera images. In some embodiments, both the first images and the second images are depth images (i.e., depth maps). In some embodiments, the first images are camera images and the second images are depth images. In some embodiments, the first images are depth images and the second images are camera images. After sensor data220is captured by sensors206, sensor data220may be sent to processing module250.

At block804, a calibration level associated with calibration profile254is determined. In some embodiments, the calibration level is determined based on sensor data220, e.g., by analyzing one or both of the one or more first images and the one or more second images. For example, the one or more first images may be compared to the one or more second images and the calibration level may be determined based on the comparison. As another example, a deformation amount D of the first sensor in relation to the second sensor may be determined based on the comparison, and deformation amount D may be used as the calibration level (higher deformation amounts corresponding to lower levels of reliability). In some embodiments, the calibration level is determined by performing the steps described in reference toFIGS. 4A, 4B, and4C and/orFIGS. 5A, 5B, and 5C. In some embodiments, block804is performed by processing module250.

In some embodiments, determining the calibration level includes determining whether a head pose algorithm associated with AR device200is currently available and/or is currently generating accurate data. In some embodiments, the head pose algorithm may be used to generate map points (3D points) from sensor data220captured by sensors206. For example, the head pose algorithm may receive a pair of images and generate map points by processing the pair of images. If AR device200becomes too deformed, the head pose algorithm will either be unable to converge or will be unable to generate accurate map points. In either case, the head pose algorithm may be considered to be “unavailable”. In some embodiments, the calibration level may be associated with whether the head pose algorithm is available by, for example, having a first value (e.g., 1) when available and a second value (e.g., 0) when unavailable or, in some embodiments, having a value therebetween indicating a level of availability (e.g., 0.5).

At block806, it is determined whether to perform a first calibration process, a second calibration process, or neither based on the calibration level. For example, the calibration level may have a single value that may be compared to one or more calibration thresholds. In some instances, the calibration level may be normalized to have a value between 0 and 1. In one example, the calibration level may be compared to a first calibration threshold807-1to determine whether the first calibration process or the second calibration process is to be performed. It may be determined to perform the first calibration process when the calibration level is above first calibration threshold807-1and to perform the second calibration process when the calibration level is below first calibration threshold807-1, or vice-versa. In some embodiments, it may be determined to perform neither calibration process when the calibration level is above a second calibration threshold807-2, which may be higher than first calibration threshold807-1. Alternatively or additionally, it may be determined whether the calibration level is within a range of values, whether the calibration level is included in a list of values, whether the calibration level is greater than or less than a previously determined the calibration level by some threshold amount, whether the calibration level is less than a threshold for a particular amount of time (e.g., 250 milliseconds), or the like. In some embodiments, block806is performed by processing module250.

In another example in which deformation amount D is used as the calibration level, deformation amount D may be compared to a first deformation threshold to determine whether the first calibration process or the second calibration process is to be performed. It may be determined to perform the first calibration process when deformation amount D is below the first deformation threshold and to perform the second calibration process when deformation amount D is above the first deformation threshold, or vice-versa. In some embodiments, it may be determined to perform neither calibration process when deformation amount D is below a second deformation threshold, which may be lower than the first deformation threshold. Alternatively or additionally, it may be determined whether deformation amount D is within a range of values, whether deformation amount D is included in a list of values, whether deformation amount D is greater than or less than a previously determined deformation amount D by some threshold amount, whether deformation amount D is greater than a threshold for a particular amount of time (e.g., 250 milliseconds), or the like.

If it is determined at block806that the first calibration process is to be performed, e.g., the calibration level is less than first calibration threshold807-1, method800proceeds to block808. At block808, a first calibration process is performed which includes calibrating rotation parameter R while translation parameter T is not modified, i.e., only rotation parameter R is calibrated. Performing the first calibration process may include generating a calibrated rotation parameter R′ to be used for replacing and/or updating rotation parameter R. In some embodiments, the first calibration process is performed using sensor data220. The first calibration process may include minimizing an error equation in which translation parameter T is held constant (to its most current value) and rotation parameter R is varied (e.g., fluctuated) over a range of possible values. The value of rotation parameter R for which the error equation is minimized is set as calibrated rotation parameter R′. In some embodiments, block808is performed by processing module250.

If it is determined at block806that the second calibration process is to be performed, e.g., the calibration level is greater than first calibration threshold807-1(but less than second calibration threshold807-2), method800proceeds to block810. At block810, a second calibration process is performed which includes calibrating both translation parameter T and rotation parameter R, which may include generating a calibrated translation parameter T′ and a calibrated rotation parameter R′ to be used for replacing and/or updating translation parameter T and rotation parameter R, respectively. The second calibration process may include minimizing an error equation in which both translation parameter T and rotation parameter R are varied over a range of possible values. The values of translation parameter T and rotation parameter R for which the error equation is minimized are set as calibrated translation parameter T′ and calibrated rotation parameter R′, respectively. In some embodiments, block810is performed by processing module250.

FIG. 9illustrates a method900for calibrating AR device200, according to some embodiments of the present invention. Performance of method900may include performing more or fewer steps than those shown inFIG. 9, and steps of method900need not be performed in the order shown. One or more steps of method900may correspond to one or more steps of method800. Although method900is described in reference to calibrating an AR device, the method may be used to calibrate any device having two sensors whose spatial relationship is modeled by a calibration profile having a translation parameter and a rotation parameter.

In some embodiments, method900begins at block902in which sensor data220(i.e., first sensor data) is captured by sensors206. Block902may include one or more steps described in reference to block802.

At block903, additional sensor data221(i.e., second sensor data) is captured by additional sensor207. Additional sensor207may be a separate sensor from sensors206. In one example, additional sensor207is a strain gauge positioned over a portion of AR device200(e.g., extending between two of sensors206) for determining the strain to AR device200.

At block904, a calibration level associated with calibration profile254is determined based on additional sensor data221(i.e., second sensor data). In some embodiments, the calibration level is determined by analyzing one or more images of additional sensor data221. Block904may include one or more steps described in reference to block804. In some embodiments, block904is performed by processing module250.

At block906, it is determined whether to perform a first calibration process, a second calibration process, or neither based on the calibration level. Block906may include one or more steps described in reference to block806. In some embodiments, block906is performed by processing module250.

If it is determined at block906that the first calibration process is to be performed, method900proceeds to block908. At block808, a first calibration process is performed which includes calibrating rotation parameter R using sensor data220(i.e., first sensor data) while translation parameter T is not modified. Block908may include one or more steps described in reference to block808. In some embodiments, block908is performed by processing module250.

If it is determined at block906that the second calibration process is to be performed, method900proceeds to block910. At block910, a second calibration process is performed which includes calibrating both translation parameter T and rotation parameter R using sensor data220(i.e., first sensor data). Block910may include one or more steps described in reference to block810. In some embodiments, block910is performed by processing module250.

FIG. 10illustrates a method1000for calibrating AR device200, according to some embodiments of the present invention. Performance of method1000may include performing more or fewer steps than those shown inFIG. 10, and steps of method1000need not be performed in the order shown. One or more steps of method1000may correspond to one or more steps of methods800,900. For example, method1000may comprise an epipolar calibration1050which may correspond to one or more steps described in reference to block808, and an online calibration1052which may correspond to one or more steps described in reference to block810. Although method1000is described in reference to calibrating an AR device, the method may be used to calibrate any device having two sensors whose spatial relationship is modeled by a calibration profile having a translation parameter and a rotation parameter.

In some embodiments, method1000begins at block1002in which sensor data220is captured by sensors206. Block1002may include one or more steps described in reference to block802.

At block1004, a calibration level associated with calibration profile254is determined. Block1004may include one or more steps described in reference to blocks804,904. In some embodiments, block1004is performed by processing module250.

At block1006, it is determined whether to perform a first calibration process, a second calibration process, or neither based on the calibration level. Block1006may include one or more steps described in reference to block806. In some embodiments, block1006is performed by processing module250.

If it is determined at block1006that the first calibration process is to be performed, method1000proceeds to block1008. At block1008, a first equation EQ.1 is minimized by varying (e.g., fluctuating) rotation parameter R over a range of possible values while translation parameter T is held constant (to its most recently updated value in calibration profile254). Although various error equations may be used for first equation EQ.1, in some implementations a variant of the Sampson error may be used as follows:

EQ.⁢1=∑i⁢(xi′·E·xi)2(E·xi)12+(E·xi)22+(ET·xi′)12+(ET·xi′)22
where (·)k denotes the k-th component in the vector, E=[T]x·R is the essential matrix, and x and x′ are the corresponding features from left and right images in normalized image coordinates. Advantages of using this variant of the Sampson error in EQ.1 include: (1) the feature coordinates used are in normalized image coordinates, (2) the essential matrix E is more computationally efficient than the fundamental matrix, and (3) the intrinsics of the cameras are assumed to not change. In one particular implementation, essential matrix E is a 3×3 matrix. Once EQ.1 is minimized, the value of rotation parameter R for which the equation is minimized is set and outputted as calibrated rotation parameter R′.

If it is determined at block1006that the second calibration process is to be performed, method1000proceeds to block1010where online calibration1052is performed. Online calibration1052aims to minimize the reprojection error between observed and predicted image points with respect to rotation and translation of the sensors (e.g., cameras). At block1010, a second equation EQ.2 is minimized by varying (e.g., fluctuating) translation parameter T and rotation parameter R over a range of possible values. Although various error equations may be used for second equation EQ.2, in some implementations the following error equation may be used:

EQ.⁢2=∑i⁢∑j⁢∑k∈ℂ⁢Vij·P⁡(Πjk⁡(Trigjck,pi),xij,k)
where i is the index for points, j is the index for rig positions at keyframes, k is the index for cameras,is the set of cameras, Trigjckis the extrinsic transformation from rig to sensor k (e.g., camera k), Πjkis the projection function for sensor k in rig j, xij,kis the measurement of 3D point piin sensor k, P is the function to compute the reprojection error vector between two points, and Vijis a value of either 0 or 1 based on visibility of point i through sensor k located at keyframe position j (equal to 1 if visible and 0 if not visible). The projection function Πjkis dependent on translation parameter T and rotation parameter R, as the transformation from rig center point to each sensor relates to T and R for each sensor. Once EQ.2 is minimized, the values of translation parameter T and rotation parameter R for which the equation is minimized are set and outputted as calibrated translation parameter T′ and calibrated rotation parameter R′, respectively. Additional description of the extrinsic transformation from rig to camera is illustrated inFIG. 15.

At block1012, either one or both of calibrated translation parameter T′ and calibrated rotation parameter R′ are used to replace and/or update translation parameter T and rotation parameter R, respectively. If epipolar calibration1050was performed, then rotation parameter R is replaced and/or updated. If online calibration1052was performed, then both translation parameter T and rotation parameter R are replaced and/or updated. After performance of block1012, method1000proceeds to block1002, repeating the described steps.

FIG. 11illustrates a method1100for calibrating AR device200, according to some embodiments of the present invention. Performance of method1100may include performing more or fewer steps than those shown inFIG. 11, and steps of method1100need not be performed in the order shown. One or more steps of method1100may correspond to one or more steps of methods800,900,1000. For example, method1100may comprise an epipolar calibration1150, which may correspond to one or more steps described in reference to blocks808and1008, and an online calibration pathway1152, which may correspond to one or more steps described in reference to blocks810and1010. Although method1100is described in reference to calibrating an AR device, the method may be used to calibrate any device having two sensors whose spatial relationship is modeled by a calibration profile having a translation parameter and a rotation parameter.

In some embodiments, method1100begins at block1102in which sensor data220is captured by sensors206. Block1102may include one or more steps described in reference to block802.

At block1104, a calibration level of sensor data220is determined. Block1104may include one or more steps described in reference to blocks804,904. In some embodiments, block1104is performed by processing module250.

At block1106, it is determined whether to perform a first calibration process, a second calibration process, or neither based on the calibration level. Block1106may include one or more steps described in reference to block806. In some embodiments, block1106is performed by processing module250.

If it is determined at block1106that the first calibration process is to be performed, method1100proceeds to block1108. At block1108, image analysis and feature detection is performed on paired images captured by sensors206A,206B. In some embodiments, matched features between the paired images are detected or, in other embodiments, the matched features are received during or prior to performance of block1108from an external source. After obtaining the matched features and the paired images, each of the paired images are partitioned into a plurality of bins and the quantity of matched features that are located in each of the bins is determined. In various embodiments, each of the paired images are partitioned into the same number of bins, into different numbers of bins, or into bins that cover different regions of each of the paired images. In one particular embodiment, the bins may be defined by a 3×3 grid overlaid on the images. After determining the quantity of matched features for each bin, the quantities are outputted and method1100proceeds to block1110.

At block1110, it is determined whether the quantities of matched features located in each of the bins satisfy one or more feature thresholds. For example, it may be determined whether each of the quantities of matched features is greater than a feature threshold, e.g., 1, 10, 100, 1,000, and the like. In some embodiments, this inquiry may be performed on a bin-by-bin basis, such that method1100only proceeds to block1112when each of the quantities of matched features is greater than the feature threshold. In other embodiments, method1100may proceed to block1112when a majority or some requisite percentage of bins include a quantity of matched features greater than the feature threshold. In some embodiments, it may also be determined whether each of the quantities of matched features is less than a second feature threshold, e.g., 1,000, 10,000, and the like. This step may determine whether the matched features are evenly spread throughout the paired images. If it is determined that each of the quantities of matched features is greater than a first feature threshold and less than a second feature threshold, method1100may proceed to block1112. Otherwise, method1100returns to block1108in which a second set of paired images are analyzed, e.g., paired images corresponding to a subsequent frame.

At block1112, the steps described in reference to block1008are performed using the paired images and/or the matched features. Once EQ.1 is minimized, the value of rotation parameter R for which the equation is minimized is set and outputted as calibrated rotation parameter R′.

Returning to block1106, if it is determined at block1106that the second calibration process is to be performed, method1100proceeds to an online calibration pathway1152. Online calibration pathway1152may include multiple modules such as, for example, an environment reconciliation module1111and an online calibration module1113. Environment reconciliation module1111may include steps to ensure the 3D point cloud data collected by AR device200over a predetermined period of time is aligned. At block1114, bundle adjustment is performed by optimizing the sparse map (a group of map points and keyframe positions of AR device200) each time a keyframe has occurred. Accordingly, prior to performing any remaining steps at block1114, it may first be determined whether a keyframe has occurred. During operation of AR device200, a keyframe occurs when it is determined, based on sensor data220, that enough new information is present to warrant stable optimization, which corresponds to determining that AR device200has translated more than a translation threshold and rotated more than a rotation threshold (center point302being used as the location of AR device200). As an example, the translation threshold may be 10 cm and the rotation threshold may be 10 degrees.

The sparse map may comprise map points collected from sensors206. Map points may be captured by sensors206along different features in the field of view, and each map point is associated with the known position of AR device200(using center point302) when the map point was captured. This gives context to the map points that are collected such that a 3D model of the environment can be accurately reconstructed and properly interpreted. When bundle adjustment is performed, the sparse map is optimized by aligning the map points included in the sparse map using an algorithm that minimizes alignment error between points. After the sparse map is optimized, method1100proceeds to online calibration module1113.

Online calibration module1113may include several sub-processes or steps. At block1116, it is determined whether an online calibration trigger has been met. The online calibration trigger may include one or more conditions such as, but not limited to: whether a keyframe occurred, whether consecutive keyframes occurred, whether a bundle adjustment was successful, whether consecutive bundle adjustments were successful, whether the maximum distance between keyframe poses of AR device200has translated more than a threshold baseline (e.g., 1.5 meters), whether the maximum rotation between keyframe poses of AR device200has rotated more than a threshold angle (e.g., 90 degrees), whether detected features are evenly distributed across the field of view, whether detected features are evenly distributed in the z-dimension (corresponding to depth), and the like. If the one or more conditions included in the online calibration trigger are not met, then method1100returns to block1102. If the conditions are met, then method1100proceeds to block1118.

At block1118, the steps described in reference to block1010are performed using the optimized sparse map. In some embodiments, performance of block1118may use a larger data set than is used for bundle adjustment. For example, bundle adjustment may use the most recent10map point sets observed by the most recent10keyframes and camera pose positions to optimize the sparse map, while block1118may use the last100map point sets and camera pose positions. Once EQ.2 is minimized, the values of translation parameter T and rotation parameter R for which the equation is minimized are set and outputted as calibrated translation parameter T′ and calibrated rotation parameter R′, respectively.

At block1120, calibrated translation parameter T′ and calibrated rotation parameter R′ are compared to preselected acceptance criteria. In some embodiments, the acceptance criteria may require that calibrated translation parameter T′ and calibrated rotation parameter R′ be sufficiently different from translation parameter T and rotation parameter R, respectively. In some embodiments, the differences T′-T and R′-R may be compared to thresholds. If it is determined that the acceptance criteria is satisfied, method1100may proceed to block1122.

At block1122, either one or both of calibrated translation parameter T′ and calibrated rotation parameter R′ are used to replace and/or update translation parameter T and rotation parameter R, respectively. If epipolar calibration1150was performed, then rotation parameter R is replaced and/or updated. If online calibration pathway1152was performed, then both translation parameter T and rotation parameter R are replaced and/or updated. After performance of block1122, method1100returns to block1102, repeating the described steps.

In some embodiments, performance of method1100may include performing only epipolar calibration1150(i.e., the first calibration process) or only online calibration pathway1152(i.e., the second calibration process). In some embodiments, online calibration pathway1152is performed at a first time (i.e., t1) and epipolar calibration1150is performed at a second time (i.e., t2). Conversely, in some embodiments, epipolar calibration1150is performed at a first time (i.e., t1) and online calibration pathway1152is performed at a second time (i.e., t2). In some embodiments, online calibration pathway1152is performed two times consecutively (i.e., at times t1and t2) without performance of epipolar calibration1150. Similarly, in some embodiments, epipolar calibration1150is performed two times consecutively (i.e., at times t1and t2) without performance of online calibration pathway1152. As described herein, first time (i.e., t1) may precede or follow second time (i.e., t2).

One of skill in the art will appreciate that calibrated translation parameter T′ and calibrated rotation parameter R′ may be used by AR device200in many ways. In one embodiment, T′ and R′ may be used as a basis for physically adjusting a position and/or an orientation of one or more sensors (e.g., sensors206). Adjusting a position and/or an orientation of one or more sensors may improve overall performance of AR device200by controlling the position and/or orientation of at least one sensor relative to other components and/or other sensors on AR device200.

FIG. 12illustrates various steps for detecting one or more matched features1202between a left image1204and a right image1206(i.e., paired images), according to some embodiments of the present invention. For example,FIG. 12may illustrate one or more steps in connection with block1108as described in reference toFIG. 11. Each detected matched feature in left image1204maps to a detected matched feature in right image1206, and vice-versa. Matched features may be detected based on corner detection techniques or any one of various conventional image processing techniques.

FIG. 13illustrates various steps for partitioning a left image1304and a right image1306into a plurality of bins1308and for determining the quantity of matched features1302located in each of bins1308, according to some embodiments of the present invention. For example,FIG. 13may illustrate one or more steps in connection with blocks1108and1110as described in reference toFIG. 11. Left image1304and right image1306may be camera images, depth images, among other possibilities. In the particular implementation shown inFIG. 13, each of left image1304and right image1306are partitioned into 9 bins in a 3×3 arrangement. In other embodiments, different numbers of bins and different arrangements of the bins are possible. For example, each of left image1304and right image1306may be partitioned into any number of bins (e.g., 4, 16, 25, 36, etc.) having various shapes (e.g., rectangular, triangular, circular, etc.). Bins may be overlapping or non-overlapping, and the arrangements of bins for left image1304and right image1306need not be identical. For example, left image1304may be partitioned into 4 bins in a 2×2 arrangement and right image1306may be partitioned into 6 bins in a 2×3 arrangement.

One or more feature thresholds may be defined that require a certain quantity of matched features to be present in each bin and/or in a group of bins. By way of example, feature thresholds may require that each of bins1308include 5 or more of matched features1302. In the illustrated embodiment, the quantity of matched features in each bin is indicated in the parentheses to the right of the bin number. Because several of bins1308fail to meet the feature threshold (e.g., Bins4,7,8,9,12,14,15,17, and18each have fewer than 5 matched features), the feature thresholds are not satisfied. As a result, the current image pair, images1304and1306, may optionally be discarded while the corresponding features from the image pair may be retained. As subsequent image pairs are retrieved and analyzed in the same manner, features are accumulated until each of the feature thresholds are satisfied. As another example, feature thresholds may require that each grouping of 4 adjacent bins in a 2×2 arrangement include 10 or more matched features. Because the grouping of Bins14,15,17, and18only contains 5 matched features, the feature thresholds are not satisfied.

FIGS. 14A and 14Billustrate various steps for partitioning a left image1404and a right image1406into a plurality of bins1408in three-dimensional space and for determining the quantity of matched features1402located in each of bins1408, according to some embodiments of the present invention. For example,FIGS. 14A and 14Bmay illustrate one or more steps in connection with blocks1108and1110as described in reference toFIG. 11. Left image1404and right image1406may be camera images, depth images, among other possibilities. Each of left image1404and right image1406are partitioned into 27 bins in a 3×3×3 arrangement. In other embodiments, different numbers, arrangements, and shapes of bins are possible. Bins may be overlapping or non-overlapping, and the arrangements of bins for left image1404and right image1406need not be identical.

In reference toFIG. 14A, feature thresholds are defined for groups of bins with each group comprising the bins that form a plane that extends in two of the three dimensions. For example, feature thresholds require that the groups of bins forming planes in the near field, the mid-field, and the far field (with respect to the Z dimension) each include 5 or more of matched features1402. Feature thresholds also require that the groups of bins forming planes with respect to the X dimension each include 8 or more matched features and that the group of bins forming planes with respect to the Y dimension each include 7 or more matched features.FIG. 14Billustrates additional feature thresholds that require that each individual bin include 2 or more matched features. Accordingly, feature thresholds may be defined for individual bins and/or groups of bins to ensure an adequate spatial distribution of matched features1402.

FIG. 15illustrates various steps for performing bundle adjustment, according to some embodiments of the present invention. Shown inFIG. 15are map points1502viewed by various camera poses1504. Map points1502are captured by sensors206A,206B (and in some embodiments, sensors206C,206D) along different features in the field of view, and each of map points1502is associated with the known position of AR device200(center point302) when the map point was captured. Collectively, map points1502as viewed from camera poses1504make up the sparse map. The sparse map is optimized by aligning the calculated projection of the map points included in the sparse map with the corresponding observed feature of the map points using an algorithm that minimizes alignment error between the calculated projection and the observed feature.

While the forgoing description has been given in reference to AR device200and model300, other system configurations may also benefit from the calibration method described. For example, any device having two sensors with at least partially overlapping fields of view may be calibrated using the described model. The two sensors may be located on a same side of a device or on different sides. The two sensors may be displaced from each other in any of x-, y-, and z-dimensions, or a combination thereof. Additional sensors may be added to the system and calibrated using the methods disclosed. The additional sensors need not have overlapping fields of view with the first two sensors. It will be appreciated that two, three, four, or more additional sensors may be added to the system and can be calibrated using the method described.

FIG. 16illustrates a simplified computer system1600, according to an embodiment of the present invention. A computer system1600as illustrated inFIG. 16may be incorporated into devices such as AR device200as described herein.FIG. 16provides a schematic illustration of one embodiment of a computer system1600that can perform some or all of the steps of the methods provided by various embodiments. It should be noted thatFIG. 16is meant only to provide a generalized illustration of various components, any or all of which may be utilized as appropriate.FIG. 16, therefore, broadly illustrates how individual system elements may be implemented in a relatively separated or relatively more integrated manner.

The computer system1600is shown comprising hardware elements that can be electrically coupled via a bus1605, or may otherwise be in communication, as appropriate. The hardware elements may include one or more processors1610, including without limitation one or more general-purpose processors and/or one or more special-purpose processors such as digital signal processing chips, graphics acceleration processors, and/or the like; one or more input devices1615, which can include without limitation a mouse, a keyboard, a camera, and/or the like; and one or more output devices1620, which can include without limitation a display device, a printer, and/or the like.

The computer system1600might also include a communications subsystem1630, which can include without limitation a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and/or a chipset such as a Bluetooth™ device, an 802.11 device, a WiFi device, a WiMax device, cellular communication facilities, etc., and/or the like. The communications subsystem1630may include one or more input and/or output communication interfaces to permit data to be exchanged with a network such as the network described below to name one example, other computer systems, television, and/or any other devices described herein. Depending on the desired functionality and/or other implementation concerns, a portable electronic device or similar device may communicate image and/or other information via the communications subsystem1630. In other embodiments, a portable electronic device, e.g. the first electronic device, may be incorporated into the computer system1600, e.g., an electronic device as an input device1615. In some embodiments, the computer system1600will further comprise a working memory1635, which can include a RAM or ROM device, as described above.

The computer system1600also can include software elements, shown as being currently located within the working memory1635, including an operating system1640, device drivers, executable libraries, and/or other code, such as one or more application programs1645, which may comprise computer programs provided by various embodiments, and/or may be designed to implement methods, and/or configure systems, provided by other embodiments, as described herein. Merely by way of example, one or more procedures described with respect to the methods discussed above, such as those described in relation toFIG. 16, might be implemented as code and/or instructions executable by a computer and/or a processor within a computer; in an aspect, then, such code and/or instructions can be used to configure and/or adapt a general purpose computer or other device to perform one or more operations in accordance with the described methods.

As mentioned above, in one aspect, some embodiments may employ a computer system such as the computer system1600to perform methods in accordance with various embodiments of the technology. According to a set of embodiments, some or all of the procedures of such methods are performed by the computer system1600in response to processor1610executing one or more sequences of one or more instructions, which might be incorporated into the operating system1640and/or other code, such as an application program1645, contained in the working memory1635. Such instructions may be read into the working memory1635from another computer-readable medium, such as one or more of the storage device(s)1625. Merely by way of example, execution of the sequences of instructions contained in the working memory1635might cause the processor(s)1610to perform one or more procedures of the methods described herein. Additionally or alternatively, portions of the methods described herein may be executed through specialized hardware.

The communications subsystem1630and/or components thereof generally will receive signals, and the bus1605then might carry the signals and/or the data, instructions, etc. carried by the signals to the working memory1635, from which the processor(s)1610retrieves and executes the instructions. The instructions received by the working memory1635may optionally be stored on a non-transitory storage device1625either before or after execution by the processor(s)1610.

As used herein and in the appended claims, the singular forms “a”, “an”, and “the” include plural references unless the context clearly dictates otherwise. Thus, for example, reference to “a user” includes a plurality of such users, and reference to “the processor” includes reference to one or more processors and equivalents thereof known to those skilled in the art, and so forth.