Automatic calibration of scene camera for optical see-through head mounted display

An apparatus for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD) obtains eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device, and obtains object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system. The apparatus calculates screen coordinates in a screen coordinate system corresponding to a display point on the HMD, where the calculating is based on the obtained eye coordinates and the obtained object coordinates. The apparatus calculates calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system. The apparatus then derives subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

BACKGROUND

The present disclosure relates generally to augmented reality (AR) devices, e.g., AR eyeglasses, having optical see-through head mounted displays (HMD) and eye tracking capability, and more particularly, to automatic calibration of the scene camera of such AR devices. AR is a technology in which a user's view of the real world is enhanced with additional information generated from a computer model. The enhancements may include labels, 3D rendered models, or shading and illumination changes. AR allows a user to work with and examine the physical real world, while receiving additional information about the objects in it.

AR devices typically include an optical see-through HMD and one or more user input mechanisms that allow users to simultaneously see and interact with their surroundings while interacting with applications, such as e-mail and media players. User input mechanisms may include one or more of gesture recognition technology, and eye tracking technology. AR devices also allow a user to view real-world scenes through optical see-through HMDs together with two-dimensional (2D) or three-dimensional (3D) augmented reality content displayed on the HMDs.

It is difficult for the user of an AR device with optical see-through HMDs to see 3D augmented reality that is well aligned with markers or objects in the real world for each eye. In order to see a well-aligned augmented reality on each eye, a scene camera of the AR device needs to be calibrated for each eye of the user. Existing scene camera calibration processes are very cumbersome. Users typically have to go through a 12-step calibration process for each eye to be able to see a well-aligned augmented reality. After initial calibration, further calibration may be necessary if the user repositions the AR device on his face

Furthermore, since the eye distance between a user's eye and the HMD varies among users, the calibrated data, also referred to as “projection matrix,” for one user does not work well for other users. Accordingly, while one user of an AR device may see 3D augmented reality aligned with the real world object, another user of the same AR device may not have the same experience. As such, each individual user of an AR device needs to go through the cumbersome calibration steps for experiencing AR properly aligned with real world. In the best case, relying on pre-existing calibration data, users still need 4-step calibration process.

SUMMARY

In an aspect of the disclosure, a method, an apparatus, and a computer program product for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD) are disclosed. An example apparatus obtains eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device, and obtains object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system. The apparatus calculates screen coordinates in a screen coordinate system corresponding to a display point on the HMD, where the calculating is based on the obtained eye coordinates and the obtained object coordinates. The apparatus calculates calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system. The apparatus then derives subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

DETAILED DESCRIPTION

FIG. 1is an illustration of an example AR device100in the form of a pair of eyeglasses. The AR device100is configured such that the user of the device is able to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs, including both two-dimensional (2D) and three-dimensional (3D) AR content. The AR device100may also be configured to allow the user to interact with the content and possibly with remote devices, systems or networks through wireless communication. The AR device may also provide feedback to the user as a result of such interactions, including for example, audio, video or tactile feedback. To these ends, the example AR device100includes a pair of optical see-through HMDs102,104, an on-board processing system106, one or more sensors, such as a scene camera108, one or more eye tracking components (not visible) for each of the right eye and left eye, one or more user-interaction feedback devices110and a transceiver112.

The processing system106and the eye tracking components provide eye tracking capability. Depending on the eye tracking technology being employed, eye tracking components may include one or both of eye cameras and infra-red emitters, e.g. diodes. The processing system106and the scene camera108provide gesture tracking capability.

The feedback devices110provide perception feedback to the user in response to certain interactions with the AR device. Feedback devices110may include a speaker or a vibration device. Perception feedback may also be provided by visual indication through the HMD.

The transceiver112facilitates wireless communication between the processing system106and remote devices, systems or networks. For example, the AR device may communicate with remote servers through the transceiver112for purposes of remote processing, such as on-line searches through remote search engines.

As mention above, the AR device100allows a user to view real-world scenes through optical see-through HMDs together with content displayed on the HMDs. For example, with reference toFIG. 2, as a user is viewing a real-world scene200through the optical see-through HMDs102,104, the scene camera108may capture an image of the scene and send the image to the on-board processing system106. The processing system106may process the image and output AR content202for display on the HMDs102,104. The content202may provide information describing what the user is seeing. In some cases, the processing system106may transmit the image through the transceiver112to a remote processor (not shown) for processing. The processing system106may also display one or more application icons204,206,208on the HMDs102,104and output application content, such as e-mails, documents, web pages, or media content such as video games, movies or electronic books, in response to user interaction with the icons.

User interaction with the AR device100is provided by one or more user input mechanisms, such as a gesture tracking module or an eye-gaze tracking module. Gesture tracking is provided by the scene camera108in conjunction with a gesture tracking module of the processing system106. With gesture tracking, a user may attempt to activate an application by placing his finger on an application icon204,206,208in the field of view of the AR device. The scene camera108captures an image of the finger and sends the image to the gesture tracking module. The gesture tracking module processes the image and determines coordinates of a gesture point corresponding to where the user is pointing. The processing system106compares the coordinate location of the gesture point to the coordinate location of the icon on the display. If the locations match, or are within a threshold distance of each other, the processing system106determines that the user has selected the icon204,206,208and accordingly, launches the application.

Eye-gaze tracking is provided by the eye tracking components (not visible) in conjunction with an eye tracking module of the processing system106. A user may attempt to activate an application by gazing at an application icon204,206,208in the field of view of the AR device. The eye tracking components capture images of the eyes, and provide the images to the eye tracking module. The eye tracking module processes the images and determines coordinates of an eye-gaze point corresponding to where the user is looking. The processing system106compares the coordinate location of the eye-gaze point to the coordinate location of the icon on the display. If the locations match, or are within a threshold distance of each other, the processing system106determines that the user has selected the icon204,206,208and accordingly, launches the application. Often, such eye-gaze based launching is coupled with another form of input, e.g., gesture, to confirm the user's intention of launching the application.

FIG. 3is a diagram illustrating elements of an example AR device300with optical see-through HMDs302. The AR device300may include one or more sensing devices, such as infrared (IR) diodes304facing toward the wearer of the AR device and eye cameras306facing toward the wearer. A scene camera308facing away from the wearer captures images of the field of view seen by the user through the HMD302. The cameras306,308may be video cameras. While only one IR diode304and one eye camera306are illustrated, the AR device300typically includes several diodes and cameras for each of the left eye and right eye. A single scene camera308is usually sufficient. For ease of illustration only one of each sensor type is shown inFIG. 3.

The AR device300includes an on-board processing system310, which in turn includes one or more of an eye tracking module312and a gesture tracking module314. The object selection processor316functions to determine whether interactions of the user, as characterized by one or more of the eye tracking module312and the gesture tracking module314, correspond to a selection of an object, e.g., application icon, displayed on the HMD302and visible in the field of view. If an interaction does correspond to a selection by the user, for example, a selection of an icon to launch an application334, the object selection processor316outputs a command to the application. A tracking calibration module318calibrates the one or more tracking modules if the tracking module is determined to be inaccurate.

The on-board processing system310may also include a scene camera/AR calibration module320, a graphical user interface (GUI) adjustment module322, and a perception feedback module324. As described further below, the scene camera/AR calibration module320calibrates the AR device so that AR content displayed on the optical see-through HMD302is aligned with real world objects seen through the HMD. The GUI adjustment module322may adjust the parameters of GUI objects displayed on the HMD to compensate for eye-tracking or gesture-tracking inaccuracies detected by the object selection module316. Such adjustments may precede, supplement, or substitute for the actions of the tracking calibration module318. The feedback module324controls one or more feedback devices326to provide perception feedback to the user in response to one or more types of user interactions. For example, the feedback module a feedback device326to output sound when a user selects an icon in the field of view using a gesture or eye gaze.

The AR device300further includes memory328for storing program code to implement the foregoing features of the on-board processing system310. A communications module330and transceiver332facilitate wireless communications with remote devices, systems and networks. For example, in one implementation, an image of a real-world object may be captured by the scene camera308and transmitted by the communications module330and the transceiver332to a remote search engine, with subsequent search results being received by the transceiver.

With further respect to eye tracking capability, the diodes304and eye cameras306, together with the eye tracking module312, provide eye tracking capability as generally described above. In the example implementation ofFIG. 3, the eye tracking capability is based on known infrared technology. One such known technology uses infrared light emitting diodes and infrared sensitive video camera for remotely recording images of the eye. Infrared light output by the diode304enters the eye and is absorbed and re-emitted by the retina, thereby causing a “bright eye effect” that makes the pupil brighter than the rest of the eye. The infrared light also gives rise to an even brighter small glint that is formed on the surface of the cornea. The eye tracking module312acquires a video image of the eye from the eye camera306, digitizes it into a matrix of pixels, and then analyzes the matrix to identify the location of the pupil's center relative to the glint's center, as well as a vector between these centers. Based on the determined vector, the eye tracking module312outputs eye gaze coordinates defining an eye gaze point (E).

As mentioned above, AR devices having optical see-through HMDs require calibration in order to render AR content that is properly aligned with real-world object. The state of the art in AR calibration generally requires at least a twelve step calibration process for both eyes to obtain calibration data, e.g., a projection matrix (P), and thus exhibits cumbersome user experience. Further, projection matrixes (P) are user specific so that a projection matrix of one user does not work well for other users. Therefore, individual user needs to go through such cumbersome calibration steps for experiencing AR properly aligned with real world.

Disclosed herein is an automatic AR device calibration approach where the user does not need to perform any manual calibration steps. In this approach, an entire projection matrix (P) for positioning AR on an HMD so as to be properly aligned with real-world objects is computed. The projection matrix (P) is computed on the fly based on the eye position of a user as provided by the eye tracking module312, known coordinates of the real-world object as provided by a model view matrix (M), and known transformations of different coordinate systems, such as the scene camera308coordinate system, the eye camera306coordinate system, and the HMD302coordinate system. These coordinates and transformations may be provided by functional modules of the AR calibration module320. In another possible approach, only a few of the parameters in the projection matrix (P) are determined for a current user of the AR device, with the remaining parameters being carried over from a prior user of the AR device.

FIG. 4is an illustration400of a pinhole camera model, showing a pinhole camera converting a 3D world point pworld=[X Y Z 1]Tto a 2D image point pscreen=[u v 1]T. The calibration approach disclosed herein is based on the pinhole camera model. The image of a 3D point402, denoted by pworld=[X Y Z 1]T, is formed by an optical ray412from pworld402passing through the optical center C408and intersecting an image plane406at a 2D image point, denoted by pscreen=[u v 1]T. The three points pworld, pscreen, and C are collinear. For illustration purpose, the image plane406is positioned between the scene point402and the optical center408, which is mathematically equivalent to the physical setup under which the image plane is in the other side with respect to the optical center.

A simple pinhole model camera is used for describing the transformation from a real-world 3D scene defined by a plurality of 3D points402(only one is shown for clarity of illustration) defined in the world coordinate system410to a 2D screen image defined by a corresponding plurality of 2D image points404(only one is shown for clarity of illustration) on an image surface406.

The model is defined by a set of extrinsic parameters [R, t] and intrinsic parameters (A). Extrinsic parameters [R, t] define the camera position408and orientation with respect to the world coordinate system410and can be described with a transformation containing rotation (R) and translation (t) as shown in Eq. (1).

Intrinsic parameters (A) define the optical properties of the camera and cam be defined as shown in Eq. (2).

Intrinsic⁢⁢parameters=A=[αγu00βv0001]Eq.⁢(2)where:(u0, v0) are the coordinates of the principal point,α and β the scale factors in image u and v axes, andγ the parameter describing the skew of the two image axes.

InFIG. 4, the angle between the two image axes is denoted by θ, and we have γ=α cot θ. If the pixels are rectangular, then θ=90° and γ=0.

These intrinsic and extrinsic parameters define a camera's projection matrix (P) as shown in Eq. (3).
Camera projection matrix, P=A[R t]  Eq.(3)

Once this projection matrix (P) is obtained from calibration, a 2D point pscreen404on the image plane406can be computed from a world point402pworldusing Eq. (4).
spscreen=PpworldEq.(4)where s is an arbitrary scale factor.

The task of camera calibration is to determine the parameters of the transformation between an object in 3D space and the 2D image observed by the camera from visual information (images). The transformation includes the above described extrinsic parameters (sometimes called external parameters): orientation (rotation (R) parameters of Eq. 1) and location (translation (t) parameters of Eq. 1) of the camera, i.e., [R t], and the above described intrinsic parameters (sometimes called internal parameters).

The rotation matrix (R), although consisting of nine elements, only has three degrees of freedom. The translation vector t has 3 parameters. Therefore, there are six extrinsic parameters and five intrinsic parameters, leading a total of eleven parameters.

To be able to correctly merge the real and the virtual world during user interaction with a dynamic scene, an AR system maintains a computer model to represent the location of real and virtual objects. The spatial relationships are normally modeled using linear transformation matrices. As 4-by-4 matrices, they can be aggregated through multiplication to symbolize the traversal through local coordinate systems and so describe the exact location of surrounding objects relative to the user's eye.

FIG. 5is an illustration500of an HMD augmented reality system, showing the coordinate systems involved in a typical HMD AR system. Similar to the pinhole camera, the HMD AR system transforms a 3D world point502to a 2D screen point504. However, unlike the pinhole camera model, the HMD AR model consists of more than two coordinate systems. Here, a 3D world point502in the world coordinate system506is first converted into a target (i.e. image target) coordinate system508. The coordinates of the 3D world point in the target coordinates system508are then converted into a scene camera coordinate system510. The coordinates of the 3D world point in the scene camera coordinates system510are then converted into an eye coordinate system (monocular)512. Finally the coordinates of the 3D world point in the eye camera coordinate system512are converted into a screen coordinate system514.

The transformation across all five coordinate systems is as shown in Eq. (5). Here, and throughout all equations, Tx-yrepresents transformation from an x coordinate system to a y coordinate system.
pscreen=TE-STC-ETT-CTW-TpworldEq.(5)

Here, it is assumed that the world-to-target coordinate system is identity as shown in Eq. (6).
TW-T=I (assumption)  Eq.(6)

The transformation from target coordinate system508to scene camera coordinate system510can be obtained from a model view matrix (M) as shown in Eq. (7).
TT-C=M  Eq.(7)

Therefore, the transformation from scene camera coordinate system510to eye coordinate system512and from eye coordinate system512to screen coordinate system514in the HMD calibration process is estimated. Eq. (8) shows the final transformation from 3D world point502to 2D screen point504.
Need to estimate, P=TE-STC-EEq.(8)

A single point active alignment method (SPAAM) for monocular HMD calibration has been proposed. In this method, a user wearing the HMD aligns a 2D point shown on the display with a real world point in the real world coordinate system by moving her body and head. From a set of such correspondences, the projection matrix P is computed as shown in Eq. (9).
pscreen=PMpworldEq.(9)

Since a projection matrix contains eleven degrees of freedom (DOF), a user needs to perform at least six step calibration (each step provides two equations for x and y) for an eye, and repeat the same method for the other eye.

FIG. 6is an illustration600of a user602aligning augmented reality604with an image target606for conventional HMD calibration. This method draws a rectangle at a fixed screen position. The user wearing the HMD aligns the rectangle604with the AR target606. Once the rectangle604and target606are aligned, the user602taps the screen. This is repeated for 6 rectangles for each eye, all drawn at slightly different locations.

The formula for calculating the projection matrix is as follows:
PMV=C  Eq.(10)where:P=Projection matrixM=Model-view matrixV=3D vertices of the rectangle to be drawnC=screen coordinates of rectangle (on the image plane)

The static rectangle604is drawn with a static model view matrix (M), and the projection matrix (P) of the scene camera. When the user aligns the static rectangle604with the real world marker606, they are aligning the screen coordinates C with the real world marker606. When the alignment is done, the coordinates of the origin or center (C) of the screen is the same value as the center of the scene camera. Thus, a user's aligned model-view matrix (M) is used to calculate the user's projection matrix (P). Since the projection matrix (P) contains eleven degrees of freedom, six pairs of screen coordinates are used to calculate the projection matrix (P).

A decomposition of the user's projection matrix (P) calculated from this approach is as follows:
P=A[R t]  Eq.(11)where:[R t]=rotation matrix (R) and translation matrix (t) from the scene camera coordinate system510to the eye coordinate system512.A=projection from the 3D model of eye coordinate system512to 2D image plane516. A composes intrinsic parameters of the eye, including the focal length, and the image center.

Automatic Zero Step Calibration for HMD:

In the AR calibration approach disclosed herein, a reference user's projection matrix (P) is not relied on for computing a new user's projection matrix (P). Instead, the entire projection matrix (P) of the user is automatically computed. Such computation may occur each time a user puts on the AR device, or periodically while the user is already wearing the AR device. As mention above, the computation of the projection matrix (P) is based on the new user's eye position, real-world object coordinates obtained from a model view matrix (M), and transformations of different coordinate systems.

FIG. 7is an illustration700of the various coordinate systems with respect to an AR device having a right eye HMD702and a left eye HMD704. Each coordinate system has a corresponding origin and direction. The origin of the target coordinate system706is the center of the object in the real world that an AR is to be associated with. This object may also be referred to as the target or marker. The origin of the scene camera coordinate system708is the center of the scene camera308of the AR device. The origin of the left eye coordinate system710is the center, or eye-gaze base point, of the user's left eye as determined by the eye camera306of the left eye and the eye tracking module312. The origin of the right eye coordinate system712is the center, or eye-gaze base point, of the user's right eye as determined by the eye camera306of the right eye and the eye tracking module312. The origin of the left display coordinate system714is the center of the left HMD704. The origin of the right display coordinate system716is the center of the right HMD702.

FIG. 8is an illustration800of aspects of a calibration process for a single display808. Coordinates of an eye-gaze base point802, generally corresponding to the center of the user's eye, are obtained from the eye tracking module312. Coordinates (pworld) of a real-world target804, also referred to as an object or marker, are obtained based on a model view matrix (M). As noted above by Eq. (7), the model view matrix (M) corresponds to a transformation from a target coordinate system706to a scene camera coordinate system708. The model view matrix (M) is derived using known processes and is provided to the AR device for use in the calibration process.

Once the coordinates of the eye-gaze base point802and the coordinates (pworld) world) of the target804are obtained, a ray806is defined. The ray806originates from the eye-gaze base point802, intersects the display plane808and terminates at the target coordinates (pworld)804. In terms of processing, a set of liner equations is solved to find the intersection point810of the ray806and the display808. The coordinates of the intersection point corresponds to pscreen. A projection matrix (P) is determined as follows:
P=Mpworld/pscreenEq.(12)where:M is the known model view matrixpworldare the coordinates of the target804, andpscreenare the coordinates of the intersection point810.

Once the projection matrix (P) is determined, subsequent display points pscreenfor other real-world object points may be determined using the above Eq. 9.

FIG. 9is a flow chart900of a method of calibrating an AR device having an optical see-through HMD. The method may be performed by an AR device. In step902, the AR device obtains eye coordinates802in an eye coordinate system corresponding to a location of an eye of a user of the AR device.

In step904, the AR device obtains object coordinates804in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device. The real-world object may be captured by a scene camera having a scene camera coordinate system.

In step906, the AR device calculates screen coordinates810in a screen coordinate system corresponding to a display point on the HMD. The calculating is based on the eye coordinates and the object coordinates. In one configuration, the AR device calculates screen coordinates by defining a ray that originates from the eye coordinates, intersects the HMD and terminates at object coordinates; and calculating an intersection point of the ray and the HMD, the intersection point being the display point.

In step908, the AR device calculates calibration data based on the screen coordinates, the object coordinates, and a transformation from the target coordinate system to the scene camera coordinate system. The transformation from the target coordinate system to the scene camera coordinate system may be characterized provided by a model view matrix (M), and the calibration data may correspond to a projection matrix (P). In step910, the AR devices subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

FIG. 10is a diagram1000illustrating elements of an AR device1002that provides AR calibration. The AR device1002includes an eye coordinate obtaining module1004, an object coordinate obtaining module1006, a screen coordinate calculation module1008, a calibration data calculation module1010, and an AR display coordinate module1012. These modules may correspond to one or more of the modules ofFIG. 3. For example, all of the foregoing modules may be part of the AR calibration and AR display module320.

The eye coordinate obtaining module1004obtains eye coordinates in an eye coordinate system. The eye coordinates correspond to a location of an eye of a user of the AR device. The eye coordinates may be obtained from an eye tracking module312.

The object coordinate obtaining module1006obtains object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device. The real-world object may be captured by a scene camera308having a scene camera coordinate system. Objects within the scene data are detected and their coordinates are determined based on a transformation from the target coordinate system to the scene camera coordinate system. Such transformation may be based on a known model view matrix (M).

The screen coordinate calculation module1008calculates screen coordinates in a screen coordinate system corresponding to a display point on the HMD302. The calculating may be based on the eye coordinates obtained by the eye coordinate obtaining module1004and the object coordinates obtained by the object coordinate obtaining module1006.

The calibration data calculation module1010calculates calibration data based on the screen coordinates, the object coordinates and the transformation from the target coordinate system to the scene camera coordinate system. The AR display coordinate module1012derives subsequent screen coordinates for the display of AR in relation to other real-world object points or target points based on the calibration data. The calibration data may be a projection matrix (P).

The AR device, as illustrated inFIGS. 3 and 10may include additional modules that perform each of the steps of the algorithm in the aforementioned flow chart ofFIG. 9. As such, each step in the aforementioned flow chart ofFIG. 9may be performed by a module and the apparatus may include one or more of those modules. The modules may be one or more hardware components specifically configured to carry out the stated processes/algorithm, implemented by a processor configured to perform the stated processes/algorithm, stored within a computer-readable medium for implementation by a processor, or some combination thereof.

FIG. 11is a diagram1100illustrating an example of a hardware implementation for an apparatus1002′ employing a processing system1114. The processing system1114may be implemented with a bus architecture, represented generally by the bus1124. The bus1124may include any number of interconnecting buses and bridges depending on the specific application of the processing system1114and the overall design constraints. The bus1124links together various circuits including one or more processors and/or hardware modules, represented by the processor1104, the modules1004,1006,1008,1010,1012and the computer-readable medium/memory1106. The bus1124may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.

The processing system1114includes a processor1104coupled to a computer-readable medium/memory1106. The processor1104is responsible for general processing, including the execution of software stored on the computer-readable medium/memory1106. The software, when executed by the processor1104, causes the processing system1114to perform the various functions described supra for any particular apparatus. The computer-readable medium/memory1106may also be used for storing data that is manipulated by the processor1104when executing software. The processing system further includes at least one of the modules1004,1006,1008,1010and1012. The modules may be software modules running in the processor1104, resident/stored in the computer readable medium/memory1106, one or more hardware modules coupled to the processor1104, or some combination thereof.

In one configuration, the apparatus1002/1002′ includes means for obtaining eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device, means for obtaining object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system, means for calculating screen coordinates in a screen coordinate system corresponding to a display point on the HMD, the calculating being based on the eye coordinates and the object coordinates, means for calculating calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system, and means for deriving subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data. The aforementioned means may be one or more of the aforementioned modules of the apparatus1002and/or the processing system1114of the apparatus1002′ configured to perform the functions recited by the aforementioned means.

In summary, AR device calibration achieved by computing an entire projection matrix on the fly based on a user's eye position as provided by an eye tracking module, and know transformations of different coordinate system, such as scene camera, eye camera and display. In an aspect of the disclosure, a method, an apparatus, and a computer program product for automatically calibrating a scene camera of a head mounted display. The method uses the eye gaze base point readings to calculate the user's projection matrix for each eye in real time. Then the projection matrix and the position of real-world marker/object together determine the coordinates of the augmented reality on the display:A) The eye gaze base point readings are obtained in real-time. The position of a marker/object is known.B) A ray passing through the eye gaze base point and the object/marker center intersects with the display.C) The intersection points are calculated repeatedly for several different points and are used for the calculation of projection matrix for each eye.D) The projection matrix for each eye is updated when the eye gaze base point changes, e.g., when the glasses move.

The advantage of this method is that: There is no scene camera calibration process required, as compared to the traditional 12-step calibration process. The traditional scene camera calibration method works for only one glasses position: if the glasses are moved on the nose, users will not see a well-aligned AR. The above method utilizes the real-time eye gaze base point reading, which will get updated if the glasses are moved. Therefore, the augmented reality rendering result is robust to glasses movement.

A method of calibrating a scene camera of a head mounted display (HMD) with eye tracking sensors worn by a user includes obtaining an eye gaze base point of the user when a target/marker/object is visible or virtually exists to the user through the HMD, calculating an intersection point of a ray with a display plane corresponding to the head mounted display, the ray passing through the eye and the first of the plurality of targets/markers/objects, repeating the obtaining and calculating for a plurality of different targets/markers/objects, and calculating a projection matrix for each eye based on the plurality of intersection points.

An apparatus for calibrating a scene camera of a head mounted display (HMD) with eye tracking sensors worn by a user includes means for obtaining an eye gaze base point of the user when a target/marker/object is visible or virtually exists to the user through the HMD, means for calculating an intersection point of a ray with a display plane corresponding to the head mounted display, the ray passing through the eye and the first of the plurality of targets/markers/objects, means for repeating the obtaining and calculating for a plurality of different targets/markers/objects, and means for calculating a projection matrix for each eye based on the plurality of intersection points.

Another apparatus for calibrating a scene camera of a head mounted display (HMD) with eye tracking sensors worn by a user, includes a memory; and at least one processor coupled to the memory and configured to obtain an eye gaze base point of the user when a target/marker/object is visible or virtually exists to the user through the HMD, calculate an intersection point of a ray with a display plane corresponding to the head mounted display, the ray passing through the eye and the first of the plurality of targets/markers/objects, repeat the obtaining and calculating for a plurality of different targets/markers/objects, and calculate a projection matrix for each eye based on the plurality of intersection points.

A computer program product for calibrating a scene camera of a head mounted display (HMD) with eye tracking sensors worn by a user, includes a computer-readable medium comprising code for obtaining an eye gaze base point of the user when a target/marker/object is visible or virtually exists to the user through the HMD, calculating an intersection point of a ray with a display plane corresponding to the head mounted display, the ray passing through the eye and the first of the plurality of targets/markers/objects, repeating the obtaining and calculating for a plurality of different targets/markers/objects, and calculating a projection matrix for each eye based on the plurality of intersection points.