Patent Description:
Mixed reality (MR) is foreseen to become an integral technology in the networked society and potently disrupt the consumer electronics market. Mixed reality encompasses Augmented Reality (AR) and Augmented Virtuality (AV).

AR is oftentimes performed via handheld devices, such as smartphones and tablets, or via Head-Mounted Displays (HMD) (also known as Head-Mounted Display Devices) such as Microsoft HoloLens. AR via an HMD implies layering information such as text, images, or videos, on top of the user's view of the real world via a see-through display.

Display devices with see-through displays, such as HMDs, require calibration in order to provide an immersive user experience. The objective of calibration is to be able to display virtual objects, such as text, images, and videos, on the see-through display of the HMD such that the displayed virtual objects are correctly aligned with real-world objects seen by the user through the see-through display. In practice, the user's eye pose must be known, in addition to the pose of the display and the respective pose of any camera comprised in the HMD, such as a front-facing camera for capturing images of the real-world and/or tracking real-world objects, and an eye-facing camera for tracking the user's eyes.

An overview of known calibration methods for HMDs has been given by <NPL>). Such calibration methods are typically performed manually when the HMD is in use, while some of the calibration methods can be automated. Common to some known calibration methods is that they rely on non-practical calibration rigs and/or expensive calibration equipment.

For instance, a manual calibration procedure for establishing a view matrix that is calibrated to the user's specific eye characteristics is disclosed in <CIT>.

Also known are calibration methods based on corneal imaging, relying on an eye-facing camera for imaging a reflection of the real-world scene by the user's cornea, as described in "<NPL>).

<CIT> discloses solutions for recalibration of outward-facing cameras supported by a see-through, head-mounted, mixed-reality display system having a flexible portion between see-through displays for the eyes. Each outward facing camera has a fixed spatial relationship with a respective or corresponding see-through display positioned to be seen through by a respective eye. For front facing cameras, the fixed spatial relationship allows a predetermined mapping between positions on an image sensor of each camera and positions on the respective display. The mapping may be used to register a position of a virtual object to a position of a real object. A change in a first flexible spatial relationship between the outward facing cameras can be automatically detected. A second spatial relationship between the cameras is determined. A registration of a virtual object to a real object may be updated based on the second spatial relationship.

<CIT> discloses solutions for enabling user interaction with virtual objects generated in virtual space on a first display device. Using sensor and camera data of the first display device, a real-world object with a marker on its surface is identified. Virtual objects are generated and displayed in the virtual 3D space relative to the marker on the real-world object. Manipulation of the real-world object in real 3D space results in changes to attributes of the virtual objects in the virtual 3D space. The marker comprises information regarding particular the renders to be generated. Different virtual objects can be generated and displayed based on information comprised in the markers. When the real-world object has sensors, sensor data from the real-world object is transmitted to the first display device to enhance the display of the virtual object, or the virtual scene, based on sensor input. Local or remote storage can further define, enhance, or modify characteristics of the real-world object.

It is an object of the invention to provide an improved alternative to the above techniques and prior art.

More specifically, it is an object of the invention to provide improved solutions for calibrating a see-through head-mounted display device. In particular, it is an objective of the invention to provide solutions for automatically selecting a calibration object for use in a calibration procedure.

These and other objects of the invention are achieved by means of different aspects of the invention, as defined by the independent claims. Embodiments of the invention are characterized by the dependent claims.

According to a first aspect of the invention, a head-mounted display device is provided. The display device is configured to be worn by a user and comprises an at least partially see-through display, a front-facing camera, a wireless-communications interface, and processing means. The front-facing camera is operative to capture a first image of a real-world scene. Typically, when worn by the user, the captured real-world scene is the scene in front of the user. The processing means is operative to select a calibration object from one or more real-world objects, or parts thereof, which are visible in the first image, by identifying a communications device comprising a display among the one or more real-world objects and selecting as the calibration object at least one of the display of the communications device and graphical content displayed thereon. The communications device comprising a display is identified using the wireless-communications interface. The processing means is further operative to derive a calibration transformation for calculating a display position based on a real-world position, such that a virtual object which is displayed on the display at the display position is aligned with a corresponding real-world object located at the real-world position, as seen by the user. The processing means is further operative to receive a representation of the displayed graphical content from the communications device via the wireless-communications interface. The calibration transformation may, e.g., be represented by a matrix or other form of mathematical representation which is suitable for describing a transformation between two coordinate systems.

According to a second aspect of the invention, a method performed by a head-mounted display device is provided. The display device is configured to be worn by a user. The method comprises selecting a calibration object from one or more real-world objects, or parts thereof, which are visible in a first image of a real-world scene. The first image is captured by a front-facing camera comprised in the display device. The calibration object is selected by identifying a communications device comprising a display among the one or more real-world objects and selecting as the calibration object at least one of the display of the communications device and graphical content displayed thereon. The communications device comprising a display is identified using a wireless-communications interface comprised in the display device. The method further comprises deriving a calibration transformation for calculating a display position based on a real-world position, such that a virtual object which is displayed at the display position on an at least partially see-through display is aligned with a corresponding real-world object located at the real-world position, as seen by the user. The at least partially see-through display is comprised in the display device. The method further comprises receiving a representation of the displayed graphical content from the communications device via the wireless-communications interface.

According to a third aspect of the invention, a computer program is provided. The computer program comprises computer-executable instructions for causing a head-mounted display device to perform the method according to an embodiment of the second aspect of the invention, when the computer-executable instructions are executed on a processing unit comprised in the display device.

According to a fourth aspect of the invention, a computer-readable storage medium is provided. The computer-readable storage medium has the computer program according to the third aspect of the invention stored thereon.

The invention makes use of an understanding that an improved calibration procedure for see-through Head-Mounted Displays or Display Devices (HMDs) is achieved by automatically selecting of a calibration object among one or more real-world objects for use in a calibration procedure, e.g., any of the calibration procedures which are known in the art. Thereby, the user of an HMD is alleviated from keeping, or carrying, a dedicated calibration object. Embodiments of the invention select a suitable calibration object among one or more real-world objects, i.e., physical objects in the surrounding of the user which are visible in the first image which is captured by the front-facing camera. These are real-world objects which are in the field-of-view of the front-facing camera, such as furniture, household appliances, buildings, doors, windows, vehicles, street signs, tablets, smartphones, laptops, checkerboards, and so forth.

According to an embodiment of the invention, the display device further comprises an eye-facing camera which is operative to capture a second image of a reflection of the real-world scene by a cornea of the user. The processing means is operative to select the calibration object from one or more real-world objects, or parts thereof, which are visible in both the first image and the second image, and to derive the calibration transformation using the first image and the second image. This embodiment of the invention relates to calibration methods which rely on corneal imaging. Optionally, the calibration object is selected from one or more real-world objects, or parts thereof, which are visible in a region of the first image which corresponds to a field-of-view of the eye-facing camera. Typically, the field of view of the eye-facing camera is smaller than that of the front-facing camera. Since the calibration object needs to be visible in both the first image and the second image, in order to derive the calibration transformation as part of a calibration procedure relying on corneal imaging, the calibration object is advantageously selected among the real-world objects which are visible in a part of the first image which corresponds to the field-of-view of the eye-facing camera. In practice, these are objects which are also visible in the second image captured by the eye-facing camera. Advantageously, since the first image (captured by the front-facing camera) is typically superior to the second image (captured by the eye-facing camera), owing to the imperfections of the cornea, identifying real-world objects by image processing the first image is easier, more reliable, and less resource consuming, as compared to the second image.

According to an embodiment of the invention, the calibration object may be selected based on a distortion of the calibration object in the second image as compared to the first image. This is the case if a calibration method which relies on corneal imaging is employed. Preferably, the object with least distortion is selected. This is advantageous as the calibration transformation is derived by comparing the same object as captured by the first and the second image.

According to an embodiment of the invention, the calibration object is selected based on a visual appearance of the calibration object. For instance, real-world objects which are clearly visible and/or have high contrast, and which accordingly are easy to detect by image processing or object recognition, are preferably selected.

According to an embodiment of the invention, a previously selected calibration object among the one or more real-world objects is selected as the calibration object. This may be achieved by maintaining a database of used calibration objects.

According to an embodiment of the invention, the display device comprises one or more motion sensors which are operative to track a motion of the display device. The processing means is operative to estimate a duration of time during which the one or more real-world objects remain visible in the first image, and select the calibration object based on the estimated duration of time during which the calibration object remains visible in the first image. The duration of time during which the one or more real-world objects are visible in the first image is estimated based on the tracked motion of the display device. Preferably, a real-world object is selected which is visible in the first image, and optionally in the second image, for a duration of time sufficiently long to perform the calibration procedure. As an alternative, a real-world object which is moving slowly, or not moving at all, relative to the field-of-view, may be selected.

According to an embodiment of the invention, the calibration object is selected by identifying one or more real-world objects by matching visual features of the one or more real-world objects against information pertaining to visual features of real-world objects which is stored in a database, and selecting the calibration object based on information obtained from the database. The information obtained from the database may indicate a respective suitability of the identified real-world objects, or parts thereof, as calibration object. The information may, e.g., relate to visual appearance, dimensions, composition of the real-world object in terms of geometrical shapes, or the like. In this respect, a real-world object is considered more suitable as calibration object if its visual appearance is characterized by a high contrast or by geometrical shapes which are easy to detect by image processing. For instance, a checkerboard is characterized by being composed of simple geometrical shapes and by high contrast. Preferably, the most suitable calibration object is selected. Communications device comprising a display may, e.g., be identified by establishing wireless communications with nearby communications devices, by querying the type, capabilities, or make/model, of the nearby communications devices. Advantageously, a nearby communications device which is in the field-of-view of the front-facing camera may display graphical content in the form of a dedicated calibration pattern which is characterized by high contrast and which is composed of simple geometrical shapes, such as rectangles, squares, circles, or the like. The display device receives a representation of the displayed graphical content from the communications device via the wireless-communications interface. In other words, the communications device reports to the display device what graphical content is currently displayed, such that the display device may use the displayed graphical content as calibration object. As an alternative, the display device may transmit an instruction to the communications device via the wireless-communications interface, to display the graphical content on the display of the communications device. The instruction to display the graphical content on the display of the communications device may be a request, such as a message or signal, in response to which the communications device displays the graphical content. Optionally, the display device may transmit a representation of the graphical content to the communications device via the wireless-communications interface.

According to an embodiment of the invention, the display device is further operative to receive an instruction to display a virtual object, the instruction comprising a corresponding real-world position of the virtual object when being displayed to the user. This is the position in the real-world scene where the virtual object appears to be placed. The instruction may, e.g., be received from an AR application which is executed by the display device, or which utilizes the display device for displaying virtual objects to the user. The display device is further operative to calculate a display position of the virtual object by applying the calibration transformation to the received real-world position, and display the virtual object at the calculated display position on the display.

Even though advantages of the invention have in some cases been described with reference to embodiments of the first aspect of the invention, corresponding reasoning applies to embodiments of other aspects of the invention.

Further objectives of, features of, and advantages with, the invention will become apparent when studying the following detailed disclosure, the drawings, and the appended claims. Those skilled in the art realize that different features of the invention can be combined to create embodiments other than those described in the following.

The above, as well as additional objects, features, and advantages, of the invention, will be better understood through the following illustrative and non-limiting detailed description of embodiments of the invention, with reference to the appended drawings, in which:.

The invention will now be described more fully herein after with reference to the accompanying drawings, in which certain embodiments of the invention are shown. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

In the following, embodiments of the Head-Mounted Display Device (HMD) are described with reference to <FIG>, which shows an HMD <NUM> configured to be worn by a user. HMD <NUM> may, e.g., be attached to a head <NUM> of the user using straps, or the like. HMD <NUM> comprises an at least partially see-through display <NUM> through which the user can view a real-world scene which is in front of the user, using eyes <NUM>. Display <NUM> may be used for displaying virtual objects, such as text, images, videos, or other types of graphical content, to the user, such that the displayed virtual objects are overlaid onto the real-world scene. HMD <NUM> further comprises a front-facing camera <NUM> which is operative to capture a first image of the real-world scene, and processing means <NUM> which is operative to cause HMD <NUM> to perform in accordance with embodiments of the invention set forth herein.

More specifically, HMD <NUM> is operative to select a calibration object from one or more real-world objects, or parts thereof, <NUM>-<NUM> which are visible in the first image captured by front-facing camera <NUM>. The first image represents a view of the real-world scene which is within a field-of-view <NUM> of front-facing camera <NUM>. In the present context, a real-world object may be any physical object, such as a piece of furniture, a household appliance, a building, a door, a window, a vehicle, a street sign, a tablet, a smartphone, a laptop, a checkerboard, and so forth. In <FIG>, a few real-world objects are exemplified as a tablet <NUM>, a box <NUM>, and a car <NUM>. Embodiments of the invention may select an entire real-world object as calibration object such as, e.g., box <NUM>, or a part of a real-world object, such as, e.g., a display <NUM> of tablet <NUM>, or graphical content <NUM> displayed thereon. In <FIG>, graphical content <NUM> is exemplified as a checkerboard, which is a commonly used calibration pattern in the field, owing to its high contrast and composition with simple geometrical shapes, which facilitates image processing and object recognition.

HMD <NUM> is further operative to derive a calibration transformation for calculating a display position based on a real-world position. The display position is calculated such that a virtual object which is displayed on display <NUM> at the display position, relative to a coordinate system of display <NUM>, is aligned with a corresponding real-world object located at the real-world position, relative to a coordinate system of front-facing camera <NUM>, as seen by the user when viewing the real-world scene with his/her eyes <NUM> through display <NUM>.

In AR relying on see-through displays, such as HMD <NUM>, an important task is to calibrate the HMD in order to establish the relative orientation and position, i.e., the pose, of the different coordinate systems. For the purpose of elucidating the invention, several coordinate systems may be associated with HMD <NUM>. A first coordinate system may be associated with display <NUM>, a second coordinate system may be associated with front-facing camera <NUM>, a third coordinate system may be associated with an optional eye-facing camera <NUM> (described further below), and a fourth coordinate system may be associated with eyes <NUM> of the user. Only if properly calibrated, a virtual object can be displayed on display <NUM> such that it appears to be placed at a desired position in the real-world scene, enabling an immersive user experience. The issue at hand is outlined, and a manual calibration procedure proposed, in <CIT>. An overview of different calibration procedures can be found in "<NPL>).

Overlaying a virtual object onto the real-world scene is illustrated in <FIG>, which shows the view of the real-world scene as seen by the user, with eyes <NUM>, through display <NUM>. In <FIG>, real-world objects, or parts thereof, <NUM>-<NUM> are seen through display <NUM>, whereas virtual object <NUM>, here illustrated as a checkerboard which is identical to checkerboard <NUM> displayed as graphical content on display <NUM> of tablet <NUM>, is overlaid onto the real-world scene by displaying it on display <NUM>. Under the assumption that checkerboard <NUM> is to be displayed at a display position such that it appears to be placed at the real-world position of checkerboard <NUM> which is displayed on display <NUM> of tablet <NUM>, HMD <NUM> needs to correct the display position so as to displace checkboard <NUM> such that it is aligned with, i.e., overlaps, checkboard <NUM> displayed on tablet <NUM>, as seen by eyes <NUM>. The required displacement of checkerboard <NUM>, which in <FIG> is illustrated by arrow <NUM>, is the objective of a calibration procedure.

HMD <NUM> may optionally comprise an eye-facing camera <NUM> which is operative to capture a second image of a reflection of the real-world scene by a cornea <NUM> of the user. Corneal imaging is a technique which utilizes a camera for imaging a person's cornea, in particular that of the user of the HMD, for gathering information about what is in front of the person and also, owing to the spherical nature of the human eyeball, for gathering information about objects in a field-of-view which potentially is wider than the viewing field-of-view. Such objects may potentially be outside the camera's field-of-view and even be located behind the camera. The technique is made possible due to the highly reflective nature of the human cornea, and also the availability of high-definition cameras in consumer devices such as HMDs.

If the calibration procedure relies on corneal imaging, HMD <NUM> is operative to select the calibration object from one or more real-world objects, or parts thereof, <NUM>-<NUM> which are visible in both the first image and the second image, and to derive the calibration transformation using the first image and the second image. Further optionally, HMD <NUM> may be operative to select the calibration object from one or more real-world objects, or parts thereof, <NUM>-<NUM> which are visible in a region of the first image which corresponds to field-of-view <NUM> of eye-facing camera <NUM>. Since the first image (captured by front-facing camera <NUM> with field-of-view <NUM>) is typically superior to the second image (captured by eye-facing camera <NUM> with field-of-view <NUM>), owing to the reflection off the imperfect surface of cornea <NUM>, identifying real-world objects by image processing and object recognition is easier, more reliable, and less resource consuming, using the first image as compared to the second image. By limiting the region of the first image which needs to be processed for identifying real-world objects which are suitable as calibration objects, embodiments of the invention require less computing resources and accordingly less power.

The calibration procedure itself, i.e., establishing the relative pose of the different coordinate systems associated with an HMD, is outside the scope of this disclosure. It suffices to say that a real-world calibration object is used for deriving the calibration transformation, e.g., a matrix or any other form of mathematical representation which is suitable for describing a transformation of coordinates between two coordinate systems. It will be appreciated that, depending on the design of the HMD, one or more of the coordinate systems may have fixed poses relative to each other and/or HMD <NUM>. For instance, this is the case for HMDs in which display <NUM>, front-facing camera <NUM>, and optionally eye-facing camera <NUM>, are contained in a single unit, such that the different components cannot move relative to each other during normal use.

The calibration objects which are used in the art are typically dedicated calibration objects, such as a checkerboard or a calibration rig (see, e.g., "<NPL>). Rather than using a dedicated calibration object which the user of HMD <NUM> has to keep, and possibly carry with him/her, embodiments of the invention rely on utilizing a real-world object which is available in the vicinity of the user when HMD <NUM> requires calibration. Depending on the design of an HMD and its usage, calibration is typically required if the pose of eyes <NUM> has changed, i.e., the orientation and/or position of eyes <NUM> relative to HMD <NUM>. Moreover, calibration may be required if one or more of display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>, have been displaced relative to each other and/or relative to HMD <NUM>.

More specifically, HMD <NUM> may be operative to initiate, or trigger, a calibration procedure, i.e., to select the calibration object and to derive the calibration transformation, in response to any one of: receiving from the user an instruction to initiate a calibration procedure, powering up HMD <NUM>, detecting a misalignment of a displayed virtual object relative to the real-world scene, detecting that the user is different from a previous user of HMD <NUM>, detecting that HMD <NUM> has been displaced relative to at least one eye <NUM> of the user, and detecting that one or more of display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>, have been displaced relative to each other and/or relative to HMD <NUM>.

In practice, a misalignment of a displayed virtual object relative to the real-world scene may be detected by displaying a virtual object at a display position which corresponds to the real-world position of a specific real-world object. In particular, the displayed virtual object may have the same shape as the real-world object, i.e., it may be a virtual representation (or virtual copy) of the real-world object. If the displayed virtual object and the real-world object are misaligned, at least to a certain extent, the calibration procedure is triggered. The misalignment can either be detected by the user or through corneal imaging, by image processing the second image captured by eye-facing camera <NUM>, in which both the real-world object and the overlaid virtual object are visible.

A displacement of HMD <NUM> relative to at least one eye <NUM> of the user may, e.g., be detected using eye-facing camera <NUM>. This may be achieved by tracking the position(s) of the eye(s) <NUM> of the user over time. The calibration procedure is triggered if the position of the user's eye(s) <NUM> deviates from a historical average value by more than a threshold value. The threshold value may either be set by the user, by a manufacturer of HMD <NUM>, or by an AR application utilizing HMD <NUM> for displaying virtual objects to the user.

A displacement of one or more of display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>, relative to each other and/or relative to HMD <NUM> may be detected by utilizing motion sensors which are comprised in display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>.

In the following, different alternatives for selecting the calibration object among one or more real-world objects, or parts thereof, <NUM>-<NUM> are described.

For instance, HMD <NUM> may be operative to select the calibration object based on a visual appearance of the calibration object. Preferably, real-world objects which are clearly visible, have good lighting conditions and/or high contrast, and/or are composed of simple geometrical shapes, are selected. Such real-world objects are typically easy to detect by image processing and object recognition, e.g., using Scale-Invariant Feature Transform (SIFT) (see, e.g., <CIT>) or similar algorithms known in the art.

Alternatively, HMD <NUM> may be operative to select as the calibration object a previously selected calibration object among the one or more real-world objects, or parts thereof, <NUM>-<NUM>. For this purpose, HMD <NUM> may maintain a database, either in a memory comprised in HMD <NUM> (such as memory <NUM> show in <FIG>) or accessible by HMD <NUM> over a communications network (via wireless-communications interface <NUM>), e.g., a cloud-based database. In the database, information which may be used for identifying a calibration object among the one or more real-world objects may be stored, e.g., information pertaining to their visual appearance or visual features, pictures, or information pertaining to their shape, composition of geometrical shapes, and dimensions.

As a further alternative, HMD <NUM> may additionally comprise one or more motion sensors <NUM> operative to track a motion of HMD <NUM>. The one or more motion sensors <NUM> may, e.g., be based on accelerometers, gyroscopes, Global Positioning System (GPS) sensors, magnetometers, cameras, and so forth, as are known in the art and provided with regular smartphones. HMD <NUM> is operative to estimate a duration of time during which the one or more real-world objects remain visible in the first image, i.e., remain within field-of-view <NUM> of front-facing camera <NUM>, based on the tracked motion of HMD <NUM>, and select the calibration object based on the estimated duration of time during which the calibration object remains visible in the first image. This is the case if a calibration procedure is employed by HMD <NUM> which relies on tracking the selected calibration object with front-facing camera <NUM>. Preferably, a real-world object, or part thereof, <NUM>-<NUM> is selected as calibration object which is visible in the first image for a duration of time sufficiently long to perform the calibration. As an alternative, a real-world object, or part thereof, <NUM>-<NUM> which is moving slowly, or not moving at all, relative to field-of-view <NUM> may be selected as calibration object. For instance, if HMD <NUM> is more or less stationary, a stationary real-world object such as box <NUM> may be selected. If, on the other hand, HMD <NUM> is moving, e.g., because the user is turning his/her head <NUM>, a real-world object which is moving similarly may be selected, such as car <NUM>.

It will also be appreciated that, if the calibration object is selected among the one or more real-world objects, or parts thereof, <NUM>-<NUM> which are visible in a region of the first image which corresponds to field-of-view <NUM> of eye-facing camera <NUM>, the calibration object may be selected based on an estimated duration of time during which the calibration object remains visible in the second image, i.e., within field-of-view <NUM> of eye-facing camera <NUM>.

As yet a further alternative, HMD <NUM> may be operative to select the calibration object by identifying one or more real-world objects <NUM>-<NUM> by matching visual features of one or more real-world objects <NUM>-<NUM> against information pertaining to visual features of real-world objects which is stored in a database, e.g., using SIFT or similar algorithms, and selecting the calibration object based on information obtained from the database. The obtained information may indicate a respective suitability of the identified real-world objects, or parts thereof, <NUM>-<NUM> as calibration object. For instance, the information may relate to visual appearance, shape, composition of geometrical shapes, dimensions, or the like, which is utilized in deriving the calibration transformation. Preferably, the most suitable calibration object is selected.

As yet a further alternative, if the calibration transformation is derived using both the first image and the second image, by relying on corneal imaging, HMD <NUM> may be operative to select the calibration object based on a distortion of the calibration object in the second image as compared to the first image. A distortion of the calibration object in the second image may stem from the reflection off cornea <NUM>, which is spherical shape and may suffer from imperfections in the outer surface of cornea <NUM> and tears or dirt on the outer surface of cornea <NUM>. In addition, the optical element of HMD <NUM>, through which the user views the real-world scene, may contribute to the distortion of the second image. Preferably, the object with the least distortion is selected to facilitate deriving the calibration transformation.

Embodiments of the invention select as the calibration object a communications device among the one or more real-world objects, such as tablet <NUM> shown in <FIG> and <FIG>, a smartphone, a mobile phone, a computer display, a television, or the like. To this end, HMD <NUM> comprises a wireless-communications interface <NUM>, which may be based on any known wireless communications technology. For example, wireless-communications interface <NUM> may be based on a short-range radio technology like Wireless Local Arena Network (WLAN)/WiFi or Bluetooth, or a cellular radio technology like Global System for Mobile communications (GSM), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), or a <NUM> technology based on NR/NX. Communications, i.e., exchange of data, between HMD <NUM> and the communications device, comprising a wireless-communications interface which is compatible with wireless-communications interface <NUM>, may commence using any suitable protocol, e.g., the HyperText Transfer Protocol (HTTP), the Constrained Application Protocol (CoAP), or the like.

More specifically, HMD <NUM> is operative to select the calibration object by identifying, using wireless-communications interface <NUM>, a communications device comprising a display among the one or more real-world objects, such as tablet <NUM>, and selecting as the calibration object at least one of display <NUM> of tablet <NUM> and graphical content <NUM> displayed thereon. Identifying a wireless-communications devices among the one or more real-world objects may, e.g., be achieved by relying on a discovery procedure as is known from Bluetooth and other wireless-communications technologies. As part of the discovery procedure, or subsequent to the discovery procedure, HMD <NUM> may be operative to establish wireless communications with the identified communications device and/or to acquire information about a type of the communications device or its capabilities, in particular information pertaining to its display <NUM>. Utilizing a communications device with a display, such as tablet <NUM>, as calibration object is advantageous in that a display is typically easy to identify by image processing and object recognition, owing to its simple geometrical shape and high contrast, if sufficiently bright.

A further advantage of using a communications device comprising a display as calibration object is that information pertaining to graphical content which is displayed on the display of the communications device during the calibration procedure can be exchanged between HMD <NUM> and the communications device. The displayed graphical content may, e.g., be a dedicated calibration pattern which is characterized by high contrast and is composed of simple geometrical shapes (rectangles, squares, circles), such as checkerboard <NUM> shown in <FIG> and <FIG>.

More specifically, HMD <NUM> is operative to receive a representation of displayed graphical content <NUM> from tablet <NUM> via wireless-communications interface <NUM>. That is, tablet <NUM> reports what is currently displayed on display <NUM>. The displayed graphical content may be any graphical content which currently is displayed by one or more apps (applications) being executed by tablet <NUM>, but may advantageously be a dedicated calibration pattern, such as checkerboard <NUM>. Tablet <NUM> may, e.g., be operative to display a dedicated calibration pattern in response to establishing communications with HMD <NUM>, or in response to receiving an indication from HMD <NUM> that calibration is ongoing.

HMD <NUM> is operative to transmit an instruction to tablet <NUM> via wireless-communications interface <NUM>, to display graphical content <NUM> on display <NUM>. The instruction may, e.g., be a request, a message, or a signal, in response to which tablet <NUM> displays graphical content <NUM>. The displayed graphical content may be pre-defined, such as a dedicated calibration pattern. Optionally, HMD <NUM> may be operative to transmit a representation of graphical content <NUM> to tablet <NUM> via wireless-communications interface <NUM>. In this way, HMD <NUM> may control the details of the graphical content which is displayed by tablet <NUM>, and may advantageously transmit a dedicated calibration pattern to tablet <NUM>. The representation of the graphical content may, e.g., be in the form of a known computer-graphics format or image format.

HMD <NUM> may further be operative, in response to selecting the calibration object, to adapt one or more displayed virtual objects which overlap the selected calibration object, as seen by the user with eyes <NUM>. This is particularly advantageous if a calibration method based on corneal imaging is employed, e.g., as described in "<NPL>). Thereby, it is avoided that the calibration object is obscured by a displayed virtual object, which may inhibit the calibration procedure.

HMD <NUM> may further be operative, in response to selecting the calibration object, to notify the user that calibration is ongoing. Optionally, HMD <NUM> may be operative to notify the user that calibration is ongoing by displaying a marker <NUM> on display <NUM> to identify the selected calibration object, in this case box <NUM>, to the user. Thereby, the user may be requested to gaze in the direction of the selected calibration object, or to adapt his/her movement so as to facilitate the calibration procedure. The user may alternatively be requested to change his/her head pose so as to minimize a distortion of the selected calibration object in the second image, if a calibration procedure relying on corneal imaging is used, or to move closer to the selected calibration cerebration object.

Once the calibration transformation has been derived, a representation of the derived calibration transformation, such as a matrix or other suitable mathematical representation in electronic format, may be stored in a memory of HMD <NUM> (such as memory <NUM> shown in <FIG>) and subsequently used for displaying virtual objects in a manner which provides an immersive user experience. To this end, HMD <NUM> may be operative to receive an instruction to display a virtual object. The instruction may, e.g., be received from an AR application which is executed by HMD <NUM>, or from an AR application which utilizes HMD <NUM> for disapplying virtual objects to the user. For instance, HMD <NUM> may be connected, either wired or wirelessly using any suitable protocol, e.g., HTTP or CoAP, to a computing device such as a computer, a laptop, a smartphone, a tablet, or a gaming console, executing the AR application. The received instruction comprises a corresponding real-world position of the virtual object when being displayed to the user, i.e., a position in the real world at which the virtual object appears to be placed when being displayed to the user (seen with eyes <NUM>) on display <NUM>. HMD <NUM> is further operative to calculate a display position of the virtual object by applying the calibration transformation to the received real-world position, and to display the virtual object at the calculated display position on display <NUM>.

In the following, embodiments of processing means <NUM> comprised in HMD <NUM> are described with reference to <FIG>.

An embodiment <NUM> of processing means <NUM> is shown in <FIG>. Processing means <NUM> comprises a processing unit <NUM>, such as a general-purpose processor or processing circuitry, and a computer-readable storage medium <NUM>, such as a Random-Access Memory (RAM), a Flash memory, or the like. In addition, processing means <NUM> comprises one or more interfaces <NUM> ("I/O" in <FIG>) for controlling and/or receiving information from other components comprised in HMD <NUM>, such as display <NUM>, front-facing camera <NUM>, eye-facing camera <NUM>, wireless-communications interface <NUM>, and one or more motion sensors <NUM>, some of which may be optional. Memory <NUM> contains computer-executable instructions <NUM>, i.e., a computer program or software, to cause HMD <NUM> to become operative in accordance with embodiments of the invention as described herein, when computer-executable instructions <NUM> are executed on processing unit <NUM>.

An alternative embodiment <NUM> of processing means <NUM> is illustrated in <FIG>. Similar to processing means <NUM>, processing means <NUM> comprises one or more interfaces <NUM> ("I/O" in <FIG>) for controlling and/or receiving information from other components comprised in HMD <NUM>, such as display <NUM>, front-facing camera <NUM>, eye-facing camera <NUM>, wireless-communications interface <NUM>, and one or more motion sensors <NUM>, some of which may be optional. Processing means <NUM> further comprises a selection module <NUM>, a calibration module <NUM>, an optional trigger module <NUM>, and an optional display module <NUM>, which are configured to cause HMD <NUM> to become operative in accordance with embodiments of the invention as described herein.

In particular, selection module <NUM> is configured to select a calibration object from one or more real-world objects, or parts thereof, which are visible in a first image of a real-world scene, which first image captured by a front-facing camera comprised in the display device. Calibration module <NUM> is configured to derive a calibration transformation for calculating a display position based on a real-world position, such that a virtual object which is displayed at the display position on an at least partially see-through display which is comprised in the display device is aligned with a corresponding real-world object located at the real-world position, as seen by the user.

For instance, selection module <NUM> may be configured to select the calibration object from one or more real-world objects, or parts thereof, which are visible in both the first image and a second image of a reflection of the real-world scene by a cornea of the user, which second image is captured by an eye-facing camera comprised in the display device. Calibration module <NUM> may be configured to derive the calibration transformation using the first image and the second image.

Alternatively, selection module <NUM> may be configured to select the calibration object from one or more real-world objects, or parts thereof, which are visible in a region of the first image which corresponds to a field-of-view of the eye-facing camera.

As another alternative, selection module <NUM> may be configured to select the calibration object based on a distortion of the calibration object in the second image as compared to the first image.

As a further alternative, selection module <NUM> may be configured to select the calibration object based on a visual appearance of the calibration object.

As yet a further alternative, selection module <NUM> may be configured to select a previously selected calibration object among the one or more real-world objects as the calibration object.

As yet a further alternative, selection module <NUM> may further be configured to track a motion of the display device using one or more motion sensors comprised in the display device, estimate, based on the tracked motion of the display device, a duration of time during which the one or more real-world objects remain visible in the first image, and select the calibration object based on the estimated duration of time during which the calibration object remains visible in the first image.

As yet a further alternative, selection module <NUM> may be configured to select the calibration object by identifying one or more real-world objects by matching visual features of the one or more real-world objects against information pertaining to visual features of real-world objects which is stored in a database, and selecting the calibration object based on information obtained from the database, which information indicates a respective suitability of the identified real-world objects, or parts thereof, as calibration object.

Selection module <NUM> is configured to select the calibration object by identifying, using a wireless-communications interface comprised in the display device, a communications device comprising a display among the one or more real-world objects, and selecting as the calibration object at least one of the display of the communications device and graphical content displayed thereon. Selection module <NUM> is further configured to receive a representation of the displayed graphical content from the communications device via the wireless-communications interface. Alternatively, selection module <NUM> may further be configured to transmit an instruction to the communications device via the wireless-communications interface, to display the graphical content on the display of the communications device. Optionally, selection module <NUM> may further be configured to transmit a representation of the graphical content to the communications device via the wireless-communications interface.

Optional trigger module <NUM> may be configured to trigger selecting the calibration object by selection module <NUM> and deriving the calibration transformation by calibration module <NUM> in response to any one of: receiving from the user an instruction to initiate a calibration procedure, powering up the display device, detecting a misalignment of a displayed virtual object relative to the real-world scene, detecting that the user is different from a previous user of the display device, detecting that the display device has been displaced relative to at least one eye of the user, and detecting that any one of display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>, has been displaced relative to the display device.

Optional display module <NUM> may be configured, in response to selecting the calibration object by selection module <NUM>, to adapt one or more displayed virtual objects which overlap the selected calibration object as seen by the user.

Optionally, selection module <NUM> may further be configured, in response to selecting the calibration object, to notify the user that calibration is ongoing. Optionally, the user is notified that calibration is ongoing by displaying a marker on the display to identify the selected calibration object to the user.

Optional display module <NUM> mat further be configured to receive an instruction to display a virtual object, the instruction comprising a corresponding real-world position of the virtual object when being displayed to the user, calculate a display position of the virtual object by applying the calibration transformation derived by calibration module <NUM> to the received real-world position, and display the virtual object at the calculated display position on the display.

Modules <NUM>-<NUM> comprised in processing mean <NUM> may further be configured to perform additional or alternative operations in accordance with embodiments of the invention, as described herein.

Interfaces <NUM> and <NUM>, and modules <NUM>-<NUM>, as well as any additional modules comprised in processing means <NUM>, may be implemented by any kind of electronic circuitry, e.g., any one, or a combination of, analogue electronic circuitry, digital electronic circuitry, and processing means executing a suitable computer program, i.e., software.

In the following, embodiments <NUM> of the method performed by a head-mounted display device which is configured to be worn by a user are described with reference to <FIG>.

Method <NUM> comprises selecting <NUM> a calibration object from one or more real-world objects, or parts thereof, which are visible in a first image of a real-world scene, which first image captured by a front-facing camera comprised in the display device, and deriving <NUM> a calibration transformation for calculating a display position based on a real-world position, such that a virtual object which is displayed at the display position on an at least partially see-through display which is comprised in the display device is aligned with a corresponding real-world object located at the real-world position, as seen by the user.

For instance, the calibration object may be selected <NUM> from one or more real-world objects, or parts thereof, which are visible in both the first image and a second image of a reflection of the real-world scene by a cornea of the user, which second image is captured by an eye-facing camera comprised in the display device, and the calibration transformation may be derived <NUM> using the first image and the second image.

Alternatively, the calibration object may be selected <NUM> from one or more real-world objects, or parts thereof, which are visible in a region of the first image which corresponds to a field-of-view of the eye-facing camera.

As another alternative, the calibration object may be selected <NUM> based on a distortion of the calibration object in the second image as compared to the first image.

As a further alternative, the calibration object may be selected <NUM> based on a visual appearance of the calibration object.

As yet a further alternative, a previously selected calibration object among the one or more real-world objects may be selected <NUM> as the calibration object.

As yet a further alternative, method <NUM> may further comprise tracking <NUM> a motion of the display device using one or more motion sensors comprised in the display device, and estimating <NUM>, based on the tracked motion of the display device, a duration of time during which the one or more real-world objects remain visible in the first image. The calibration object is selected <NUM> based on the estimated duration of time during which the calibration object remains visible in the first image.

As yet a further alternative, selecting <NUM> the calibration object may comprise identifying one or more real-world objects by matching visual features of the one or more real-world objects against information pertaining to visual features of real-world objects which is stored in a database, and selecting the calibration object based on information obtained from the database, which information indicates a respective suitability of the identified real-world objects, or parts thereof, as calibration object.

Selecting <NUM> the calibration object comprises identifying, using a wireless-communications interface comprised in the display device, a communications device comprising a display among the one or more real-world objects, and selecting as the calibration object at least one of the display of the communications device and graphical content displayed thereon. Selecting <NUM> the calibration object further comprises receiving a representation of the displayed graphical content from the communications device via the wireless-communications interface. Alternatively, selecting <NUM> the calibration object may further comprise transmitting an instruction to the communications device via the wireless-communications interface, to display the graphical content on the display of the communications device. Optionally, selecting <NUM> the calibration object may further comprise transmitting a representation of the graphical content to the communications device via the wireless-communications interface.

Optionally, the calibration object is selected <NUM> and the calibration transformation is derived <NUM> in response to calibration being triggered <NUM> by any one of: receiving from the user an instruction to initiate a calibration procedure, powering up the display device, detecting a misalignment of a displayed virtual object relative to the real-world scene, detecting that the display device has been displaced relative to at least one eye of the user, and detecting that any one of display <NUM>, front-facing camera <NUM>, and eye-facing camera <NUM>, has been displaced relative to the display device.

Optionally, method <NUM> may further comprise, in response to selecting <NUM> the calibration object, adapting <NUM> one or more displayed virtual objects which overlap the selected calibration object as seen by the user.

Optionally, method <NUM> may further comprise, in response to selecting <NUM> the calibration object, notifying <NUM> the user that calibration is ongoing. Optionally, the user is notified <NUM> that calibration is ongoing by displaying a marker on the display to identify the selected calibration object to the user.

Optionally, method <NUM> may further comprise receiving <NUM> an instruction to display a virtual object, the instruction comprising a corresponding real-world position of the virtual object when being displayed to the user, calculating <NUM> a display position of the virtual object by applying the calibration transformation to the received real-world position, and displaying <NUM> the virtual object at the calculated display position on the display.

It will be appreciated that method <NUM> may comprise additional, or modified, steps in accordance with what is described throughout this disclosure. An embodiment of method <NUM> may be implemented as software, such as computer program <NUM>, to be executed by a processing unit comprised in a head-mounted display device, whereby the display device becomes operative in accordance with embodiments of the invention described herein. Computer program <NUM> may be stored on a computer-readable storage medium such as memory <NUM>, a Compact Disc (CD), a Digital Versatile Disc (DVD), a memory stick, or the like. Computer program <NUM> may also be carried by a data carrier signal. For instance, computer program <NUM> may be transferred to memory <NUM> over a communications network, such as the Internet, via wireless-communications interface <NUM>.

Claim 1:
A head-mounted display device (<NUM>) configured to be worn by a user (<NUM>), the display device comprising:
an at least partially see-through display (<NUM>),
a front-facing camera (<NUM>) operative to capture a first image of a real-world scene,
a wireless-communications interface (<NUM>), and
processing means (<NUM>) operative to:
select a calibration object from one or more real-world objects, or parts thereof, (<NUM>-<NUM>) which are visible in the first image, by identifying, using the wireless-communications interface, a communications device (<NUM>) comprising a display (<NUM>) among the one or more real-world objects, and selecting as the calibration object at least one of the display (<NUM>) of the communications device and graphical content (<NUM>) displayed thereon, and
derive a calibration transformation, using the selected calibration object, for calculating a display position based on a real-world position, such that a virtual object which is displayed on the display at the display position is aligned with a corresponding real-world object located at the real-world position, as seen by the user,
the display device (<NUM>) characterized in that the processing means is further operative to receive a representation of the displayed graphical content (<NUM>) from the communications device (<NUM>) via the wireless-communications interface.