Patent Description:
As used herein, VR and AR systems are described and referenced interchangeably. Unless stated otherwise, the descriptions herein apply equally to all types of MR systems, which (as detailed above) include AR systems, VR reality systems, and/or any other similar system capable of displaying virtual content.

A MR system may also employ different types of cameras in order to display content to users, such as in the form of a passthrough image. A passthrough image or view can aid users in avoiding disorientation and/or safety hazards when transitioning into and/or navigating within a MR environment. A MR system can present views captured by cameras in a variety of ways. The process of using images captured by world-facing cameras to provide views of a real-world environment creates many challenges, however.

Some of these challenges occur when attempting to align image content from multiple cameras. Often, this alignment process requires detailed timestamp information and pose information in order to perform the alignment processes. Sometimes, however, timestamp data or perhaps even pose data is not available because different cameras may be operating in different time domains such that they have a temporal offset. Furthermore, sometimes the timestamp data is simply not available because the cameras may be operating remotely from one another, and the timestamp data is not transmitted. Aligning image content provides substantial benefits, especially in terms of hologram placement and generation, so these problems present serious obstacles to the technical field. Accordingly, there is a substantial need in the field to improve how images are aligned with one another.

<CIT> describes a system for visual inspection including: a scanning system configured to capture images of an object and to compute a three-dimensional (<NUM>-D) model of the object based on the captured images; an inspection system configured to: compute a descriptor of the object based on the <NUM>-D model of the object; retrieve metadata corresponding to the object based on the descriptor; and compute a plurality of inspection results based on the retrieved metadata and the <NUM>-D model of the object; and a display device system including: a display; a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: generate overlay data from the inspection results; and show the overlay data on the display, the overlay data being aligned with a view of the object through the display. <CIT> describes methods for generating and displaying images associated with one or more virtual objects within an augmented reality environment at a frame rate that is greater than a rendering frame rate are described. The rendering frame rate may correspond with the minimum time to render images associated with a pose of a head-mounted display device (HMD). In some embodiments, the HMD may determine a predicted pose associated with a future position and orientation of the HMD, generate a pre-rendered image based on the predicted pose, determine an updated pose associated with the HMD subsequent to generating the pre-rendered image, generate an updated image based on the updated pose and the pre-rendered image, and display the updated image on the HMD. The updated image may be generated via a homographic transformation and/or a pixel offset adjustment of the pre-rendered image. <NPL> describes work that aims to build an open-source, low-latency hardware-accelerated headset for mixed (virtual or augmented) reality applications. In this project, the motion-to-photon latency is reduced by leveraging specialized hardware platforms as well as computer vision algorithms to build a headset from scratch. In this project, the headset is capable of running simple mixed reality demo applications (cubemap rendering for VR, 3D static object overlay for AR) with a motion-of-photon latency of <NUM>.

Embodiments disclosed herein relate to systems, devices (e.g., wearable devices, hardware storage devices, etc.), and methods for aligning and stabilizing images generated by an integrated camera that is physically mounted to a computer system (e.g., perhaps an HMD) with images generated by a detached camera that is physically unmounted from the computer system.

In some embodiments, a first image is generated using the integrated camera. This first image is used to determine a first pose of the computer system. Additionally, a first timestamp is determined for the first image. The embodiments also acquire a second image generated by the detached camera. The second image is aligned with the first image. An overlaid image is generated by overlaying the second image onto the first image based on the alignment process. A pose difference is identified between a current pose of the computer system at a current timestamp and the first pose that was determined using the first image at the first timestamp. Late stage reprojection (LSR) is applied to the overlaid image to transform pixels in the overlaid image to account for the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp. After applying the LSR to the overlaid image, the embodiments display the overlaid image.

Embodiments disclosed herein relate to systems, devices (e.g., wearable devices, hardware storage devices, etc.), and methods for aligning and stabilizing (e.g., perhaps via late stage reprojection (LSR)) images generated by an integrated camera that is physically mounted to a computer system (e.g., perhaps an HMD) with images generated by a detached camera that is physically unmounted from the computer system.

In some embodiments, a first image is generated using the integrated camera. This first image is used to determine a first pose of the computer system. Additionally, a first timestamp is determined for the first image. A second image, which was generated by the detached camera, is acquired. The second image is aligned with the first image so that an overlaid image can be generated. A pose difference is identified between a current pose of the computer system at a current timestamp and the first pose. Late stage reprojection (LSR) is applied to the overlaid image to account for the pose difference. After applying the LSR, the embodiments display the overlaid image.

The following section outlines some example improvements and practical applications provided by the disclosed embodiments. It will be appreciated, however, that these are just examples only and that the embodiments are not limited to only these improvements.

The disclosed embodiments provide substantial improvements, benefits, and practical applications to the technical field. By way of example, the disclosed embodiments improve how images are generated and displayed and improve how image content is aligned, even without the use of timestamp data.

That is, the embodiments solve the problem of not having the exact timestamp of a remote or detached camera image when attempting to align that image's content with another image to create a single composite or overlaid image. There may be a variety of reasons as to why the information regarding the timestamp might not be known. For instance, asynchronous wireless communications might be occurring between multiple devices operating over different time domains, resulting in the occurrence of not knowing the timestamp. Notwithstanding this potential lack of information, the embodiments are still able to perform image alignment because the embodiments do not necessarily perform image matching based on timestamp data. Consequently, the embodiments provide improvements to the technical field by enabling the ability to perform image matching without requiring temporal data.

Attention will now be directed to <FIG>, which illustrates an example of a head-mounted device (HMD) <NUM>. HMD <NUM> can be any type of MR system 100A, including a VR system 100B or an AR system 100C. It should be noted that while a substantial portion of this disclosure is focused on the use of an HMD, the embodiments are not limited to being practiced using only an HMD. That is, any type of scanning or camera system can be used, even systems entirely removed or separate from an HMD. As such, the disclosed principles should be interpreted broadly to encompass any type of scanning scenario or device. Some embodiments may even refrain from actively using a scanning device themselves and may simply use the data generated by the scanning device. For instance, some embodiments may at least be partially practiced in a cloud computing environment.

HMD <NUM> is shown as including scanning sensor(s) <NUM> (i.e. a type of scanning or camera system), and HMD <NUM> can use the scanning sensor(s) <NUM> to scan environments, map environments, capture environmental data, and/or generate any kind of images of the environment (e.g., by generating a 3D representation of the environment or by generating a "passthrough" visualization). Scanning sensor(s) <NUM> may comprise any number or any type of scanning devices, without limit.

In accordance with the disclosed embodiments, the HMD <NUM> may be used to generate a passthrough visualizations of the user's environment. As used herein, a "passthrough" visualization refers to a visualization that reflects the perspective of the environment from the HMD's point of view, regardless of whether the HMD <NUM> is included as a part of an AR system or a VR system. To generate this passthrough visualization, the HMD <NUM> may use its scanning sensor(s) <NUM> to scan, map, or otherwise record its surrounding environment, including any objects in the environment, and to pass that data on to the user to view.

To generate a passthrough image, the scanning sensor(s) <NUM> typically rely on its cameras (e.g., head tracking cameras, hand tracking cameras, depth cameras, or any other type of camera) to obtain one or more raw images (aka texture images) of the environment. In addition to generating passthrough images, these raw images may also be used to determine depth data detailing the distance from the sensor to any objects captured by the raw images (e.g., a z-axis range or measurement). Once these raw images are obtained, then a depth map can be computed from the depth data embedded or included within the raw images (e.g., based on pixel disparities), and passthrough images can be generated (e.g., one for each pupil) using the depth map for any reprojections, if needed.

From the passthrough visualizations, a user will be able to perceive what is currently in his/her environment without having to remove or reposition the HMD <NUM>. Furthermore, as will be described in more detail later, the disclosed passthrough visualizations will also enhance the user's ability to view objects within his/her environment (e.g., by displaying additional environmental conditions that may not have been detectable by a human eye). As used herein, a so-called "overlaid image" can be a type of passthrough image.

It should be noted that while the majority of this disclosure focuses on generating "a" passthrough image, the embodiments actually generate a separate passthrough image for each one of the user's eyes. That is, two passthrough images are typically generated concurrently with one another. Therefore, while frequent reference is made to generating what seems to be a single passthrough image, the embodiments are actually able to simultaneously generate multiple passthrough images.

In some embodiments, scanning sensor(s) <NUM> include visible light camera(s) <NUM>, low light camera(s) <NUM>, thermal imaging camera(s) <NUM>, potentially (though not necessarily, as represented by the dotted box in <FIG>) ultraviolet (UV) camera(s) <NUM>, and potentially (though not necessarily, as represented by the dotted box) a dot illuminator <NUM>. The ellipsis <NUM> demonstrates how any other type of camera or camera system (e.g., depth cameras, time of flight cameras, virtual cameras, depth lasers, etc.) may be included among the scanning sensor(s) <NUM>.

As an example, a camera structured to detect mid-infrared wavelengths may be included within the scanning sensor(s) <NUM>. As another example, any number of virtual cameras that are reprojected from an actual camera may be included among the scanning sensor(s) <NUM> and may be used to generate a stereo pair of images. In this manner, the scanning sensor(s) <NUM> may be used to generate the stereo pair of images. In some cases, the stereo pair of images may be obtained or generated as a result of performing any one or more of the following operations: active stereo image generation via use of two cameras and one dot illuminator (e.g., dot illuminator <NUM>); passive stereo image generation via use of two cameras; image generation using structured light via use of one actual camera, one virtual camera, and one dot illuminator (e.g., dot illuminator <NUM>); or image generation using a time of flight (TOF) sensor in which a baseline is present between a depth laser and a corresponding camera and in which a field of view (FOV) of the corresponding camera is offset relative to a field of illumination of the depth laser.

Generally, a human eye is able to perceive light within the so-called "visible spectrum," which includes light (or rather, electromagnetic radiation) having wavelengths ranging from about <NUM> nanometers (nm) up to about <NUM>. As used herein, the visible light camera(s) <NUM> include two or more red, green, blue (RGB) cameras structured to capture light photons within the visible spectrum. Often, these RGB cameras are complementary metal-oxide-semiconductor (CMOS) type cameras, though other camera types may be used as well (e.g., charge coupled devices, CCD).

The RGB cameras are typically stereoscopic cameras, meaning that the fields of view of the two or more RGB cameras at least partially overlap with one another. With this overlapping region, images generated by the visible light camera(s) <NUM> can be used to identify disparities between certain pixels that commonly represent an object captured by both images. Based on these pixel disparities, the embodiments are able to determine depths for objects located within the overlapping region (i.e. "stereoscopic depth matching" or "stereo depth matching"). As such, the visible light camera(s) <NUM> can be used to not only generate passthrough visualizations, but they can also be used to determine object depth. In some embodiments, the visible light camera(s) <NUM> can capture both visible light and IR light.

The low light camera(s) <NUM> are structured to capture visible light and IR light. IR light is often segmented into three different classifications, including near-IR, mid-IR, and far-IR (e.g., thermal-IR). The classifications are determined based on the energy of the IR light. By way of example, near-IR has relatively higher energy as a result of having relatively shorter wavelengths (e.g., between about <NUM> and about <NUM>,<NUM>). In contrast, far-IR has relatively less energy as a result of having relatively longer wavelengths (e.g., up to about <NUM>,<NUM>). Mid-IR has energy values in between or in the middle of the near-IR and far-IR ranges. The low light camera(s) <NUM> are structured to detect or be sensitive to IR light in at least the near-IR range.

In some embodiments, the visible light camera(s) <NUM> and the low light camera(s) <NUM> (aka low light night vision cameras) operate in approximately the same overlapping wavelength range. In some cases, this overlapping wavelength range is between about <NUM> nanometers and about <NUM>,<NUM> nanometers. Additionally, in some embodiments these two types of cameras are both silicon detectors.

One distinguishing feature between these two types of cameras is related to the illuminance conditions or illuminance range(s) in which they actively operate. In some cases, the visible light camera(s) <NUM> are low power cameras and operate in environments where the illuminance is between about <NUM> lux and about <NUM>,<NUM> lux, or rather, the illuminance range begins at about <NUM> lux and increases beyond <NUM> lux. In contrast, the low light camera(s) <NUM> consume more power and operate in environments where the illuminance range is between about <NUM> milli-lux and about <NUM> lux.

The thermal imaging camera(s) <NUM>, on the other hand, are structured to detect electromagnetic radiation or IR light in the far-IR (i.e. thermal-IR) range, though some embodiments also enable the thermal imaging camera(s) <NUM> to detect radiation in the mid-IR range. To clarify, the thermal imaging camera(s) <NUM> may be a long wave infrared imaging camera structured to detect electromagnetic radiation by measuring long wave infrared wavelengths. Often, the thermal imaging camera(s) <NUM> detect IR radiation having wavelengths between about <NUM> microns and <NUM> microns. Because the thermal imaging camera(s) <NUM> detect far-IR radiation, the thermal imaging camera(s) <NUM> can operate in any illuminance condition, without restriction.

In some cases (though not all), the thermal imaging camera(s) <NUM> include an uncooled thermal imaging sensor. An uncooled thermal imaging sensor uses a specific type of detector design that is based on a bolometer, which is a device that measures the magnitude or power of an incident electromagnetic wave / radiation. To measure the radiation, the bolometer uses a thin layer of absorptive material (e.g., metal) connected to a thermal reservoir through a thermal link. The incident wave strikes and heats the material. In response to the material being heated, the bolometer detects a temperature-dependent electrical resistance. Changes to environmental temperature cause changes to the bolometer's temperature, and these changes can be converted into an electrical signal to thereby produce a thermal image of the environment. In accordance with at least some of the disclosed embodiments, the uncooled thermal imaging sensor is used to generate any number of thermal images. The bolometer of the uncooled thermal imaging sensor can detect electromagnetic radiation across a wide spectrum, spanning the mid-IR spectrum, the far-IR spectrum, and even up to millimeter-sized waves.

The UV camera(s) <NUM> are structured to capture light in the UV range. The UV range includes electromagnetic radiation having wavelengths between about <NUM> and about <NUM>. The disclosed UV camera(s) <NUM> should be interpreted broadly and may be operated in a manner that includes both reflected UV photography and UV induced fluorescence photography.

Accordingly, as used herein, reference to "visible light cameras" (including "head tracking cameras"), are cameras that are primarily used for computer vision to perform head tracking. These cameras can detect visible light, or even a combination of visible and IR light (e.g., a range of IR light, including IR light having a wavelength of about <NUM>). In some cases, these cameras are global shutter devices with pixels being about <NUM> in size. Low light cameras, on the other hand, are cameras that are sensitive to visible light and near-IR. These cameras are larger and may have pixels that are about <NUM> in size or larger. These cameras are also sensitive to wavelengths that silicon sensors are sensitive to, which wavelengths are between about <NUM> to <NUM>. Thermal / long wavelength IR devices (i.e. thermal imaging cameras) have pixel sizes that are about <NUM> or larger and detect heat radiated from the environment. These cameras are sensitive to wavelengths in the <NUM> to <NUM> range. Some embodiments also include mid-IR cameras configured to detect at least mid-IR light. These cameras often comprise non-silicon materials (e.g., InP or InGaAs) that detect light in the <NUM> to <NUM> wavelength range.

Accordingly, the disclosed embodiments may be structured to utilize numerous different camera types. The different camera types include, but are not limited to, visible light cameras, low light cameras, thermal imaging cameras, and UV cameras. Passthrough image generation and even stereo depth matching may be performed using images generated from any one type or combination of types of the above listed camera types.

Generally, the low light camera(s) <NUM>, the thermal imaging camera(s) <NUM>, and the UV camera(s) <NUM> (if present) consume relatively more power than the visible light camera(s) <NUM>. Therefore, when not in use, the low light camera(s) <NUM>, the thermal imaging camera(s) <NUM>, and the UV camera(s) <NUM> are typically in the powered-down state in which those cameras are either turned off (and thus consuming no power) or in a reduced operability mode (and thus consuming substantially less power than if those cameras were fully operational). In contrast, the visible light camera(s) <NUM> are typically in the powered-up state in which those cameras are by default fully operational.

It should be noted that any number of cameras may be provided on the HMD <NUM> for each of the different camera types. That is, the visible light camera(s) <NUM> may include <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more than <NUM> cameras. Often, however, the number of cameras is at least <NUM> so the HMD <NUM> can perform passthrough image generation and/or stereoscopic depth matching, as described earlier. Similarly, the low light camera(s) <NUM>, the thermal imaging camera(s) <NUM>, and the UV camera(s) <NUM> may each respectively include <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or more than <NUM> corresponding cameras.

<FIG> illustrates an example HMD <NUM>, which is representative of the HMD <NUM> from <FIG>. HMD <NUM> is shown as including multiple different cameras, including cameras <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. Cameras <NUM>-<NUM> are representative of any number or combination of the visible light camera(s) <NUM>, the low light camera(s) <NUM>, the thermal imaging camera(s) <NUM>, and the UV camera(s) <NUM> from <FIG>. While only <NUM> cameras are illustrated in <FIG>, HMD <NUM> may include more or less than <NUM> cameras.

In some cases, the cameras can be located at specific positions on the HMD <NUM>. For instance, in some cases a first camera (e.g., perhaps camera <NUM>) is disposed on the HMD <NUM> at a position above a designated left eye position of any users who wear the HMD <NUM> relative to a height direction of the HMD. For instance, the camera <NUM> is positioned above the pupil <NUM>. As another example, the first camera (e.g., camera <NUM>) is additionally positioned above the designated left eye position relative to a width direction of the HMD. That is, the camera <NUM> is positioned not only above the pupil <NUM> but also in-line relative to the pupil <NUM>. When a VR system is used, a camera may be placed directly in front of the designated left eye position. For example, with reference to <FIG>, a camera may be physically disposed on the HMD <NUM> at a position in front of the pupil <NUM> in the z-axis direction.

When a second camera is provided (e.g., perhaps camera <NUM>), the second camera may be disposed on the HMD at a position above a designated right eye position of any users who wear the HMD relative to the height direction of the HMD. For instance, the camera <NUM> is above the pupil <NUM>. In some cases, the second camera is additionally positioned above the designated right eye position relative to the width direction of the HMD. When a VR system is used, a camera may be placed directly in front of the designated right eye position. For example, with reference to <FIG>, a camera may be physically disposed on the HMD <NUM> at a position in front of the pupil <NUM> in the z-axis direction.

When a user wears HMD <NUM>, HMD <NUM> fits over the user's head and the HMD <NUM>'s display is positioned in front of the user's pupils, such as pupil <NUM> and pupil <NUM>. Often, the cameras <NUM>-<NUM> will be physically offset some distance from the user's pupils <NUM> and <NUM>. For instance, there may be a vertical offset in the HMD height direction (i.e. the "Y" axis), as shown by offset <NUM>. Similarly, there may be a horizontal offset in the HMD width direction (i.e. the "X" axis), as shown by offset <NUM>.

HMD <NUM> is configured to provide passthrough image(s) <NUM> for the user of HMD <NUM> to view. In doing so, HMD <NUM> is able to provide a visualization of the real world without requiring the user to remove or reposition HMD <NUM>. These passthrough image(s) <NUM> effectively represent the view of the environment from the HMD's perspective. Cameras <NUM>-<NUM> are used to provide these passthrough image(s) <NUM>. In some implementations, the embodiments utilize a planar reprojection process when generating the passthrough images. Using this planar reprojection process is acceptable when objects in the environment are sufficiently far away from the HMD. Thus, in some cases, the embodiments are able to refrain from performing parallax correction because the objects in the environment are sufficiently far away and because that distance results in a negligible error with regard to depth visualizations or parallax issues.

Attention will now be directed to <FIG>, which illustrates an environment <NUM> in which an HMD <NUM> is operating in. HMD <NUM> is representative of the HMD <NUM> from <FIG>.

In this scenario, HMD <NUM> includes an integrated camera <NUM> that is physically mounted to the HMD <NUM>. For instance, integrated camera <NUM> may be any of the cameras <NUM>-<NUM> mentioned in <FIG>. Similarly, integrated camera <NUM> may be any of the cameras mentioned in <FIG>, such as the visible light camera(s) <NUM>, the low light camera(s) <NUM>, the thermal imaging camera(s) <NUM>, or even the UV camera(s) <NUM>. Integrated camera <NUM> is shown scanning the environment <NUM> via the field of view (FOV) <NUM>. That is, the objects included in the FOV <NUM> will be represented in an image generated by the integrated camera <NUM>.

<FIG> also shows the presence or use of a detached camera <NUM>. Here, the detached camera <NUM> is physically unmounted from the HMD <NUM>. For instance, in this particular scenario, the detached camera <NUM> is strapped or otherwise placed on the user's chest. In some scenarios, the detached camera <NUM> may not be placed on the user's body but may instead be placed on an object held by the user. As one example, suppose the detached camera <NUM> is mounted on a selfie stick or another type of extended rod. In some cases, the detached camera <NUM> may be attached to some other piece of equipment being used by the user. In some cases, the detached camera <NUM> may be entirely removed from control of the user, such as when the detached camera <NUM> is placed on the ground or perhaps on another user.

<FIG> shows how the detached camera <NUM> is associated with its own corresponding FOV <NUM>. That is, objects included within the FOV <NUM> will be captured or included in an image generated by the detached camera <NUM>. One will appreciate how both the integrated camera <NUM> and the detached camera <NUM> are able to generate still images as well as videos, without limit.

In accordance with the disclosed principles, at least a portion of the FOV <NUM> overlaps with the FOV <NUM>, as shown by the overlap <NUM> condition. This overlap <NUM> enables the embodiments to generate multiple images and then overlay image content from one image onto another image in order to generate a composite image or an overlaid image having enhanced features that would not be present if only a single image were used.

It should be noted that while this disclosure primarily focuses on the use of two images, the embodiments are able to align content from more than two images having overlapping regions. For instance, suppose <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, or even <NUM> images have overlapping content. The embodiments are able to examine each image and then align specific portions with one another. The resulting overlaid image may then be a composite image formed from any combination or alignment of the available images (e.g., even <NUM> or more images, if available). Accordingly, the embodiments are able to utilize any number of images when performing the disclosed operations and are not limited to only two images.

Suppose the integrated camera <NUM> is a low light camera and further suppose the detached camera <NUM> is a thermal imaging camera. As will be discussed in more detail later, the embodiments are able to selectively extract image content from the thermal imaging camera image and overlay that image content onto the image generated by the low light camera. In this regard, the thermal imaging content can be used to augment or supplement the low light image content, thereby providing enhanced image content to the user. Further details on these features will be provided later.

<FIG> shows a resulting image that is generated by the integrated camera <NUM> of <FIG> in the form of integrated camera image <NUM>. The shading shown in <FIG> for the integrated camera image <NUM> is provided in order to distinguish that image from any other images. The shading should not be interpreted as meaning that the integrated camera image <NUM> is any particular type of image.

By analyzing the content included in the integrated camera image <NUM>, the embodiments are able to determine a pose <NUM> of the HMD (e.g., HMD <NUM> from <FIG>). For instance, by detecting anchor points (e.g., points identified as being relatively static or non-moving), the embodiments are able to determine the orientation or pose <NUM> of the HMD relative to the surrounding environment.

Additionally, a timestamp <NUM> may be determined for the integrated camera image <NUM>. Timestamp <NUM> identifies the time at which the integrated camera image <NUM> was generated. Of course, the timestamp <NUM> may be based on any timing calculation, including an absolute time such as determined by an atomic clock or, alternatively, including any type of relative time, such as processor clock cycles and so forth.

The integrated camera <NUM> from <FIG> generated the integrated camera image <NUM>, and the integrated camera <NUM> operates at a particular refresh rate <NUM> for generating new images. This refresh rate <NUM> may be set to any value. Often, however, the refresh rate <NUM> is at least between <NUM> and <NUM>. In some cases, the refresh rate <NUM> is higher than <NUM>, such as perhaps <NUM> or higher. Often, the refresh rate <NUM> is around <NUM>.

<FIG> also shows a detached camera image <NUM>, which was generated by the detached camera <NUM> from <FIG>. In <FIG>, the detached camera image <NUM> is shown as being smaller in size than the size of the integrated camera image <NUM>, but that size discrepancy is simply for illustrative purposes only. In some cases, the detached camera image <NUM> may have a higher resolution than the resolution of the integrated camera image <NUM> while in other cases the detached camera image <NUM> may have a lower resolution than the resolution of the integrated camera image <NUM>. In some cases, the resolutions of the two images may be the same.

The integrated camera image <NUM> (e.g., a "first" image) may be one of a visible light image, a low light image, or a thermal image. The detached camera image <NUM> (e.g., a "second" image) may be a different one of the visible light image, the low light image, or the thermal image, or perhaps even the same type of image as the first image.

Similar to the discussion regarding the integrated camera image <NUM>, the embodiments are also able to use the detached camera image <NUM> to determine some additional information. Notably, these operations are not strictly necessary, and in some cases can be skipped or refrained from being performed. As such, the following discussion refers to some operations that may or may not be performed.

Specifically, the embodiments are able to analyze the content in the detached camera image <NUM> to determine a pose <NUM> of the detached camera <NUM> from <FIG>. Similarly, a timestamp <NUM> can be determined for the detached camera image <NUM>. In some cases, the timestamp <NUM> is different, or reflects a different time, relative to the timestamp <NUM> such that the two images may have a temporal offset.

The detached camera <NUM> may also have its own refresh rate <NUM>. This refresh rate <NUM> may be set to any value. Often, however, the refresh rate <NUM> is at least between <NUM> and <NUM>. In some cases, the refresh rate <NUM> is higher than <NUM>, such as perhaps <NUM> or <NUM> or even higher. Typically, the refresh rate <NUM> is around <NUM>. In some cases, the refresh rate <NUM> is the same as the refresh rate <NUM> while in other cases the refresh rate <NUM> is different from the refresh rate <NUM>. When the two refresh rates are different, then the two cameras (e.g., the integrated camera <NUM> and the detached camera <NUM> from <FIG>) operate in different time domains.

As briefly introduced earlier, in some instances the embodiments can refrain from determining the pose <NUM> and the timestamp <NUM>. The darkened arrow labeled as non-dependent <NUM> represents how the embodiments can refrain from determining the pose <NUM> and the darkened arrow labeled non-dependent <NUM> represents how the embodiments can refrain from determining the timestamp <NUM>. In some cases this non-dependency is based on a lack of information (e.g., such as in a case where the data is not transmitted) or is based on the embodiments refraining from computing the information. Further details on these aspects will be provided later.

Regarding pose determinations, <FIG> provides some additional information. <FIG> shows an integrated camera <NUM>, which is representative of the integrated cameras discussed thus far. <FIG> also shows a pose <NUM>, which is representative of the pose <NUM> from <FIG>. In accordance with the disclosed principles, the pose <NUM> refers to at least the x-y-z location of the integrated camera <NUM> relative to its environment.

In some cases, the pose <NUM> may include information detailing the <NUM> degrees of freedom, or <NUM> DOF <NUM>, information. Generally, the <NUM> DOF <NUM> refers to the movement or position of an object in three-dimensional space. The <NUM> DOF <NUM> includes surge (i.e. forward and backward in the x-axis direction), heave (i.e. up and down in the z-axis direction), and sway (i.e. left and right in the y-axis direction). In this regard, <NUM> DOF <NUM> refers to the combination of <NUM> translations and <NUM> rotations. Any possible movement of a body can be expressed using the <NUM> DOF <NUM>.

In some cases, the pose <NUM> may include information detailing the <NUM> DOF <NUM>. Generally, the <NUM> DOF <NUM> refers to tracking rotational motion only, such as pitch (i.e. the transverse axis), yaw (i.e. the normal axis), and roll (i.e. the longitudinal axis). In this regard, <NUM> DOF <NUM> allows the HMD to track rotational motion but not translational movement. As a further explanation, the <NUM> DOF <NUM> allows the HMD to determine whether a user (who is wearing the HMD) is looking left or right, whether the user is rotating his/her head up or down, or whether the user is pivoting left or right. In contrast to the <NUM> DOF <NUM>, when <NUM> DOF <NUM> is used, the HMD is not able to determine whether the user has moved in a translational manner, such as by moving to a new location in the environment.

Determining the <NUM> DOF <NUM> and the <NUM> DOF <NUM> can be performed using inbuilt sensors, such as accelerometers, gyroscopes, and magnetometers. Determining the <NUM> DOF <NUM> can also be performed using positional tracking sensors, such as head tracking sensors.

In accordance with the disclosed principles, the embodiments are able to align the integrated camera image <NUM> shown in <FIG> with the detached camera image <NUM>. That is, because at least a portion of the two camera's FOVs overlap with one another, as was described in <FIG>, at least a portion of the resulting images include corresponding content. Consequently, that corresponding content can be identified and then a merged, fused, or overlaid image can be generated based on the similar corresponding content. By generating this overlaid image, the embodiments are able to provide enhanced image content to the user, which enhanced image content would not be available if only a single image type were provided to a user. <FIG> illustrates a first type of alignment <NUM> that may be used to align image content from two (or potentially more than two) different images.

<FIG> shows an integrated camera image <NUM>, which is representative of the integrated camera image <NUM> from <FIG>, and a detached camera image <NUM>, which is representative of the detached camera image <NUM> from <FIG>. These two images are also often referred to as "texture" images.

The embodiments are able to analyze the texture images (i.e. perform computer vision feature detection) in an attempt to find any number of feature points. As used herein, the phrase "feature detection" generally refers to the process of computing image abstractions and then determining whether an image feature (e.g., of a particular type) is present at any particular point or pixel in the image. Often, corners (e.g., the corners of a wall), distinguishable edges (e.g., the edge of a table), or ridges are used as feature points because of the inherent or sharp contrasting visualization of an edge or corner.

<FIG> shows a number of example feature points in the integrated camera image <NUM>, as shown by feature point 615A, feature point 620A, and feature point 625A. Other feature points are identified using the darkened circles but are not labeled. Notice, these feature points relate to corners, edges, or other ridges, such as the folds in the blanket and pillows as well as the corners of the picture and walls. Any type of feature detector may be programmed to identify feature points. In some cases, the feature detector may be a machine learning algorithm.

As used herein, reference to any type of machine learning may include any type of machine learning algorithm or device, convolutional neural network(s), multilayer neural network(s), recursive neural network(s), deep neural network(s), decision tree model(s) (e.g., decision trees, random forests, and gradient boosted trees) linear regression model(s), logistic regression model(s), support vector machine(s) ("SVM"), artificial intelligence device(s), or any other type of intelligent computing system. Any amount of training data may be used (and perhaps later refined) to train the machine learning algorithm to dynamically perform the disclosed operations.

<FIG> also shows how the embodiments are able to analyze, examine, or review the detached camera image <NUM> to identify feature points, as shown by the darkened circles. Examples include, but are not limited to, the feature point 615B, the feature point 620B, and the feature point 625B.

In accordance with the disclosed principles, the embodiments detect any number of feature points (e.g., <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>,<NUM>, <NUM>,<NUM>, or more than <NUM>,<NUM>) and then attempt to identify correlations or correspondences between the feature points detected in the integrated camera image <NUM> and the feature points identified in the detached camera image <NUM>. For instance, the correspondence 615C has been identified in which the feature point 615A is linked, or corresponds, with the feature point 615B. Similarly, the correspondence 620C has been identified in which the feature point 620A is determined to correspond to the feature point 620B. The correspondence 625C has been identified in which the feature point 625A is determined to align or correspond with the feature point 625B. While only three correspondences are visualized, one will appreciate how any number of correspondences may be identified.

The summation or compilation of the identified correspondences (e.g., correspondences 615C, 620C, and 625C) constitute image correspondence(s) <NUM>. Accordingly, in some embodiments, the alignment <NUM> process includes identifying any number of feature points and then identifying correlations or correspondences between the feature points in the two (or more) different images.

Note, in this implementation, the embodiments refrain from determining a pose or a timestamp of the detached camera image <NUM>. Instead, the embodiments rely on the feature matching in order to determine whether or not to overlay the image content from one image onto another image. By way of additional clarification, these embodiments are non-dependent <NUM> from <FIG> on the pose and are non-dependent <NUM> on the timestamp.

Some embodiments then fit the feature or image correspondence(s) <NUM> to a motion model <NUM> in order to overlay one image onto another image to form an enhanced overlaid image. The motion model <NUM> may be any type of motion model. Generally, a motion model is a type of transformation matrix that enables a model, a known scene, or an object to be projected onto a different model, scene, or object.

In some cases, the motion model <NUM> may simply be a rotational motion model. With a rotational model, the embodiments are able to shift one image by any number of pixels (e.g., perhaps <NUM> pixels to the left and <NUM> pixels up) in order to overlay one image onto another image. For instance, once the image correspondence(s) <NUM> are identified, the embodiments can identify the pixel coordinates of those feature points or correspondence(s). Once the coordinates are identified, then the embodiments can overlay the detached camera image <NUM> onto the integrated camera image <NUM> using the rotational motion model approach described above.

In some cases, the motion model <NUM> may be more complex, such as in the form of a similarity transform model. The similarity transform model may be configured to allow for (i) rotation of either one of the integrated camera image <NUM> (e.g., a "first" image) or the detached camera image <NUM> (e.g., a "second" image), (ii) scaling of the first image or the second image, or (iii) homographic transformations of the first image or the second image. In this regard, the similarity transform model approach may be used to overlay image content from one image onto another image. Further details regarding this overlaying process will be provided later. Accordingly, in some cases, the process of aligning the detached camera image (e.g., a "second" image) with the integrated camera image (e.g., a "first" image) is performed by (i) identifying image correspondences between the second image and the first image and then, (ii) based on the identified image correspondences, fitting the correspondences to a motion model such that the second image is projected onto the first image.

<FIG> illustrates another alignment <NUM> operation that may be performed in order to align content from the two images so that the content can be overlaid to form an overlaid image. Specifically, <FIG> shows an integrated camera image <NUM> and a detached camera image <NUM>, both of which are representative of their corresponding images discussed in the earlier figures.

The integrated camera image <NUM> includes texture <NUM>. As used herein, texture <NUM> generally refers to information regarding the spatial arrangement of color or intensities that are included in the image. Similarly, the detached camera image <NUM> is shown as including texture <NUM>.

In accordance with the alignment <NUM> operation, the embodiments determine that the texture <NUM> in the integrated camera image <NUM> and/or the texture <NUM> in the detached camera image <NUM> is insufficient to perform feature matching or image correspondence matching, as was described in connection with <FIG>. For instance, perhaps an insufficient number of features points are detected in either one of the two images. Additionally, or alternatively, perhaps a sufficient number of feature points were detected but perhaps an insufficient number of correspondences were identified. Based on this initial determination, the embodiments resort or fallback to the alignment <NUM> operation, which is based on a predicted or estimated pose as determined by various inertial measurement units (IMUs).

Specifically, the integrated camera that generated the integrated camera image <NUM> is associated with a first IMU <NUM>. Similarly, the detached camera that generated the detached camera image <NUM> is associated with a second IMU <NUM>. The embodiments utilize the IMU <NUM> to determine a pose of the integrated camera based, perhaps, on an initial bootstrap visual (e.g., an initial base image generated by the integrated camera) in combination with IMU data generated by the IMU <NUM>. Similarly, the embodiments utilize the IMU <NUM> to determine a pose of the detached camera based, perhaps, on an initial bootstrap visual (e.g., an initial base image generated by the detached camera) in combination with IMU data generated by the IMU <NUM>.

Once the two poses are estimated or determined, as shown by IMU-estimated pose <NUM> and IMU-estimated pose <NUM>, the embodiments then use those poses to align one or more portions of the images with one another. Once aligned, then one or more portions of one image (which portions are the aligned portions) are overlaid onto the corresponding portions of the other image in order to generate an enhanced overlaid image. In this regard, IMUs can be used to determine poses of the corresponding cameras, and those poses can then be used to perform the alignment processes. <FIG> illustrates an example flowchart <NUM> for aligning image content from a second image (e.g., the detached camera image <NUM> from <FIG>) with the first image (e.g., the integrated camera image <NUM>) using the IMUs discussed in <FIG>.

Specifically, flowchart <NUM> initially includes an act (act <NUM>) of attempting to identify image correspondences between the second image and the first image. For instance, the embodiments may initially attempt to perform the alignment <NUM> operation discussed in <FIG> in which feature points are attempted to be used for the alignment.

Flowchart <NUM> then includes an act (act <NUM>) of determining one or both of the second image and the first image lack a sufficient amount or threshold amount of texture in order to identify the image correspondences. For instance, the texture <NUM> or <NUM> from <FIG> may not satisfy a texture threshold such that a sufficient number or threshold number of image correspondences cannot be identified.

Flowchart <NUM> then includes an act (act <NUM>) of using a first inertial measurement unit (IMU) of the computer system (e.g., perhaps IMU <NUM> from <FIG>) to estimate an IMU-estimated pose (e.g., IMU-estimated pose <NUM>) of the computer system.

Either in parallel or in serial with act <NUM>, flowchart <NUM> includes an act (act <NUM>) of using a second IMU of the detached camera (e.g., perhaps IMU <NUM>) to estimate an IMU-estimated pose of the detached camera (e.g., IMU-estimated pose <NUM>).

Flowchart <NUM> then includes an act (act <NUM>) of aligning the second image to the first image by aligning the IMU-estimated pose of the computer system with the IMU-estimated pose of the detached camera. In this regard, flowchart <NUM> generally outlines the processes that were discussed in connection with the alignment <NUM> operation of <FIG>. Accordingly, multiple different alignment techniques may be utilized to align image content or to identify image correspondences.

Based on whichever alignment process is used, the embodiments then generate an overlaid image, as shown in <FIG>. Specifically, <FIG> shows an overlaid image <NUM>, which is comprised of image content <NUM> and image content <NUM>. Of course, image content may be pulled or extracted from any number of images that have been aligned within one another, without limit.

The image content <NUM> is extracted, pulled, or drawn from the integrated camera images discussed thus far (e.g., integrated camera image <NUM> of <FIG>) while the image content <NUM> is extracted, pulled, or drawn from the detached camera images discussed thus far (e.g., detached camera image <NUM>). In some cases, the image content <NUM> includes all of the image content from the integrated camera image while in other cases image content <NUM> includes only a portion of the image content from the integrated camera image. Similarly, the image content <NUM> may, in some cases, include all of the image content from the detached camera image while in other cases the image content <NUM> includes only a portion of the image content from the detached camera image.

The amount that is included in the image content <NUM> and <NUM> may, in some cases, be dependent on the degree or level of overlap between the FOVs of the integrated camera and the detached camera. With reference to <FIG>, in this scenario, the FOV <NUM> entirely consumes, overlaps, or envelopes the FOV <NUM>. The resulting integrated camera image might then include, possibly in its entirety, the entire content included in the detached camera image. If only a portion of the two images overlap, then only content associated with that portion may be included in the overlaid image <NUM>.

Generating this overlaid image <NUM> is highly beneficial for a number of reasons. For instance, suppose the image content <NUM> is low light image content or visible light content and suppose the image content <NUM> is thermal imaging content. The thermal imaging content may be used to enhance or supplement the low light or visible light content by providing an increased amount of situational awareness of information regarding the environment.

In some cases, the image content <NUM> and/or the image content <NUM> may be at least partially transparent. For instance, suppose the image content <NUM> is overlaid on top of the image content <NUM>. The image content <NUM> may include content that is currently being overlaid by the image content <NUM>. If the image content <NUM> were at least partially transparent, then both the image content <NUM> and the image content <NUM> will be visually displayed, thereby providing even further visual enhancements or even further visual information. The transparency may be set to any value. For instance, the transparency may be set to <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, <NUM>%, or even up to <NUM>%, or any value in-between.

When an image frame (e.g.. the overlaid image <NUM> from <FIG>) is finished being rendered, the embodiments are able to determine whether the pose depicted in that frame matches with the current pose of the computer system. If the poses match, then the image can be displayed to the user. On the other hand, if the poses do not match, then a late stage reprojection (LSR) process may be performed to transform the pixels in the image to compensate for the new pose. Often, the LSR is performed to correct only for <NUM> DOF changes (e.g., yaw, pitch roll) because objects are often far removed from the HMD such that forward or backward projections can be avoided due to a planar reprojection or planar viewpoint of the objects in the scene (e.g., all objects may be assigned the same planar depth). In some cases, however, the LSR may be performed to correct for <NUM> DOF changes.

To clarify, the process of generating and rendering frames is not an instantaneous process; instead, that process takes some amount of time to execute. For instance, at <NUM> frames per second (FPS), the rendering application or HMD takes approximately <NUM> milliseconds (ms) to render the frame. Although that is a small duration in time, it may be the case that the HMD has shifted position during that time period (e.g., the user may have moved, thereby causing the HMD to move). LSR is a process by which the pixels in the image (e.g., the overlaid image <NUM>) are transformed or modified in order to account for the shift in perspective or pose.

By way of additional clarification, in an effort to reduce or eliminate some rendering errors or issues occurring as a result of differences in pose over time, the HMD is able to apply late stage corrections to make final adjustments to the image after the image is rendered by the GPU. This process is performed before the pixels are displayed so as to compensate for the latest rotation, translation, and/or magnifications resulting from the user's head movement. This adjustment process is often referred to as "Late State Adjustment", "Late Stage Reprojection", "LSR" or "LSR Adjustments. " <FIG> and <FIG> provide some useful illustrations regarding these LSR operations.

<FIG> shows an integrated camera <NUM>, which is representative of the integrated cameras discussed thus far. As discussed earlier, the embodiments are able to determine a pose <NUM> of the integrated camera <NUM> and a timestamp <NUM> of when an image was generated by the integrated camera <NUM>. In this example scenario, the pose <NUM> and timestamp <NUM> are at time T<NUM>.

Prior to the overlaid image being displayed, the HMD has shifted position such that the integrated camera <NUM>, which is representative of the integrated camera <NUM>, has shifted position. Now, the integrated camera <NUM> has a new or a current pose <NUM> and a new or current timestamp <NUM>, which reflects the time Ti. The previous timestamps, such as timestamp <NUM> and timestamp <NUM> from <FIG>, are different from the current timestamp <NUM>. The current pose <NUM> may be determined using any technique, including IMU data, head tracking data, or any other technique for identifying a pose.

The pose difference <NUM> symbolizes the difference between the pose <NUM> and the current pose <NUM>. The pose difference <NUM> may be represented using <NUM> DOF information or <NUM> DOF information. As a result of this detected pose difference <NUM>, the embodiments are triggered to perform LSR. Notably, the LSR may be performed on the integrated camera image, the detached camera image, or the overlaid image. <FIG> shows an example scenario in which the LSR is performed on the overlaid image.

In particular, <FIG> shows an overlaid image <NUM>, which is representative of the overlaid image <NUM> from <FIG>. Overlaid image <NUM> is comprised of any number of pixels <NUM>, such as pixels A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, and P. <FIG> also shows a LSR <NUM> operation being performed on the overlaid image <NUM> to account for a new pose being detected, such as current pose <NUM> from <FIG>.

As a result of performing the LSR <NUM> operation, a LSR-corrected overlaid image <NUM> is generated. Notably, one, some, or all of the pixels in the overlaid image <NUM> have been transformed, as shown by the LSR-corrected pixels <NUM>. For instance, pixel A' is a transformed version of pixel A. Similarly, pixel B' is a transformed version of pixel B. Pixels C', D', E', F', G', H', I', J', K', L', M', N', O', and P' are transformed versions of pixels C, D, E, F, G, H, I, J, K, L, M, N, O, and P, respectively. By performing the LSR <NUM> operation, the embodiments are able to correct or compensate for a detected new pose of the HMD, which includes the integrated camera. <FIG> illustrates a summary illustration of the principles discussed thus far.

Specifically, <FIG> illustrates a first time domain <NUM> and a second time domain <NUM> during which images are generated. The first time domain <NUM> may be associated with the integrated cameras discussed thus far while the second time domain <NUM> may be associated with the detached cameras. For instance, an integrated camera may generate images at a rate of <NUM> while a detached camera may generate images at a rate of <NUM>.

Image 1210A, Image 1210B, Image 1210C, Image 1210D, Image 1210E, Image 1210F, Image <NUM>, Image <NUM>, and Image 1210I may all be generated by the integrated camera throughout a period of time, as shown by the "Time" axis. In this example, the detached camera generates images at a lower or reduced rate, as shown by image 1215A, image 1215B, and image 1215C. For instance, the integrated camera generates three images for every one image the detached camera generates.

In accordance with the disclosed principles, the embodiments then perform the alignment processes described earlier in order to generate overlaid images throughout time. For instance, the embodiments utilize the image 1210B and the image 1215A to generate the overlaid image 1220A. Later, the embodiments use the image 1210C and reuse the same image 1215A to generate the overlaid image 1220B. Later, the embodiments use the image 1210D and reuse the image 1215A to generate the overlaid image 1220C. In this regard, a single image (e.g., image 1215A) may be successively used multiple times in combination with other images to generate overlaid images. The refresh rates of the two cameras can be used to determine how many iterations a single camera image can be reused. With refresh rates of <NUM> and <NUM>, one of the detached camera images can be used at least three times. Other refresh rate ratios will determine the number of times a single image will be used. Notably, the overlaid images are generated by performing the different alignment processes discussed earlier.

To complete the example, the overlaid image 1220D is generated based on a combination of image 1210E and image 1215B. The overlaid image 1220E is generated based on a combination or an alignment of image 1210F and image 1215B. The overlaid image 1220F is generated based on an alignment of image <NUM> and image 1215B. The overlaid image <NUM> is generated based on an alignment of image <NUM> and image 1215C. The overlaid image <NUM> is generated based on a combination of image 1210I and image 1215C.

Subsequently, the embodiments perform a LSR operation on the overlaid image, the integrated camera image, and/or the detached camera image. In the example shown in <FIG>, the embodiments perform LSR on the overlaid images. For instance, LSR 1225A is performed on the overlaid image 1220A, LSR 1225B is performed on the overlaid image 1220B, LSR 1225C is performed on the overlaid image 1220C, LSR 1225D is performed on the overlaid image 1220D, LSR 1225E is performed on the overlaid image 1220E, LSR 1225F is performed on the overlaid image 1220F, LSR <NUM> is performed on the overlaid image <NUM>, and LSR <NUM> is performed on the overlaid image <NUM>.

The LSR-corrected image is then displayed on a display for a user to view. For instance, after the LSR 1225A is performed on the overlaid image 1220A, the embodiments display 1230A the resulting LSR-corrected image. Subsequently, the embodiments display the next LSR corrected image, and so on and so forth as illustrated by display 1230B, display 1230C, display 1230D, display 1230E, display 1230F, display <NUM>, and display <NUM>. Each of these resulting LSR-corrected images are displayed subsequently in time relative to one another, as shown by the Time axis. Similarly, the rate at which the LSR-corrected images are displayed may correspond to the faster rate of either the integrated camera or the detached camera. In this case, the integrated camera refreshes at a faster rate as compared to the detached camera. Consequently, the display of the LSR-corrected images, or rather the rate of display of those images, may correspond to the rate of the integrated camera. In this case, the rate of display of the LSR-corrected images may be <NUM>, just like the rate of the integrated camera.

<FIG> and <FIG> illustrate flowcharts of an example method <NUM> for aligning and stabilizing (e.g., via LSR) images generated by an integrated camera (e.g., any of the integrated cameras discussed thus far) that is physically mounted to the computer system with images generated by a detached camera (e.g., any of the detached cameras discussed thus far) that is physically unmounted from the computer system.

The computer system may be a head-mounted device (HMD) worn by a user. In some implementations, the integrated camera is one camera selected from a group of cameras comprising a visible light camera, a low light camera, or a thermal imaging camera. Similarly, the detached camera is also one camera selected from the group of cameras. Furthermore, the detached camera can be oriented to cause a field of view (FOV) of the detached camera to at least partially overlap a FOV of the integrated camera. Of course, any number of additional mounted or unmounted cameras may be used as well provided their FOVs are also overlapping.

Initially, method <NUM> includes an act (act <NUM>) of generating a first image using the integrated camera. The first image may be representative of any of the integrated camera images discussed thus far.

Method <NUM> then includes an act (act <NUM>) of using the first image to determine a first pose of the computer system. For instance, the pose <NUM> from <FIG> and the pose <NUM> from <FIG> are representative of this "first" pose.

In parallel or in serial with act <NUM>, method <NUM> includes an act (act <NUM>) of determining a first timestamp of the first image. The timestamp <NUM> from <FIG> and the timestamp <NUM> from <FIG> are representative of this first timestamp.

Method <NUM> also includes an act (act <NUM>) of acquiring a second image generated by the detached camera. Act <NUM> may be performed before, after, or during any of acts <NUM>, <NUM>, or <NUM>. Furthermore, any of the disclosed detached camera images are representative of this "second" image. In some cases, method <NUM> includes an act (not illustrated) of determining that the integrated camera and the detached camera are operating at different time domains. Based on this detected difference, the embodiments are able to determine how often a particular image is to be repeatedly used, as was illustrated by the repeated use of image 1215A in <FIG>. As a specific example, the integrated camera may be detected as operating in conjunction with a <NUM> refresh rate for displaying content while the detached camera may be detected as operating in conjunction with a <NUM> refresh rate for displaying content.

Method <NUM> then includes an act (act <NUM>) of aligning the second image to the first image. Any of the alignment processes described in <FIG> or <FIG> may be used to perform the alignment process recited in act <NUM>. For instance, in some cases, aligning the second image to the first image is performed by identifying image correspondences between the second image and the first image, as illustrated in <FIG>. In the situation in which image correspondences are used to align the images, the process of aligning the second image to the first image is performed without a dependency on a timestamp or a pose associated with the second image. Instead, the alignment process is based simply on feature correspondences. In some cases, aligning the two images is based on IMU-estimated poses, as illustrated in <FIG>.

Subsequently, method <NUM> includes an act (<NUM>) of generating an overlaid image by overlaying the second image onto the first image based on the alignment process recited in act <NUM>. Any one of the overlaid images 1220A-<NUM> illustrated in <FIG> as well as the overlaid image <NUM> from <FIG> may be representative of the overlaid image in act <NUM>.

Method <NUM> continues on in <FIG> and includes an act (act <NUM>) of identifying a pose difference between a current pose of the computer system at a current timestamp and the first pose that was determined using the first image at the first timestamp. For instance, the pose difference <NUM> of <FIG> is representative of the pose difference recited in act <NUM>, where the pose difference <NUM> is based on a difference between the pose <NUM> determined at timestamp <NUM> and the current pose <NUM> determined at current timestamp <NUM>.

Method <NUM> then includes an act (act <NUM>) of applying late stage reprojection (LSR) to the overlaid image. The LSR <NUM> in <FIG> is representative of the LSR operation recited in act <NUM>. This LSR process transforms pixels in the overlaid image (e.g., pixels <NUM> of <FIG>) to account for the pose difference identified between the current pose associated with the current timestamp and the first pose associated with the first timestamp. The transformations produce the LSR-corrected pixels <NUM> of <FIG>.

After applying the LSR to the overlaid image, method <NUM> includes an act (act <NUM>) of displaying the overlaid image, which is a type of passthrough image. The image may be displayed in a display of an HMD.

Accordingly, the disclosed embodiments can be used to bring about substantial improvements to how visual content is generated, aligned, and displayed. By way of example, image content from one image can be extracted and overlaid onto another image in order to provide an enhanced visualization for a user. This visualization will enable the user to improve his/her interactions with the computer system. Furthermore, the disclosed alignment processes may be performed irrespective of any timing differences between the images.

Attention will now be directed to <FIG> which illustrates an example computer system <NUM> that may include and/or be used to perform any of the operations described herein. Computer system <NUM> may take various different forms. For example, computer system <NUM> may be embodied as a tablet 1400A, a desktop or a laptop 1400B, a wearable device such as an HMD 1400C (which is representative of the HMDs discussed herein), a mobile device, or any other type of standalone device, as represented by the ellipsis 1400D. Computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with computer system <NUM>.

In its most basic configuration, computer system <NUM> includes various different components. <FIG> shows that computer system <NUM> includes one or more processor(s) <NUM> (aka a "hardware processing unit") and storage <NUM>. Although not illustrated, the computer system <NUM> may include any of the features recited in connection with <FIG> and <FIG>, as well as any other features recited in this disclosure. It should be noted how none of the disclosed features are mutually exclusive and that any feature recited herein may be combined with any other feature recited herein.

Regarding the processor(s) <NUM>, it will be appreciated that the functionality described herein can be performed, at least in part, by one or more hardware logic components (e.g., the processor(s) <NUM>). For example, and without limitation, illustrative types of hardware logic components/processors that can be used include Field-Programmable Gate Arrays ("FPGA"), Program-Specific or Application-Specific Integrated Circuits ("ASIC"), Program-Specific Standard Products ("ASSP"), System-On-A-Chip Systems ("SOC"), Complex Programmable Logic Devices ("CPLD"), Central Processing Units ("CPU"), Graphical Processing Units ("GPU"), or any other type of programmable hardware.

Any type of depth detection may be utilized by the computer system <NUM> and by the scanning sensor(s). Examples include, but are not limited to, stereoscopic depth detection (both active illumination (e.g., using a dot illuminator), structured light illumination (e.g., <NUM> actual camera, <NUM> virtual camera, and <NUM> dot illuminator), and passive (i.e. no illumination)), time of flight depth detection (with a baseline between the laser and the camera, where the field of view of the camera does not perfectly overlap the field of illumination of the laser), range finder depth detection, or any other type of range or depth detection.

Machine learning (ML) may also be utilized by the disclosed embodiments, as discussed earlier. ML may be implemented as a specific processing unit (e.g., a dedicated processing unit as described earlier) configured to perform one or more specialized operations for the computer system <NUM>. As used herein, the terms "executable module," "executable component," "component," "module," "model," or "engine" can refer to hardware processing units or to software objects, routines, or methods that may be executed on computer system <NUM>. The different components, modules, engines, models, and services described herein may be implemented as objects or processors that execute on computer system <NUM> (e.g. as separate threads). ML models and/or the processor(s) <NUM> can be configured to perform one or more of the disclosed method acts or other functionalities.

Storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well.

Storage <NUM> is shown as including executable instructions (i.e. code <NUM>). The executable instructions represent instructions that are executable by the processor(s) <NUM> (or perhaps even a ML model) of computer system <NUM> to perform the disclosed operations, such as those described in the various methods.

The disclosed embodiments may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such as processor(s) <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below. Embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are "physical computer storage media" or a "hardware storage device. " Computer-readable media that carry computer-executable instructions are "transmission media. " Thus, by way of example and not limitation, the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media (aka "hardware storage device") are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSD") that are based on RAM, Flash memory, phase-change memory ("PCM"), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

Computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors (e.g., one or more remote cameras) or devices via a network <NUM>. For example, computer system <NUM> can communicate with any number devices or cloud services to obtain or process data. In some cases, network <NUM> may itself be a cloud network. Furthermore, computer system <NUM> may also be connected through one or more wired or wireless networks <NUM> to remote/separate computer systems(s) that are configured to perform any of the processing described with regard to computer system <NUM>.

A "network," like network <NUM>, is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. Computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

Claim 1:
A computer system (<NUM>) configured to align and stabilize images generated by an integrated camera (<NUM>) that is physically mounted to the computer system (<NUM>) with images generated by a detached camera (<NUM>) that is physically unmounted from the computer system (<NUM>), wherein the computer system is a head-mounted device, HMD, worn by a user, and wherein the detached camera is oriented to cause a field of view, FOV, of the detached camera to at least partially overlap a FOV of the integrated camera, said computer system (<NUM>) comprising:
one or more processors (<NUM>); and
one or more computer-readable hardware storage devices (<NUM>) that store instructions (<NUM>) that are executable by the one or more processors (<NUM>) to cause the computer system (<NUM>) to at least:
generate (<NUM>) a first image (<NUM>) using the integrated camera (<NUM>);
use (<NUM>) the first image (<NUM>) to determine a first pose (<NUM>) of the computer system (<NUM>);
determine (<NUM>) a first timestamp (<NUM>) of the first image (<NUM>);
acquire (<NUM>) a second image (<NUM>) generated by the detached camera (<NUM>);
align (<NUM>) the second image (<NUM>) to the first image (<NUM>);
generate (<NUM>) an overlaid image (<NUM>) by overlaying the second image (<NUM>) onto the first image (<NUM>) based on said aligning;
identify (<NUM>) a pose difference (<NUM>) between a current pose (<NUM>) of the computer system (<NUM>) at a current timestamp (<NUM>) and the first pose (<NUM>) that was determined using the first image (<NUM>) at the first timestamp (<NUM>);
apply (<NUM>) late stage reprojection, LSR, (<NUM>) to the overlaid image (<NUM>) to transform pixels (<NUM>) in the overlaid image (<NUM>) to account for the pose difference (<NUM>) identified between the current pose (<NUM>) associated with the current timestamp (<NUM>) and the first pose (<NUM>) associated with the first timestamp (<NUM>); and
after applying the LSR (<NUM>) to the overlaid image (<NUM>), display (<NUM>) the overlaid image (<NUM>).