Patent Description:
As used herein, VR and AR systems are described and referenced interchangeably. Unless stated otherwise, the descriptions herein apply equally to all types of mixed-reality systems, which (as detailed above) includes AR systems, VR reality systems, and/or any other similar system capable of displaying virtual objects.

Some MR experiences include virtual content (e.g., holograms) presented based on (<NUM>) the positioning of the user (and/or the MR system) relative to a physical environment and/or (<NUM>) the geometry of the physical environment. For example, some AR systems or pass-through VR systems may present virtual content relative real-world objects of an environment in which the user is positioned (or virtual representations thereof), such as a hologram of a figurine standing on a real-world table in the user's environment. An MR system may accurately present of such virtual content (e.g., in a manner that accommodates changes in user perspective) by maintaining a precise awareness of the position and orientation of the MR system (and/or the user) within a physical environment (e.g., with centimeter level accuracy).

To maintain such an awareness, many MR systems rely on simultaneous localization and mapping (SLAM) techniques. SLAM techniques often involve constructing and/or updating a map of a real-world environment while simultaneously tracking the location of an agent (e.g., an MR system) within the environment. In some instances, SLAM systems construct a map of a real-world environment from scratch, while in some instances, SLAM systems at least partially utilize a previously constructed map of a real-world environment.

A map of a real-world environment may include keyframes, which are generated based on captured image frames of the environment. A keyframe may include information about the position and orientation of an MR system that existed while the MR system captured the image frame for generating the keyframe. A keyframe may also be associated with various anchor points (e.g., features extracted from the image frame associated with the keyframe), and the MR system may utilize the anchor points as anchors for measuring pose changes of the MR system (e.g., frame-to-frame pose changes).

To initialize an MR experience using SLAM techniques, an MR system may facilitate user movement throughout a physical environment to allow the MR system to obtain a sufficient number of keyframes of the physical environment to construct a map of the physical environment. However, initial map construction is often time-consuming and significantly limits the versatility of MR systems for providing diverse MR experiences. For example, a user may be exposed to and gain a foreknowledge of a physical environment during the initial map construction process, which may limit the novelty of the MR experience for the user (e.g., the user will have already seen significant portions of the physical environment). For instance, gaming and/or training MR experiences may be limited in scope or effectiveness by requiring users to walk through the physical gaming or training space before the MR activity begins.

To avoid an initial map construction process, as noted above, an MR system may at least partially utilize a previously constructed map of a physical environment to initialize an MR experience. However, in order to use a previously constructed map of a physical environment to initialize an MR experience (e.g., to facilitate accurate hologram presentation), an MR system often must have an awareness of the location of the MR system relative to the physical environment. Stated differently, the MR system must localize itself within the physical environment (an MR system may also need to relocalize itself within the physical environment if tracking fails during the MR experience (e.g., where sensors of the MR system temporarily fail). As used herein, the terms "localize" and "relocalize" are sometimes interchangeably with regard to a MR system identifying its relative position within an environment.

Often, MR systems localize within a physical environment using keyframes of the previously constructed map of the environment, such as by determining feature correspondences to identify a keyframe associated with anchor points that correspond to current anchor points extracted from a current (or recent) image captured by the MR system. However, searching among the keyframes of a map may be computationally intensive and time-consuming, especially for maps of large real-world spaces that include numerous keyframes.

To avoid the time delay associated with localization by determining feature correspondences, described above, some MR experiences prescribe well-defined entrance points within the real-world environment, which indicate where the user should position themselves before initiating the MR experience in order to provide the MR system with an awareness of its location relative to the real-world environment. However, relying on well-defined entrance points also limits the versatility of MR experiences (e.g., removing the possibility of users to seamlessly enter an MR experience from different entrance points that are not predefined by the system).

Accordingly, conventional localization and/or relocalization techniques employed by MR systems may limit the types of experiences that MR system may provide. For example, where a large gaming or training arena is associated with a previously constructed dense map (containing numerous keyframes) for facilitating MR experiences within the arena, conventional MR systems may fail to allow users to seamlessly enter an MR experience in the arena from diverse entrance points that are not predefined by the MR system. Furthermore, conventional MR systems may fail to quickly relocalize if tracking is lost during the MR experience.

For at least the foregoing reasons, there is an ongoing need and desire for improved techniques and systems that may facilitate rapid system localization within an environment, particularly for use in MR experiences associated with maps of real-world environments that include numerous keyframes.

<CIT> describes systems, apparatus and methods disclosed herein facilitate vision based mobile device location determination. In some embodiments, a method for estimating a position of a mobile device may comprise: detecting that the mobile device is in communication with at least one of a plurality of devices, where each of the plurality of devices associated with a corresponding device identifier. The capture of at least one image by an image sensor coupled to the mobile device may be triggered, based, in part on: the device identifier corresponding to the device in communication with the mobile device, and/or a field of view of the image sensor. A location of the mobile device may then be determined, based, in part, on the at least one captured image.

<CIT> describes systems and methods of use pertaining to a visual mapping and transportation management system for determining a location of a user and directing a vehicle to the user's location. Embodiments include a navigation application installed upon a user's mobile computing device and configured to transmit a user image from the device to an image-matching server storing a map composed of keyframes, each having a stored image, a known geometric pose, and numerous extracted interest features. The server also includes a processor configured to extract interest features from the user image, compare the interest features between the user image and the stored images, identify common interest features between the two, and based on the common interest features and known geometric poses of the stored images, determine a global geometric pose of the user image before directing the vehicle to the user's location. Other embodiments are also disclosed.

Disclosed embodiments include systems and methods for using global positioning to accelerate Simultaneous Localization and Mapping (SLAM)-based relocation, including systems and methods for GPS-based and sensor-based relocalization.

Disclosed embodiments include computer-executable instructions that are executable by a system, such as a head-mounted display (HMD), to configure the system to perform methods associated with various acts, including an act of obtaining radio-based positioning data indicating an estimated position of the system within a mapped environment. Disclosed methods also include identifying, based on the estimated position, a subset of keyframes of a map of the mapped environment, wherein the map of the mapped environment includes a plurality of keyframes captured from a plurality of locations within the mapped environment, and the plurality of keyframes are associated with anchor points identified within the mapped environment. Then, the system performs relocalization within the mapped environment based on the subset of keyframes.

Further examples that are not covered by the claimed invention include computer-executable instructions that are executable by a system, such as a head-mounted display (HMD), to configure the system to perform methods associated with various acts, including an act of tracking an estimated position of the system within the environment using a first tracking mode of the system. The disclosed methods also include detecting a presence of a triggering condition for selectively switching from the first tracking mode to a second tracking mode of the system, wherein the triggering condition is at least partially based on first tracking obtained according to the first tracking mode and wherein the second tracking mode comprises a high-fidelity tracking mode relative to the first tracking mode. Then, in response to detecting the presence of the triggering condition, the system selectively activates the second tracking mode of the system and tracking a position of the system within the environment using the second tracking mode.

Disclosed embodiments include computer-executable instructions that are executable by a system, such as a head-mounted display (HMD), to configure the system to perform methods associated with various acts, including an act of generating head tracking data associated with the system, wherein the head tracking data is generated based on at least (<NUM>) visual tracking images obtained using one or more cameras associated with the system and (<NUM>) inertial tracking data obtained using one or more inertial tracking components associated with the system. The disclosed methods also include obtaining radio-based positioning data generated using one or more radio-based positioning components associated with the system while limiting a search space within a map of the mapped environment based on the radio-based positioning data. Then, a pose of the system is determined within the search space using at least a portion of the head tracking data as input.

Disclosed embodiments are generally directed to systems and methods for performing GPS-based and sensor-based relocalization.

Those skilled in the art will recognize, in view of the present disclosure, that at least some of the disclosed embodiments may be implemented to address various shortcomings associated with conventional approaches for facilitating system localization/relocalization within a corresponding environment. The following section outlines some example improvements and/or practical applications provided by the disclosed embodiments. It will be appreciated, however, that the following are examples only and that the embodiments described herein are in no way limited to the example improvements discussed herein.

In some implementations, localizing a system within an environment based on a subset of keyframes of a map of an environment, rather than all keyframes of a map, allows the system to localize within the environment with reduced compute time and/or improved accuracy (as compared with conventional localization techniques). For example, a system may refrain from considering keyframes of the map that are not within the identified subset of keyframes as candidates for localization and may therefore refrain from analyzing anchor points of a potentially significant number of keyframes. By way of illustrative, non-limiting example, a system may reduce compute time for localization by reducing the candidate keyframes from about <NUM>,<NUM> keyframes to about <NUM> keyframes. Furthermore, because the subset of keyframes may be identified based on an estimated position of the system, the chances for false localization results may be reduced according to the presently disclosed embodiments.

Furthermore, by reducing the latency associated with localizing a system within an environment, at least some implementations of the present disclosure may provide versatile systems that may provide diverse MR experiences. For example, a system of the present disclosure may, in some instances, facilitate rapid and/or seamless localization of users entering a physical environment mapped for an MR experience from unmapped locations surrounding the physical environment, even where such entrance points are not predefined for the system. A rapid and/or seamless transition into an MR experience may enable users to rapidly begin engaging with accurately displayed virtual content, rather than waiting for the system to localize to facilitate entry into the MR experience.

Although the present disclosure focuses, in some respects, on facilitating rapid system localization within an MR context, it will be appreciated, in view of the present disclosure, that at least some of the principles disclosed herein are applicable to other tracking implementations, such as, for instance, drone navigation (or other autonomous navigation). For example, at least some embodiments disclosed enable tracking of a system under different tracking modes. A first tracking mode may provide an estimated position with less precision and/or fidelity than a second tracking mode. For instance, a first tracking mode may comprise a GPS tracking mode, and a second tracking mode may comprise SLAM. A system may track its GPS position according to the first tracking mode and intelligently determine whether to activate SLAM based on GPS data. For example, while traveling over regions of an environment that are not of interest or regions where SLAM would be impractical and/or unnecessary (e.g., airspace between a drone departure point and a drone landing point), a system may utilize GPS tracking, thereby avoiding expenditure of computational resources and battery power associated with SLAM. Upon determining that GPS position indicates that the system is within proximity to a target region, region of interest, or other portion of the environment where precise tracking is needed or desired (e.g., a drone landing point), the system may selectively activate SLAM (and may optionally localize the system, such as where the region of interest is a mapped region of a map of the environment for facilitating an MR environment).

Having just described some of the various high-level features and benefits of the disclosed embodiments, attention will now be directed to <FIG>. These Figures illustrate various conceptual representations, architectures, methods, and supporting illustrations related to the disclosed embodiments. The disclosure will then turn to <FIG>, which presents an example computer system that may include and/or be used to facilitate the disclosed principles.

Attention will now be directed to <FIG>, which illustrates an example of a head-mounted device (HMD) <NUM> that includes sensor(s) <NUM> for use within various operational environments (e.g., indoor environment <NUM>, outdoor environment <NUM>). HMD <NUM> can comprise or implement any type of MR system, including a VR system or an AR system. It should be noted that while a substantial portion of this disclosure is focused, in some respects, on the use of an HMD, the embodiments are not limited to being practiced using only an HMD. That is, any type of system can be used, even systems entirely removed or separate from an HMD. As such, the disclosed principles should be interpreted broadly to encompass any type of tracking scenario or device. Some embodiments may even refrain from actively using sensor(s) <NUM> themselves and may simply use the data generated by another sensor device. For instance, some embodiments may at least be partially practiced in a cloud computing environment.

<FIG> illustrates HMD <NUM> as including sensor(s) <NUM>, including camera(s) <NUM>, GPS(s) <NUM>, and inertial measurement unit(s) (IMU(s)) <NUM>. IMU(s) <NUM> may comprise various inertial tracking components, such as accelerometer(s) <NUM>, gyroscope(s) <NUM>, compass(es) <NUM> (e.g., one or more magnetometers), and/or barometer(s) <NUM>. Those skilled in the art will recognize, in view of the present disclosure, that the sensor(s) <NUM> illustrated in <FIG> in association with HMD <NUM> are not necessarily exhaustive, and that an HMD <NUM> may comprise any number of additional or alternative sensor(s) <NUM> in accordance with the present disclosure (e.g., eye tracking systems, microphones, and/or other sensing apparatuses). In some implementations, an HMD <NUM> includes fewer sensors than those depicted in <FIG>.

The accelerometer(s) <NUM>, gyroscope(s) <NUM>, and compass(es) <NUM> are configured to measure inertial tracking data. Specifically, the accelerometer(s) <NUM> is/are configured to measure acceleration, the gyroscope(s) <NUM> is/are configured to measure angular velocity data, and the compass(es) <NUM> is/are configured to measure heading data. In some instances, an HMD <NUM> utilizes the inertial tracking components thereof (e.g., the components of IMU(s) <NUM>)) to obtain three degree of freedom (3DOF) pose data associated with the HMD <NUM> (e.g., where visual tracking data, described below, is unavailable, unreliable, and/or undesired). As used herein, 3DOF refers to position (e.g., rotation) information associated with rotational axes about three perpendicular directional axes (e.g., pitch, yaw, and roll). For example, the 3DOF pose data associated with the HMD <NUM> may comprise an estimated orientation of the HMD <NUM> based on a gravity vector determined from acceleration data, a north orientation from compass data, and rotation derived from gyroscope data.

The inertial tracking components/system of the HMD <NUM> (i.e., the accelerometer(s) <NUM>, gyroscope(s) <NUM>, and compass(es) <NUM>, which may be part of IMU(s) <NUM>) may operate in concert with a visual tracking system to form a head tracking system that generates pose data for the HMD <NUM>. In some instances, a visual tracking system includes one or more cameras (e.g., one or more of camera(s) <NUM>) that capture image data of an environment. In some instances, the HMD <NUM> obtains visual tracking data based on the images captured by the visual tracking system, such as objects within the environment that may provide an anchor for determining movement of the HMD <NUM> relative to the environment.

For example, visual-inertial Simultaneous Localization and Mapping (SLAM) techniques may comprise fusing (e.g., with a pose filter) visual tracking data obtained by one or more cameras (e.g., camera(s) <NUM>) with inertial tracking data obtained by the accelerometer(s) <NUM>, gyroscope(s) <NUM>, and/or compass(es) <NUM> to estimate six degree of freedom (6DOF) positioning (i.e., pose) of the HMD <NUM> in relative to an environment and in real time (or near real time). 6DOF refers to positioning/velocity information associated with three perpendicular directional axes and the three rotational axes (often referred to as pitch, yaw, and roll) about each of the three perpendicular directional axes (often referred to as x, y, and z).

Unless otherwise specified, any reference herein to a "pose" or a related term describing positioning and/or orientation may refer to 3DOF or 6DOF pose.

The visual tracking system of an HMD <NUM>, in some instances, includes a stereo pair of head tracking images (e.g., camera(s) <NUM>) that is configured to obtain depth maps of the user's environment to provide visual mapping of the user's environment. The HMD <NUM> may utilize the visual mapping data of the environment to accurately display virtual content with respect to the user's environment, as well as to facilitate frame-to-frame pose tracking of the HMD <NUM> within the environment. Visual mapping data may also enable location sharing between users in a shared mixed-reality environment.

In some instances, the visual tracking system(s) of an HMD <NUM> (e.g., camera(s) <NUM>) is/are implemented as one or more dedicated cameras. In other instances, the visual tracking system(s) is/are implemented as part of a camera system that performs other functions. Accordingly, an HMD <NUM> may utilize camera(s) <NUM> to scan environments, map environments, capture environmental data, and/or generate any kind of images of the environment. For example, in some instances, the HMD <NUM> is configured to generate a 3D representation of the real-world environment or generate a "pass-through" visualization. Furthermore, the camera(s) <NUM> of an HMD <NUM> may comprise any type of camera(s), such as, by way of non-limiting example, visible light cameras, low light cameras, thermal imaging cameras, ultraviolet cameras, near-infrared cameras, and/or others.

<FIG> also illustrates that HMD <NUM> may comprise GPS(s) <NUM> that obtains GPS data to track a global position of the HMD <NUM>. Although <FIG> depicts GPS(s) <NUM> as a component of HMD <NUM>, those skilled in the art will recognize, in view of the present disclosure, that other forms/types of radio-based positioning systems are within the scope of this disclosure. By way of non-limiting example, an HMD <NUM> can include any form/combination of bearing measurement systems, beam systems, transponder systems, hyperbolic systems, and/or other global navigation satellite systems (e.g., Galileo, QZSS, Beidou, etc.).

<FIG> also illustrates that, in some instances, an HMD <NUM> is associated with a user instrument <NUM>. The user instrument <NUM> may comprise any type of handheld and/or wearable device that is usable in conjunction with the HMD <NUM> (or another system). For example, in some instances, a user instrument <NUM> is a controller, a medical/dental instrument, a first responder tool, etc. In some implementations, the user instrument <NUM> comprises sensor(s) <NUM>, which may correspond in at least some respects to sensor(s) <NUM> of the HMD <NUM>. For example, in some instances, sensor(s) <NUM> of a user instrument <NUM> comprise inertial tracking components, (e.g., similar to IMU(s) <NUM> and/or components thereof), and/or cameras (e.g., similar to camera(s) <NUM>) to facilitate pose tracking of the user instrument <NUM> within an environment (e.g., via SLAM).

In some implementations, the HMD <NUM> and the user instrument <NUM> are configured to share data through a wired or wireless link (e.g., ultra-wideband, WLAN, infrared communication, Bluetooth, and/or others). In this regard, in some instances, sensor obtained by the sensor(s) <NUM> of the HMD <NUM> may be shared and/or associated with the user instrument <NUM>, and/or vice versa. In some instances, at least some sensor devices mounted on or associated with the HMD <NUM> may be leveraged for the user instrument <NUM> as well. In one example, the HMD <NUM> may comprise GPS(s) <NUM>, while the user instrument <NUM> omits a GPS. Notwithstanding, the GPS data obtained by the GPS(s) <NUM> of the HMD <NUM> may be associated with the user instrument <NUM>, such that the GPS(s) <NUM> of the HMD <NUM> is operable to provide GPS tracking of both the HMD <NUM> and the user instrument <NUM> (e.g., operating on the assumption that the user instrument <NUM> stays within relatively close proximity to the HMD <NUM>, such as within a range of about one meter).

Attention is now directed to <FIG>, which illustrates an example environment <NUM> in which a position of an MR system may be tracked. The environment <NUM> includes various regions, such as a wooded area <NUM>, building <NUM>, building <NUM>, and region <NUM> (which includes building <NUM> and portions of the environment <NUM> surrounding the building <NUM>). As will be described in more detail hereinafter, the environment <NUM> is associated with a map of the environment (e.g., map <NUM>, see <FIG>) that includes visual mapping data configured to facilitate MR experiences within at least some portions of the environment <NUM>. For instance, a map of the environment may comprise visual mapping data associated with region <NUM> for facilitating SLAM within region <NUM> of the environment <NUM> (e.g., to facilitate accurate hologram presentation within region <NUM>), whereas the map of the environment <NUM> may at least partially omit visual mapping data associated with the wooded area <NUM> such that the map is not configured to facilitate SLAM within the wooded area <NUM>. In this regard, by way of example, the region <NUM> may be distinguished from the wooded area <NUM> as a target region or region of interest within the environment <NUM>.

<FIG> illustrates a user <NUM> within the environment. Although not explicitly illustrated in <FIG>, the user <NUM> is equipped with an HMD <NUM> and a user instrument <NUM> (or another system) as described hereinabove with reference to <FIG>. Accordingly, in some instances, the HMD <NUM> and/or the user instrument <NUM> are configured to obtain tracking data <NUM> (e.g., via sensor(s) <NUM> and/or <NUM>) to facilitate tracking of the HMD <NUM> and/or the user instrument <NUM> within at least some portions of the environment <NUM>. For example, the HMD <NUM> and/or the user instrument <NUM> associated with the user <NUM> may obtain GPS data <NUM> (e.g., via GPS(s) <NUM>), IMU data <NUM> (e.g., via accelerometer(s) <NUM>, gyroscope(s) <NUM>, compass(es) <NUM>), and/or barometer(s) <NUM>), and/or image data <NUM> (e.g., via camera(s) <NUM>) within the environment <NUM>. In some instances (e.g., where the user approaches or is within region <NUM>), a portion of the tracking data <NUM> may form SLAM data or head tracking data (e.g., based on image data <NUM> of visual tracking images and inertial tracking data, such as IMU data <NUM>).

<FIG> illustrates the user <NUM> traveling between position 250A within the wooded area <NUM> of the environment <NUM> toward position 250B within the region <NUM> (near building <NUM>) of the environment <NUM>. As noted above, a map of the environment may omit map data associated with the wooded area <NUM> of the environment (see <FIG>), which may prevent the system associated with the user <NUM> (e.g., HMD <NUM> and/or user instrument <NUM>) from becoming aware of the user's location within the wooded area <NUM> of the environment <NUM> via SLAM.

Accordingly, in some examples, as the user <NUM> approaches the region <NUM> from the wooded area <NUM> (or from another area for which a map of the environment <NUM> omits visual mapping data, or for which visual mapping data is not intended for use), the system associated with the user <NUM> may need to establish its position relative to the region <NUM> (i.e., the system may need localize or relocalize) in order to facilitate SLAM to provide an accurate MR experience for the user <NUM> within region <NUM>. However, a map of the environment <NUM> may comprise a large amount of visual mapping data for region <NUM> (and/or other portions of the environment <NUM>, such as for building <NUM>), which may cause latency and/or false localization in conventional systems.

Furthermore, at least some systems disclosed herein are configured to use tracking data (e.g., GPS data <NUM> and/or IMU data <NUM>) to intelligently/selectably identify and select a subset of visual mapping data (e.g., a subset of keyframes) to use for localizing or relocalizing within an environment <NUM> (e.g., to facilitate MR experiences therein).

<FIG> illustrates an example map <NUM> associated with the environment <NUM>. As indicated above, the map <NUM> includes visual mapping data within a region <NUM> of the map <NUM>, which corresponds to region <NUM> of the environment <NUM>. For example, the map <NUM> includes visual mapping data in the form of a plurality of keyframes <NUM> and a plurality of anchor points <NUM> within region <NUM>. In some instances, the visual mapping data within the region <NUM> of the map <NUM> are obtained prior to the entry of the user <NUM> to the region <NUM> of the environment <NUM> as illustrated in <FIG> (e.g., the visual mapping data may be obtained at a prior time by the system associated with the user <NUM> and/or by another system). Stated differently, at least region <NUM> of the map <NUM> may be considered to represent a pre-mapping of region <NUM> of the environment <NUM> that is configured to facilitate an MR experience within the region <NUM> upon localization within the region <NUM> (e.g., without first requiring users to capture or obtain new visual mapping data for region <NUM>).

In some implementations, the keyframes <NUM> of the map <NUM> are obtained based on images captured using one or more cameras (e.g., cameras of HMD <NUM>, user instrument <NUM>, and/or another system). For example, in one implementation, a keyframe <NUM> may be obtained based on a stereo pair of images of a captured portion of the environment <NUM> within region <NUM> (e.g., captured by a stereo camera pair). A system may extract features from the stereo pair of images to identify anchor points <NUM> within the captured portion of the environment <NUM> within region <NUM>. In some instances, a feature (sometimes referred to as "keypoints," "points of interest," or "features points") refers to a pixel within an image that comprises rich texture information, such as edges, corners, and/or other readily identifiable structures.

The anchor points <NUM> may provide reference points for tracking pose changes of agents relative to the captured portion of the environment within region <NUM> (e.g., during MR experiences). Furthermore, a system may perform depth calculations on the stereo pair of images to obtain depth data identifying the distances between the various anchor points <NUM> and the stereo camera pair during image capture. From the depth data, a position and orientation of the stereo camera pair relative to the anchor points <NUM> during image capture may be obtained or established. The positioning and orientation of the stereo camera pair relative to the anchor points <NUM> while capturing a stereo pair of images may become stored within the map <NUM> in association with anchor points <NUM> extracted from the captured stereo pair of images. In <FIG>, each keyframe <NUM> is represented as a square located at the position from which the keyframe was captured, and the arrow extending from the square indicates the orientation from which the keyframe was captured. The anchor points <NUM> are represented in <FIG> as points.

Numerous keyframes <NUM> captured from a plurality of locations within the region <NUM>, as well as numerous anchor points <NUM>, may be obtained and stored as part of a map <NUM> of an environment <NUM>, as illustrated in <FIG> for region <NUM> of the map <NUM>, which is representative of region <NUM> of the environment <NUM>. A system associated with the user <NUM> may utilize keyframes <NUM> and anchor points <NUM> to track its pose relative to the mapped portion of the map <NUM> of the environment <NUM>. For example, when an initial position and orientation of the system are not known with confidence (e.g., when the system newly enters the region <NUM>, as indicated in <FIG>, or when the system has lost tracking during an MR experience within the region <NUM>), the system may capture one or more current images of the region <NUM> at a current timepoint (e.g., image data <NUM> from <FIG>).

The system may extract features from the current image(s) and compare those features with the anchor point <NUM> associated with the various keyframes <NUM> of the map <NUM>. The system may identify a particular keyframe <NUM> that is associated with anchor points <NUM> that have a highest correspondence with the features extracted from the current image(s). The system may then estimate its position and orientation within the map <NUM> (i.e., the system may localize or relocalize) based on the position and orientation of the particular keyframe <NUM> associated with the anchor points <NUM> that have the highest correspondence with the features extracted from the current image(s) (see also <FIG> and <FIG> and attendant description).

Subsequently, the system may track frame-to-frame pose changes of the system relative to the anchor points <NUM> (e.g., based on updated image data <NUM> and/or IMU data <NUM>), and the system may use anchor points <NUM> associated with other keyframes <NUM> (e.g., other pre-mapped keyframes <NUM> or new keyframes captured by the system) to maintain accurate tracking as the user <NUM> moves throughout the region <NUM>.

As indicated hereinabove, localizing a system relative to region <NUM> by identifying a particular pre-mapped keyframe <NUM> that is associated with anchor points <NUM> that have the highest correspondence to features extracted from one or more current image(s) (e.g., image data <NUM>) may be computationally expensive and/or time-consuming, particularly for conventional systems that search through all pre-mapped keyframes <NUM> as candidates for localization (and especially where a map <NUM> includes numerous pre-mapped keyframes <NUM>, as expressed in <FIG>). Thus, at least some systems disclosed herein are configured to use tracking data (e.g., GPS data <NUM> and/or IMU data <NUM>) to intelligently select a subset of keyframes <NUM> to use as candidates for localizing or relocalizing.

<FIG> illustrates an estimated position <NUM> of the user <NUM> and an estimated orientation <NUM> of the user <NUM> (or the system associated with the user) relative to the map <NUM> of the environment <NUM> in which the user <NUM> is positioned. The estimated position <NUM> and estimated orientation <NUM> of the user <NUM> at least partially correspond to the position 250B of the user <NUM> within the environment <NUM>. For example, while the user <NUM> is positioned at position 250B within the environment, a system associated with the user <NUM> (e.g., HMD <NUM> and/or user instrument <NUM>) may obtain GPS data <NUM> and/or IMU data <NUM>.

In some implementations, the GPS data <NUM> (and, in some instances, IMU data <NUM> such as barometer data) indicates the estimated position <NUM> of the user <NUM> relative to the map <NUM>. For instance, the GPS data <NUM> may indicate an estimated longitude (or "x" or "east" position) and latitude (or "y" or "north" position) (and, in at least some instances, an estimated altitude, or "z" or "up" position) of the GPS(s) <NUM> associated with the user <NUM>, thereby providing an estimated global position of the user <NUM>. It should be noted that spatial positions within the map <NUM> may be correlated with GPS coordinates, as represented in map <NUM> of <FIG> by the axes indicating latitude, longitude, and altitude.

The correlation between the spatial positions within the map <NUM> and GPS coordinates may be established in various ways, such as during a pre-mapping process (e.g., recording GPS coordinates for systems that capture keyframes <NUM> and/or anchor points for constructing the map <NUM>), manual overlay of GPS coordinates onto the map <NUM>, and/or other ways. Thus, the estimated global position of the user <NUM> within the environment <NUM> may indicate an estimated position <NUM> of the user <NUM> relative to the map <NUM>.

Furthermore, in some implementations, the IMU data <NUM> indicates the estimated orientation <NUM> relative to the map <NUM>. For example, the IMU data <NUM> may comprise heading data obtained by compass(es) <NUM> (e.g., relative to a north vector), which may indicate an estimated heading or yaw of the system associated with the user <NUM>. The IMU data <NUM> may also comprise an estimated pitch and roll of the system based on angular velocity data obtained by gyroscope(s) <NUM> (integrated over time to obtain rotation angles) and a gravity vector obtained by accelerometer(s) <NUM> (e.g., to address drift). <FIG> illustrates a gravity vector and a north vector in association with map <NUM>, indicating that the map <NUM> may be correlated with the reference directions used to generate the estimated orientation <NUM>.

The estimated position <NUM> and/or the estimated orientation <NUM> may provide a basis for identifying a subset of keyframes <NUM> of the map <NUM> that a system searches through for localization. <FIG> illustrates an example of defining a search space <NUM> within the map <NUM>. As is evident in <FIG>, the search space <NUM> is defined based on and/or about the estimated position <NUM> of the system associated with the user <NUM>, described hereinabove.

The search space <NUM> identifies one or more keyframes of the plurality of keyframes <NUM> of the map <NUM> that were captured from positions within the environment <NUM> that correspond to the search space <NUM>. For example, <FIG> shows that keyframes 420A, 420B, 420C, 420D, and 420E are identified as within the search space <NUM> defined based on the estimated position <NUM>. In this way, a system uses the estimated position <NUM> to select keyframes (e.g., keyframes 420A, 420B, 420C, 420D, and 420E) that were captured from locations in environment <NUM> that are within proximity to the position of the user <NUM> within the environment <NUM>. The keyframes 420A, 420B, 420C, 420D, and 420E may comprise a subset of keyframes <NUM> that the system associated with the user <NUM> may utilize as candidates for localizing the system upon entry into the region <NUM> of the environment <NUM>, represented as region <NUM> in the map <NUM>.

Accordingly, in some instances, the system associated with the user <NUM> reduces or limits a search space to include only a subset (rather than all) of the keyframes <NUM> of the map <NUM> for performing localization. Thus, the system may reduce latency and/or computational burden associated with localizing the system relative to the region <NUM> of the environment <NUM>. Furthermore, in some instances, utilizing only a subset of keyframes (e.g., keyframes 420A, 420B, 420C, 420D, and 420E) for localization reduces the chance of false localization (e.g., where an identified keyframe of highest correspondence is associated with a location within the environment <NUM> that is remote from the current position of the user <NUM> within the environment <NUM>).

The size and shape of the search space <NUM> represented in <FIG> is provided as an example only, and other sizes and/or shapes are within the scope of this disclosure. A search space <NUM> can comprise any suitable shape (e.g., a sphere, cylinder, various prism, etc.) of any suitable size. Furthermore, the size and/or shape of the search space <NUM> is intelligently determined based on various factors. According to the invention, a system associated with the user defines a size and/or shape of the search space <NUM> based on a confidence measure associated with radio-based positioning data, for example with the GPS data <NUM>, represented in <FIG> as GPS confidence <NUM>.

A high degree of GPS confidence <NUM> may result in a smaller search space <NUM> (e.g., within a range of about three meters to about two meters or less in length or diameter), whereas a low degree of GPS confidence <NUM> may result in a larger search space <NUM> (e.g., within a range of about three to seven meters or more in length or diameter).

Confidence measures associated with GPS data <NUM> (or other radio-based positioning data) may include one or more of a signal to noise ratio, a number of GNSS formats available to the radio-based positioning device, the radio frequency bands employed by the radio-based positioning device, antenna characteristics, a number of simultaneous GNSS receive channels available to the radio-based positioning device (e.g., the number of satellites in view), the positioning of the satellites in view, the operational mode of the radio-based positioning device, the algorithm used in the radio-based positioning device (e.g., unscented Kalman filter, alpha-beta filter, position averaging filter, one-dimensional Kalman filter), the type of error correction algorithm(s) employed by the radio-based positioning device, and/or others.

In some implementations, a system utilizes additional factors in determining the size and/or shape of the search space <NUM>. In some implementations, a system may define the size and/or shape of the search space <NUM> based on the geometry or other characteristics of the portion of the environment <NUM> (or portion of the map <NUM> of the environment <NUM>) proximate to the estimated position <NUM> of the user <NUM>, represented in <FIG> as region geometry <NUM>.

By way of example, in some instances, the keyframes <NUM> of the map <NUM> that are near (e.g., within a few meters of) the estimated position <NUM> of the user <NUM> may be associated with sparse or few anchor points <NUM> (e.g., for wide open areas of the environment <NUM> or areas of the environment <NUM> that include sparse or few identifiable structures such as edges or corners), which may cause the system to select a larger size for the search space <NUM>. In other instances, the keyframes <NUM> of the map <NUM> that are near the estimated position <NUM> of the user <NUM> may be associated with dense anchor points (e.g., for indoor portions of the environment <NUM> or areas that include dense identifiable structures such as edges or corners), which may cause the system to select a smaller size for the search space <NUM>.

Furthermore, in some instances, a size and/or shape of a search space <NUM> may partially depend on a confidence measure associated with the IMU data <NUM>. For example, where a system defines a search space <NUM> partially based on altitude data obtained by barometer(s) <NUM> of IMU(s) <NUM> of the system associated with the user <NUM>, a barometer confidence <NUM> may influence the size and/or shape of the search space <NUM>.

In addition to reducing the number of keyframes <NUM> of the map <NUM> used for localizing the system associated with the user <NUM> by defining a search space <NUM> and identifying a subset of keyframes within the search space <NUM> (e.g., keyframes 420A, 420B, 420C, 420D, and 420D) as described above, a system may employ additional techniques for further reducing the number of keyframes used to localize the system. <FIG> illustrates an example of identifying a subset of keyframes within the search space <NUM> based on the estimated orientation <NUM> described hereinabove.

As indicated above, each keyframe <NUM> of the map <NUM> may comprise an indication of the orientation from which the keyframe <NUM> was captured, shown in <FIG> as arrows extending from each keyframe <NUM>. Accordingly, the keyframes 420A, 420B, 420C, 420D, and 420D within the search space <NUM> include a keyframe orientation associated with each of the keyframes. A system may compare the estimated orientation <NUM> of the system associated with the user <NUM> to the keyframe orientations of the keyframes within the search space <NUM> (e.g., keyframes 420A, 420B, 420C, 420D, and 420D) to determine which subset of keyframes within the search space <NUM> to use as candidates for localization.

For instance, in some implementations, the system defines a threshold orientation similarity <NUM>. The system may then perform a keyframe orientation analysis <NUM> to determine which keyframes within the search space <NUM> (e.g., keyframes 420A, 420B, 420C, 420D, and 420D) comprise an orientation that meets or exceeds the threshold orientation similarity <NUM>. For example, as illustrated in <FIG>, a keyframe orientation analysis <NUM> may include analyzing an orientation 520A of keyframe 420A to determine whether a difference between the orientation 520A and the estimated orientation <NUM> meets or exceeds the threshold orientation similarity <NUM>. A keyframe orientation analysis <NUM> may comprise performing a similar analysis for other orientations associated with other keyframes, such as orientation 520B of keyframe 420B, orientation 520C of keyframe 420C, orientation 520D of keyframe 420D, and orientation 520D of keyframe 420D.

It should be noted that an orientation of a keyframe may "meet or exceed" a threshold orientation similarity <NUM> in various ways, depending on the format or definition of the threshold orientation similarity <NUM>, which may be somewhat arbitrary. For example, a threshold orientation similarity <NUM> may indicate a maximum orientation difference value from the estimated orientation <NUM>, as depicted in <FIG>. In such examples, a difference between the estimated orientation <NUM> and an orientation of a particular keyframe that is greater than the maximum orientation difference value fails to meet or exceed the threshold orientation similarity <NUM>, whereas a difference between the estimated orientation and an orientation of a particular keyframe that is equal to or less than the maximum orientation difference value meets or exceeds the threshold orientation similarity <NUM>.

In the example shown in <FIG>, a system determines via keyframe orientation analysis <NUM> that orientation 520A of keyframe 420A, orientation 520D of keyframe 420D, and orientation 520E of keyframe 420E meet or exceed the threshold orientation similarity <NUM>, whereas orientation 520B of keyframe 420B and orientation 520C of keyframe 420C fail to meet or exceed the threshold orientation similarity <NUM>. For instance, a difference between orientation 520A and the estimated orientation <NUM> may be equal to or less than a maximum orientation difference value indicated by the threshold orientation similarity <NUM>, such that the keyframe 420A meets or exceeds the threshold orientation similarity <NUM>.

Similarly, a difference between orientation 520D and the estimated orientation <NUM> may be equal to or less than a maximum orientation difference value indicated by the threshold orientation similarity <NUM>, such that the keyframe 420D meets or exceeds the threshold orientation similarity <NUM>.

Additionally, in some embodiments, a difference between orientation 520E and the estimated orientation <NUM> may be equal to or less than a maximum orientation difference value indicated by the threshold orientation similarity <NUM>, such that the keyframe 420E meets or exceeds the threshold orientation similarity <NUM>.

<FIG> illustrates a circle around each of keyframes 420A, 420D, and 420E within the search <NUM>, indicating that keyframes 420A, 420D, and 420E may form a subset of keyframes that the system associated with the user <NUM> may utilize for localization.

In contrast, a difference between orientation 520B and the estimated orientation <NUM> may be greater than a maximum orientation difference value indicated by the threshold orientation similarity <NUM>, such that the keyframe 420B fails to meet or exceed the threshold orientation similarity <NUM>. Furthermore, a difference between orientation 520C and the estimated orientation <NUM> may be greater than a maximum orientation difference value indicated by the threshold orientation similarity <NUM>, such that the keyframe 420C fails to meet or exceed the threshold orientation similarity <NUM>. <FIG> omits a circle around each of keyframes 420B and 420C, indicating that the subset of keyframes that the system associated with the user <NUM> utilizes for localization may omit keyframes 420B and 420C.

Thus, <FIG> illustrates that a system may perform a keyframe orientation analysis <NUM> of the keyframes within the search space <NUM> (e.g., keyframes 420A, 420B, 420C, 420D, and 420E) to determine a subset of keyframes within the search space <NUM> to use for localization. Such techniques may allow a system to further reduce the number of candidate keyframes for localization, which may facilitate localization at reduced computational cost and/or latency.

Although <FIG> focuses, in some respects, on an example in which the threshold orientation similarity <NUM> is represented as a maximum orientation difference value, a threshold orientation similarity <NUM> may be represented in other ways without departing from the principles described herein. For example, a threshold orientation similarity <NUM> may indicate a range of orientation values that includes the estimated orientation <NUM>, and a keyframe may meet or exceed the threshold orientation similarity <NUM> if the keyframe comprises an orientation that is within the range of orientation values indicated by the threshold orientation similarity <NUM>.

In another example, a threshold orientation similarity <NUM> may indicate threshold dot product value, and a keyframe may meet or exceed the threshold orientation similarity <NUM> if the dot product between an orientation of the keyframe and the estimated orientation <NUM> is equal to or greater than the threshold dot product value. A threshold orientation similarity <NUM> may also take on other forms within the scope of this disclosure.

Importantly, it should be noted that a keyframe "meets or exceeds" a threshold orientation similarity <NUM> when an orientation of the keyframe is sufficiently similar to the estimated orientation <NUM>.

<FIG> also illustrates that a threshold orientation similarity <NUM> may be defined or generated based on a confidence measure associated with IMU data <NUM>, represented in <FIG> as IMU confidence <NUM>. A system may determine IMU confidence <NUM> based on various factors, such as an amount of time that has elapsed or an amount of motion that has been sensed since tracking was lost within the environment <NUM> (e.g., for relocalization), an amount of detected acceleration, whether components of the IMU <NUM> were saturated, and/or others.

In some instances, where IMU confidence <NUM> is low, a system defines a broader threshold orientation similarity <NUM> to compensate for potential inaccuracy of the estimated orientation <NUM> (e.g., by defining a high maximum orientation difference value, defining a broad range of orientation values, defining a low threshold dot product value, etc.). In other instances, where IMU confidence is high a system may define a narrow threshold orientation similarity <NUM> (e.g., by defining a low maximum orientation difference value, defining a narrow range of orientation values, defining a high threshold dot product value, etc.), which may further improve efficiency of relocalization processing.

Accordingly, a system associated with the user <NUM> within the environment <NUM> may utilize an estimated position <NUM> (based on GPS data <NUM> and/or IMU data <NUM>) and/or an estimated orientation <NUM> (based on IMU data <NUM>) of the system to limit a localization search space within the map <NUM> of the environment <NUM> to identify a subset of keyframes <NUM> within the map <NUM> to use as candidate keyframes for localization. Thus, the system may refrain from using keyframes <NUM> of the map <NUM> that are not included in the subset of keyframes <NUM> as candidate keyframes for localization, which may improve efficiencies associated with localization within the environment <NUM>.

Continuing with the example shown in <FIG>, <FIG> and <FIG> illustrate examples of identifying a keyframe of highest correspondence from the subset of keyframes (e.g., keyframes 420A, 420D, and 420E) identified as candidate keyframes for localization, as described hereinabove. As noted above, a system may identify a keyframe of highest correspondence as part of localizing or relocalizing the system within an environment (e.g., environment <NUM>). <FIG> illustrates a keyframe image <NUM> captured by a system (e.g., HMD <NUM>, user instrument <NUM>, or another system) used to obtain visual mapping data to construct the map <NUM> of the environment <NUM>.

As noted above, a keyframe <NUM> may be obtained based on one or more images of a captured portion of an environment (e.g., environment <NUM>). In the example shown in <FIG>, keyframe image <NUM> is an image captured by a system to add keyframe 420E to the map <NUM> of <FIG>. Accordingly, keyframe image <NUM> captures a portion of the building <NUM> of the environment <NUM>.

A system may extract features from keyframe image <NUM> to identify anchor points within the captured portion of the environment. <FIG> illustrates example anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM>. The anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM> may provide reference points for tracking pose changes of agents relative, and the anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM> may also facilitate localization or relocalization.

For example, <FIG> illustrates an HMD image <NUM>, which represents image data <NUM> captured by the system associated with the user <NUM> while the user was located at position 250B within the environment <NUM>. Thus, HMD image <NUM> also captures a portion of the building <NUM> of the environment <NUM>.

When attempting to localize or relocalize, in some implementations, a system extracts feature points from the HMD image <NUM>. <FIG> illustrates example feature points 620A, 620B, 620C, 620D, 620E, 620F, and <NUM>. In some instances, the system determines feature correspondences (e.g., performs feature matching) to identify one or more keyframes of the subset of keyframes (e.g., keyframes 420A, 420D, and 420E) that are associated with anchor points that correspond to the feature points identified from the HMD image <NUM>.

A feature point and an anchor point correspond to one another when they both describe a same portion of an environment. In some instances, a system determines whether a feature point and an anchor point correspond to one another based on a comparison between a feature descriptor associated with the feature point and a feature descriptor associated with the anchor point.

In some instances, a feature descriptor (also referred to as a "feature vector") results from extracting image data/statistics from a local image/pixel patch around an identified feature point (or anchor point). A feature descriptor may operate as an identifier for the feature point about which the feature descriptor is centered. Various approaches exist for extracting feature descriptors, such as local histogram approaches, N-jets approaches, and/or others. For example, a feature descriptor may be identified based on a histogram of gradient magnitudes (e.g., changes in intensity and/or color) and/or orientations (e.g., edge orientations) for pixels within an image patch centered on a feature point (or anchor point).

<FIG> illustrates feature descriptors as dashed boxes that surround the various feature points (feature points 620A, 620B, 620C, 620D, 620E, 620F, and <NUM>) and anchor points (anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM>).

In the example shown in <FIG>, the system associated with the user <NUM> compares feature descriptors of the feature points of the HMD image <NUM> (e.g., feature points 620A, 620B, 620C, 620D, 620E, 620F, and <NUM>) to the anchor points of the keyframe image <NUM> (e.g., anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM>). The system determines that, for example, feature points 620A, 620B, 620C, and 620D describe the same corners of the window of the building <NUM> as anchor points 660A, 660B, 660C, and 660D, respectively, feature points 620E and 620F describe the same portions of the roof of the building <NUM> as anchor points 660E and 660F, respectively, and feature point <NUM> describes the same vertical edge portion of the building <NUM> as anchor point <NUM>. (It is noted that anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM> correspond to the premapping of the environment, such as through keyframe 420E of <FIG>, or other keyframes).

The system may also compare the feature descriptors of the feature points of the HMD image <NUM> to anchor points associated with other keyframes of the subset of keyframes (e.g., keyframes 420A and 420D) to determine a keyframe of highest correspondence to the HMD image <NUM>. In some implementations, a keyframe of highest correspondence is identified based on a number of anchor points associated therewith that correspond to the feature points of the HMD image <NUM>, based on one or more similarity measures between one or more of the feature descriptors associated with the anchor points of the keyframe and one or more feature descriptors associated with the feature points of the HMD image <NUM>, and/or other factors. By way of example, the system may determine that keyframe 420D is the keyframe of highest correspondence.

As noted above, a position and orientation of the keyframe of highest correspondence may be stored within the map <NUM>. Accordingly, the system associated with the user <NUM> may estimate the position and orientation (e.g., the pose) of the HMD <NUM> relative to the environment based on at least the position and orientation of the keyframe of highest correspondence and/or the anchor points associated with the keyframe of highest correspondence, thereby localizing or relocalizing the HMD <NUM> relative to the map <NUM> of the environment <NUM>.

<FIG> focuses, in at least some respects, on localizing the HMD <NUM> of the user <NUM> at position 250B within the environment <NUM> using at least the HMD image <NUM> and the subset of keyframes described hereinabove (e.g., keyframes 420A, 420B, and 420E). However, at least some of the principles described herein may be applied to localize a user instrument <NUM> associated with the HMD <NUM> (and/or the user <NUM>). For example, the sensor(s) <NUM> of a user instrument <NUM> of the user <NUM> may comprise dedicated camera(s), IMU(s), and GPS(s), such that the techniques and principles described herein for localization and relocalization may be applied to the user instrument <NUM> independent of the HMD <NUM>.

In other instances, the sensor(s) <NUM> of the user instrument <NUM> may comprise dedicated camera(s) and IMU(s) but omit a GPS. However, in some instances, GPS(s) <NUM> mounted on the HMD <NUM> may be leveraged for localizing or relocalizing the user instrument <NUM>, in addition to localizing or relocalizing the HMD <NUM>. For example, in some instances, a system associated with the user <NUM> may operate on an assumption that the user instrument <NUM> is within a predetermined distance to the HMD <NUM> (e.g., within a few meters or less). Accordingly, an estimated position of the HMD <NUM> determined based on GPS data associated with the HMD <NUM> may be used to define a search space to identify a subset of keyframes for localizing the user instrument <NUM>.

In some instances, the search space for localizing the user instrument <NUM> may be broadened to account for the lack of a dedicated GPS mounted on the user instrument <NUM>. The system may then utilize IMU data specific to the user instrument <NUM> to further narrow the subset of keyframes based on an estimated orientation of the user instrument <NUM> (e.g., in the manner described hereinabove with reference to <FIG>).

One will appreciate that the subset of keyframes for localizing the user instrument <NUM> may at least partially differ from the subset of keyframes for localizing the HMD <NUM>. For example, <FIG> illustrates a user instrument image <NUM> captured by one or more cameras of the user instrument <NUM>. As is evident from <FIG>, the perspective from which the user instrument <NUM> captures the building <NUM> in the user instrument image <NUM> differs from the perspective from which the HMD <NUM> captures the building <NUM> in the HMD image <NUM>. Thus, a different estimated orientation may apply to the user instrument <NUM> as compared to the estimated orientation of the HMD <NUM>. Thus, different keyframes may be considered to be sufficiently similar in orientation to the estimated orientation of the user instrument <NUM> as compared to the estimated orientation of the HMD <NUM>.

After identifying a subset of keyframes to use for localizing the user instrument <NUM>, a system may identify feature points (e.g., feature points 720A, 720B, 720C, 720D, 720E, 720F, <NUM>) within a user instrument image <NUM> captured by one or more cameras of the user instrument <NUM>, as shown in <FIG>. The system may then compare feature descriptors of the feature points of the user instrument image <NUM> to the anchor points of the keyframe image <NUM> (e.g., anchor points 660A, 660B, 660C, 660D, 660E, 660F, and <NUM>). The system determines that, for example, feature points 720A, 720B, 720C, and 720D describe the same corners of the window of the building <NUM> as anchor points 660A, 660B, 660C, and 660D, respectively, feature points 720E and 720F describe the same portions of the roof of the building <NUM> as anchor points 660E and 660F, respectively, and feature point <NUM> describes the same vertical edge portion of the building <NUM> as anchor point <NUM>.

The system may also compare the feature descriptors of the feature points of the user instrument image <NUM> to anchor points associated with other keyframes of the subset of keyframes for localizing the user instrument <NUM> to determine a keyframe of highest correspondence to the user instrument image <NUM>. The system may then estimate the position and orientation (e.g., the pose) of the user instrument <NUM> relative to the environment based on at least the position and orientation of the keyframe of highest correspondence and/or the anchor points associated with the keyframe of highest correspondence, thereby localizing or relocalizing the user instrument <NUM> relative to the map <NUM> of the environment <NUM>.

In some instances, after identifying a subset of keyframes to localize the HMD <NUM> or the user instrument <NUM> by defining a search space <NUM> within the map <NUM>, a system associated with the user <NUM> may fail to localize or relocalize the HMD <NUM> or the user instrument <NUM> based on the keyframes of the identified subset of keyframes. Thus, in some implementations, the system is configured to modify the search space <NUM> to identify additional keyframes <NUM> as candidates for localization. For example, a system may increase the size of the search space <NUM> in response to failing to localize or relocalize.

As indicated above with reference to <FIG>, a system associated with the user <NUM> (e.g., the HMD <NUM>, the user instrument <NUM>, and/or another system) may obtain tracking data <NUM> (comprising GPS data <NUM>, IMU data <NUM>, and image data <NUM>) while the user is at position 250B within the environment <NUM>. In some implementations, the system associated with the user <NUM> is configured to selectively transition between different tracking modes within the environment.

In some instances, selectively transitioning between different tracking modes may avoid unnecessary data acquisition and thereby save battery and/or computational resources. For example, <FIG> illustrates the environment <NUM> from <FIG>, depicting the user <NUM> moving from position 250A within the wooded area <NUM> of the environment to position 250B of region <NUM> of the environment <NUM>.

As indicated hereinabove, a map <NUM> of the environment <NUM> may omit visual mapping data for facilitating localization within the wooded area <NUM> of the environment. Accordingly, <FIG> illustrates that a system associated with the user <NUM> may track the position of the system within the environment <NUM> using a first tracking mode <NUM> while the user is positioned within the wooded area <NUM> of the environment <NUM>.

In some instances, the first tracking mode <NUM> comprises GPS tracking <NUM> (the first tracking mode <NUM> may, in some instances, comprise IMU tracking <NUM>, indicated in <FIG> with dashed lines). In some implementations, when operating under the first tracking mode <NUM>, the system refrains from performing operations associated with SLAM, such as capturing image data, extracting features, performing depth calculations, tracking frame-to-frame 6DOF pose, etc., thereby saving on computational and/or battery resources. For example, the system may rely on GPS tracking <NUM> to maintain coarse positional awareness of the system relative to the environment <NUM>, and the system may refrain from obtaining image data while the GPS tracking <NUM> indicates that the system is substantially within the wooded area <NUM> (e.g., because localization within the wooded area may be hindered in view of the map <NUM> omitting visual mapping data for the wooded area <NUM>).

It will be appreciated, in view of the present disclosure, that the system may operate in under the first tracking mode <NUM> in other portions of the environment <NUM> for which visual mapping data does exist or is not intended for use.

As the user <NUM> transitions from position 250A within the wooded area <NUM> of the environment <NUM> to position 250B within region <NUM> of the environment <NUM>, the system associated with the user may detect a triggering condition <NUM>. In some implementations, the triggering condition <NUM> is configured for selectively switching from the first tracking mode <NUM> to a second tracking mode <NUM>. In some instances, the second tracking mode <NUM> comprises a high-fidelity tracking mode relative to the first tracking mode <NUM>. For example, the second tracking mode <NUM> may comprise SLAM <NUM>, which utilizes image data <NUM> and IMU data <NUM> to track 6DOF pose of the system with centimeter or millimeter precision (e.g., in contrast to GPS tracking <NUM> of the first tracking mode <NUM>, which provides global position tracking of the system precision in the range of a few meters). The second tracking mode <NUM> may, in some instances, utilize GPS data <NUM>, indicated in <FIG> with dashed lines.

In some implementations, the triggering condition <NUM> is at least partially based on data obtained via GPS tracking <NUM> according to the first tracking mode <NUM> (and/or other tracking data obtained via the first tracking mode <NUM>). For example, in some instances, the triggering condition includes determining that an estimated position of the system (e.g., an estimated position based on GPS tracking <NUM>) is within a threshold proximity to a particular portion of the environment <NUM> or map <NUM> of the environment <NUM>.

In some instances, the particular portion of the environment <NUM> or map <NUM> of the environment <NUM> is a target region within the environment, which may comprise a pre-mapped region of the environment <NUM> for which a map <NUM> of the environment <NUM> includes visual mapping data (e.g., region <NUM> of the environment <NUM>, for which map <NUM> comprises keyframes <NUM> and anchor points <NUM>).

As noted above, a map <NUM> of the environment may be correlated with GPS coordinates, such that the system may determine the global position of the system relative to keyframes, anchor points, and/or other data represented in the map <NUM>. Thus, in some implementations, a triggering condition <NUM> comprises a determination that one or more keyframes <NUM> and/or anchor points <NUM> stored within a map <NUM> of the environment are within a threshold proximity to a global position of the system obtained via GPS tracking <NUM> under the first tracking mode <NUM> (e.g., a determination that one or more keyframes <NUM> are within a search space <NUM> for the system based on an updateable estimated position <NUM> of the system).

In some instances, in response to detecting the presence of the triggering condition <NUM> described above, the system associated with the user <NUM> may selectively activate the second tracking mode <NUM> and thereafter continue to track a high-fidelity position (e.g., a 6DOF pose) of the system relative to the environment <NUM> using the second tracking mode <NUM> (e.g., via SLAM <NUM>). In some instances, upon activating the second tracking mode <NUM>, obtains image data <NUM> and IMU data <NUM> to localize itself relative to the environment <NUM> by defining a search space <NUM> based on an estimated position <NUM> of the system (based on GPS data <NUM>) and identifying a subset of keyframes <NUM> within the search space <NUM> based on an estimated orientation <NUM> of the system (based on IMU data <NUM>) as candidate keyframes for localization.

Although <FIG> focuses, in at least some respects, on selectively transitioning between tracking modes for a system associated with user <NUM> (e.g., HMD <NUM> and/or user instrument <NUM>), the principles described herein may be applicable to other devices. For example, a drone or other autonomous device may operate under a first tracking mode <NUM> while flying through the air in transit to a target region (e.g., a landing or delivery region). Then, based on tracking data obtained under the first tracking mode, the drone may determine, as a triggering condition <NUM>, that the drone has entered or is within sufficient proximity to the target region (e.g., the drone may have entered a target region of GPS coordinates). In response to the triggering condition, the drone may activate a second tracking mode <NUM> and enable SLAM to provide high-fidelity tracking functionality (e.g., to enable the drone to land relative to structures within the environment and/or accurately deliver a package).

<FIG> illustrates an example flow diagram <NUM> depicting acts associated with methods for facilitating relocalization within a mapped environment. The discussion of the various acts represented in flow diagram <NUM> includes references to various hardware components described in more detail with reference to <FIG> and <FIG>.

Act <NUM> of flow diagram <NUM> includes obtaining radio-based positioning data indicating an estimated position. Act <NUM> is performed, in some instances, using GPS(s) <NUM> and/or one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, the radio-based positioning data comprises global positioning system (GPS) data obtained by a GPS device associated with the system. In some instances, the estimated position indicated by the radio-based positioning data is a position relative to a map of a mapped environment.

Act <NUM> of flow diagram <NUM> includes obtaining inertial tracking data indicating an estimated orientation. Act <NUM> is performed, in some instances, using IMU(s) <NUM> and/or one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, the inertial tracking data is obtained by one or more inertial tracking components, such as accelerometer(s) <NUM>, gyroscope(s) <NUM>, compass(es) <NUM>, and/or barometer(s) <NUM>, which may comprise components of IMU(s) <NUM>. In some instances, the estimated orientation indicated by the inertial tracking data is an orientation relative to a map of a mapped environment.

Act <NUM> of flow diagram <NUM> includes identifying a subset of keyframes of a map. Act <NUM> is performed, in some instances, using one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, the subset of keyframes of the map are identified based on the estimated position and/or the estimated orientation. The map may comprise a map of a mapped environment and may comprise a plurality of keyframes captured from a plurality of locations within the mapped environment. The plurality of keyframes is, in some instances, associated with anchor points identified within the mapped environment.

Identifying a subset of keyframes of a map according to act <NUM> may include various acts. For example, flow diagram <NUM> illustrates that act 906A includes defining a search space based on the estimated position. According to the invention, defining the search space is based on a confidence measure associated with the radio-based positioning data. Also, in some instances, defining of the search space is partially based on a geometry of a region of the mapped environment proximate to the estimated position. Additionally, in some instances, the search space is partially based on barometer data.

Act 906B includes identifying one or more keyframes based on the search space. In some instances, the one or more keyframes are captured from one or more positions within the mapped environment that correspond to the search space.

Act 906C includes identifying one or more keyframes that meet or exceed a threshold orientation similarity to the estimated orientation. In some instances, the threshold orientation similarity comprises a maximum orientation difference value. In some instances, the threshold orientation similarity comprises a range of orientation values. In some instances, the threshold orientation similarity comprises a dot product value. In some implementations, the threshold orientation similarity is based on a confidence measure associated inertial tracking data.

Act <NUM> of flow diagram <NUM> includes relocalizing the system within the mapped environment based on the subset of keyframes. Act <NUM> is performed, in some instances, using one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). Act <NUM> may also comprise refraining from relocalizing the system within the mapped environment based on keyframes of the plurality of keyframes of the map that are not included in the subset of keyframes, thereby reducing computational cost and/or latency.

Relocalizing the system within the mapped environment based on the subset of keyframes according to act <NUM> may include various acts. For example, flow diagram <NUM> illustrates that act 908A includes determining a keyframe of highest correspondence. In some instances, determining a keyframe of highest correspondence comprises determining feature correspondences between the one or more visual tracking images and anchor points associated with each keyframe of the subset of keyframes.

Act 908B includes estimating a pose relative to the keyframe of highest correspondence. For example, the pose may be estimated relative to an orientation and position associated with the keyframe of highest correspondence, and/or relative to one or more anchor points associated with the keyframe of highest correspondence.

Flow diagram <NUM> illustrates arrow <NUM> extending from act <NUM> to act <NUM>, indicating that, in some instances, in response to failing to relocalize the system within the mapped environment based on the subset of keyframes, a system may modify the subset of keyframes to include additional keyframes of the plurality of keyframes.

<FIG> illustrates an example flow diagram <NUM> depicting acts associated with transitioning between tracking modes within an environment. The discussion of the various acts represented in flow diagram <NUM> includes references to various hardware components described in more detail with reference to <FIG> and <FIG>.

Act <NUM> of flow diagram <NUM> includes tracking an estimated position using a first tracking mode. Act <NUM> is performed, in some instances, using sensor(s) <NUM> and/or <NUM> and/or one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, the first tracking mode comprises GPS tracking.

Act <NUM> of flow diagram <NUM> includes detecting a presence of a triggering condition based on first tracking data obtained according to the first tracking mode. Act <NUM> is performed, in some instances, using sensor(s) <NUM> and/or <NUM> and/or one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, the triggering condition is at least partially based on first tracking data obtained according to the first tracking mode (e.g., GPS data).

In some instances, the triggering condition is operable for selectively switching from the first tracking mode to a second tracking mode of the system. In some instances, the second tracking mode comprises a high-fidelity tracking mode relative to the first tracking mode. For example, in some instances, the second tracking mode comprises simultaneous localization and mapping (SLAM). In some implementations, the triggering condition comprises a determination that the estimated position of the system tracked using the first tracking mode of the system is within a threshold proximity to either (<NUM>) a target region within the environment or (<NUM>) a pre-mapped portion of the environment.

Act <NUM> of flow diagram <NUM> includes selectively activating a second tracking mode. Act <NUM> is performed, in some instances, using sensor(s) <NUM> and/or <NUM> and/or one or more processor(s) <NUM> (e.g., of an HMD <NUM>, a user instrument <NUM>, and/or another system). In some instances, selectively activating the second tracking mode is performed in response to detecting the presence of the triggering condition. As noted above, the second tracking mode may comprise simultaneous location and mapping (SLAM). In some instances, activating the second tracking mode includes tracking a position of the system within the environment using the second tracking mode. Tracking the position of the system within the environment using the second tracking mode comprises localizing the system within the environment based on one or more keyframes associated with the pre-mapped portion of the environment.

Having just described the various features and functionalities of some of the disclosed embodiments, the focus will now be directed to <FIG> which illustrates an example computer system <NUM> that may include and/or be used to facilitate the embodiments described herein, including the acts described in reference to the foregoing Figures. In some implementations, this computer system <NUM> may be implemented as part of a mixed-reality HMD, or any other systems described herein.

Computer system <NUM> may take various different forms. For example, computer system <NUM> may be embodied as a tablet, a desktop, a laptop, a mobile device, a cloud device, an HMD, or a standalone device, such as those described throughout this disclosure. Computer system <NUM> may also be a distributed system that includes one or more connected computing components/devices that are in communication with computer system <NUM>. <FIG> specifically calls out how computer system <NUM> may be embodied as a tablet 1100A, a laptop 1100B, or an HMD 1100C, but the ellipsis 1100D indicates that computer system <NUM> may be embodied in other forms as well.

The computer system <NUM> includes various different components. <FIG> shows that computer system <NUM> includes one or more processors <NUM> (aka a "hardware processing unit"), a machine learning (ML) engine <NUM>, graphics rendering engine(s) <NUM>, a display system <NUM>, input/output (I/O) interfaces <NUM>, one or more sensors <NUM>, and storage <NUM>.

Regarding the processor(s) <NUM>, it will be appreciated that the functionality described herein can be performed, at least in part, by one or more hardware logic components (e.g., the processor(s) <NUM>). For example, and without limitation, illustrative types of hardware logic components/processors that can be used include Field-Programmable Gate Arrays ("FPGA"), Program-Specific or Application-Specific Integrated Circuits ("ASIC"), Application-Specific Standard Products ("ASSP"), System-On-A-Chip Systems ("SOC"), Complex Programmable Logic Devices ("CPLD"), Central Processing Units ("CPU"), Graphical Processing Units ("GPU"), or any other type of programmable hardware.

As used herein, the terms "executable module," "executable component," "component," "module," or "engine" can refer to hardware processing units or to software objects, routines, or methods that may be executed on computer system <NUM>. The different components, modules, engines, and services described herein may be implemented as objects or processors that execute on computer system <NUM> (e.g. as separate threads).

The ML engine <NUM> may be implemented as a specific processing unit (e.g., a dedicated processing unit as described earlier) configured to perform one or more specialized operations for the computer system <NUM>. The ML engine <NUM> (or perhaps even just the processor(s) <NUM>) can be configured to perform any of the disclosed method acts or other functionalities.

In some instances, the graphics rendering engine <NUM> is configured, with the hardware processing unit <NUM>, to render one or more virtual objects within the scene. As a result, the virtual objects accurately move in response to a movement of the user and/or in response to user input as the user interacts within the virtual scene. The computer system <NUM> may include a display system <NUM> (e.g., laser diodes, light emitting diodes (LEDs), microelectromechanical systems (MEMS), mirrors, lens systems, diffractive optical elements (DOEs), display screens, and/or combinations thereof) for presenting virtual objects within the scene.

I/O interface(s) <NUM> includes any type of input or output device. Such devices include, but are not limited to, touch screens, displays, a mouse, a keyboard, a controller, and so forth. Any type of input or output device should be included among I/O interface(s) <NUM>, without limitation.

During use, in some instances, a user of the computer system <NUM> is able to perceive information (e.g., a mixed-reality environment) through a display screen that is included among the I/O interface(s) <NUM> and that is visible to the user. The I/O interface(s) <NUM> and sensors <NUM>/<NUM> may also include gesture detection devices, eye tracking systems, and/or other movement detecting components (e.g., head tracking cameras, depth detection systems, gyroscopes, accelerometers, magnetometers, acoustic sensors, global positioning systems ("GPS"), etc.) that are able to detect positioning and movement of one or more real-world objects, such as a user's hand, a stylus, and/or any other object(s) that the user may interact with while being immersed in the scene.

The computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors <NUM> (e.g., one or more remote cameras, accelerometers, gyroscopes, acoustic sensors, magnetometers, etc.). It will be appreciated that the external sensors include sensor systems (e.g., a sensor system including a light emitter and camera), rather than solely individual sensor apparatuses.

Storage <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If computer system <NUM> is distributed, the processing, memory, and/or storage capability may be distributed as well.

Storage <NUM> is shown as including executable instructions (i.e. code <NUM>). The executable instructions (i.e. code <NUM>) represent instructions that are executable by the processor(s) <NUM> of computer system <NUM> to perform the disclosed operations, such as those described in the various methods. Storage <NUM> is also shown as including data <NUM>. Data <NUM> may include any type of data, including image data, depth/disparity maps and/or other depth data, pose data, tracking data, and so forth, without limitation.

The disclosed embodiments may comprise or utilize a special-purpose or general-purpose computer including computer hardware, such as, for example, one or more processors (such as processor(s) <NUM>) and system memory (such as storage <NUM>), as discussed in greater detail below.

Disclosed embodiments also include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures. Such computer-readable media can be any available media that can be accessed by a general-purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions in the form of data are "physical computer storage media" or a "hardware storage device. " Computer-readable media that merely carry computer-executable instructions without storing the computer-executable instructions are "transmission media. " Thus, by way of example and not limitation, the current embodiments can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

Computer storage media (aka "hardware storage device") are computer-readable hardware storage devices, such as RAM, ROM, EEPROM, CD-ROM, solid state drives ("SSD") that are based on RAM, Flash memory, phase-change memory ("PCM"), or other types of memory, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code means in the form of computer-executable instructions, data, or data structures and that can be accessed by a general-purpose or special-purpose computer.

Computer system <NUM> may also be connected (via a wired or wireless connection) to external sensors (e.g., one or more remote cameras) or devices via a network <NUM>. For example, computer system <NUM> can communicate with any number devices or cloud services to obtain or process data. In some cases, network <NUM> may itself be a cloud network. Furthermore, computer system <NUM> may also be connected through one or more wired or wireless networks <NUM> to remote/separate computer systems(s) <NUM> that are configured to perform any of the processing described with regard to computer system <NUM>.

A "network," like network <NUM>, is defined as one or more data links and/or data switches that enable the transport of electronic data between computer systems, modules, and/or other electronic devices. When information is transferred, or provided, over a network (either hardwired, wireless, or a combination of hardwired and wireless) to a computer, the computer properly views the connection as a transmission medium. Computer system <NUM> will include one or more communication channels that are used to communicate with the network <NUM>. Transmissions media include a network that can be used to carry data or desired program code means in the form of computer-executable instructions or in the form of data structures. Further, these computer-executable instructions can be accessed by a general-purpose or special-purpose computer.

Computer-executable (or computer-interpretable) instructions comprise, for example, instructions that cause a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a certain function or group of functions.

One will also appreciate how any feature or operation disclosed herein may be combined with any one or combination of the other features and operations disclosed herein, within the scope of the appended claims.

Claim 1:
A system (<NUM>, <NUM>, <NUM>) for facilitating relocalization within a mapped environment (<NUM>), the system comprising:
one or more processors (<NUM>); and
one or more hardware storage devices (<NUM>) storing computer-executable instructions (<NUM>) that are executable by the one or more processors (<NUM>) to configure the system (<NUM>, <NUM>, <NUM>) to facilitate relocalization within a mapped environment (<NUM>) by at least configuring the system (<NUM>, <NUM>, <NUM>) to perform the following:
obtain radio-based positioning data (<NUM>) indicating an estimated position (<NUM>) of the system within the mapped environment (<NUM>);
identify, based on the estimated position (<NUM>), a subset of keyframes (420A, 420B, 420C, 420D, 420E) of a map (<NUM>) of the mapped environment (<NUM>), wherein the keyframes correspond to images of the mapped environment, the map (<NUM>) of the mapped environment (<NUM>) comprising a plurality of keyframes (<NUM>) captured from a plurality of locations within the mapped environment (<NUM>), the plurality of keyframes (<NUM>) being associated with anchor points (<NUM>) identified within the mapped environment (<NUM>), wherein the anchor points (<NUM>) correspond to features in the mapped environment, wherein identifying the subset of keyframes of the map of the mapped environment comprises:
defining a search space within the map based on the estimated position, wherein the defining of the search space is based on a confidence measure associated with the radio-based positioning data; and
identifying one or more keyframes of the plurality of keyframes, the one or more keyframes being captured from one or more positions within the mapped environment that correspond to the search space;
and
relocalize the system (<NUM>, <NUM>, <NUM>) within the mapped environment (<NUM>) based on the subset of keyframes (420A, 420B, 420C, 420D, 420E) by determining a keyframe of highest correspondence to one or more visual tracking images captured by one or more cameras associated with the system, wherein determining the keyframe of highest correspondence comprises determining feature correspondences between the one or more visual tracking images and anchor points associated with each keyframe of the subset of keyframes.