Patent Description:
The present disclosure relates generally to motion tracking, and more particularly, to a method, system, and computer program product for motion tracking in connection with augmented reality systems, e.g., including a head mounted tracking device and a foot mounted tracking device.

Recently there has been an explosion of interest in augmented reality (AR) well beyond the research community where the field was forged in the early years of the International Symposium on Mixed and Augmented Reality (ISMAR) conference and its prequels. The popular press has adopted the vision of the pioneering researchers, in which AR will become an indispensable tool to augment human performance by providing enhanced situational awareness and visual guidance to complete tasks quickly and accurately without advance training.

For the past several years it seemed that the early focus on head mounted display (HMD)-based AR had largely given way to tablet and telephone AR because the devices became widely available to consumers, and advertisers saw the novelty of simple video AR as a way to reach them. Wearable AR systems leave the user's hands free and are able to provide an always-on information display that is ready to provide augmentations quickly when they are needed.

This renewed interest in HMDs still faces challenges, including the need for optical technologies to produce small comfortable HMDs with sufficient field of view (FOV), and head-tracking that can produce convincing spatio-temporal registration of augmentations to their corresponding physical objects in unprepared real-world environments. Additional details may be found in <NPL>; <NPL>; <NPL>. ;<NPL>; and <NPL>.

The ability to operate without markers has been demonstrated in many indoor and outdoor environments at impressive scale, and for video-see-through AR (such as tablets and telephones) vision-based techniques also produce rock-solid registration with no noticeable swim or mis-registration. However optical see-through registration is a much harder problem because the view of the physical world cannot be delayed to match the view of virtual augmentations, and the alignment is not able to be simply matched up in a video image, which puts a much greater demand on absolute <NUM>-DOF pose accuracy and relative calibration accuracy of the tracker to the display.

Thus, there remains an unmet need for high rate, low latency head tracking for AR systems that works robustly in both indoor and outdoor environments without the need for installing any external equipment or markers in the environment. In addition, there is an unmet need for AR systems that are able to function in the absence of global positioning system (GPS) data with intermittent visual features and in the presence of magnetic interference.

The following documents disclose technological background: <NPL>; <NPL>.

In light of the above described problems and unmet needs, aspects of the design, development and testing of an augmented reality (AR) system are presented herein. These aspects may be used, e.g., for aerospace and ground vehicles in order to meet stringent accuracy and robustness requirements. A system is presented that is able to acquire and maintain yaw alignment in the real world: outdoor environment, indoor environment, in an urban environment, in buildings, in caves, etc. Additional aspects include system architecture, motion-tracking algorithms, and harmonization techniques for implementing a precision AR system for someone on the ground (e.g., a pedestrian).

Aspects of the disclosure are set out in the appended independent claims.

Additional advantages and novel features of these aspects will be set forth in part in the description that follows, and in part will become more apparent to those skilled in the art upon examination of the following or upon learning by practice of the disclosure.

Various example aspects of the systems, methods, and computer-readable media will be described in detail, with reference to the following figures, wherein:.

Several aspects of vision-inertial pedestrian tracking features will now be presented with reference to various systems, apparatuses, and methods. These systems, apparatuses, and methods will be described in the following detailed description and illustrated in the accompanying drawings by various blocks, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as "elements"). Whether such elements are implemented as hardware or software depends upon the particular application and design constraints imposed on the overall implementation.

Software shall be construed broadly to include instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

If implemented in software, the functions may be stored on or encoded as one or more instructions or code on a computer-readable medium or media. Storage media may be any available media that is able to be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to carry or store desired program code in the form of instructions or data structures and that may be accessed by a computer. Disk and disc, as used herein, includes CD, laser disc, optical disc, digital versatile disc (DVD), and floppy disk, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

This application contains various features that relate to <CIT>, the entire contents of which are incorporated herein by reference.

Aspects of the system and method presented herein may be compatible with various mission computers that may be used in ground military training and/or missions, for example. In order to be compatible with, e.g., different mission computers (MC) that may already be present in an HMD, an HMD in accordance with aspects of the present disclosure may implement the head-tracking, rendering and/or display functions that are common to all ground AR systems, and none of the mission-specific functions, such as targeting, cueing, enhanced vision, and/or synthetic vision. As such a system may be responsible for rendering, but may not decide what is to be rendered, the interface to the MC may allow the MC to define and download an arbitrary set of "symbols," including any 2D or 3D shapes involving line segments of any color or thickness and/or bitmaps. Each symbol may be specified by the MC to be ground-stabilized or head-stabilized.

<FIG> is an overall system diagram of an example tracking system <NUM> (e.g., an AR vision-inertial pedestrian tracking system) for use in accordance with aspects of the present disclosure. The example system of <FIG> includes, for example, a HMD <NUM> with a head mounted camera (illustrated in <FIG>) and a display <NUM>. A control unit <NUM> and image generator (illustrated in <FIG>) may be coupled to one another via an interface box <NUM> and one or more cables and/or wires <NUM>. The control unit <NUM> and image generator may receive tracking data (e.g., from the camera) relating at least one of a head position or a head orientation of the pedestrian. The image generator may generate an image that is displayed on display <NUM>. In an aspect, the camera may include a natural feature tracker (e.g., hybrid optical-based inertial tracker) running a contemporaneous localization and mapping (SLAM) algorithm, such as ORB-SLAM (ORB indicates "Oriented FAST and Rotated BRIEF" features), while searching for known landmarks to provide absolute map registration at the controller.

The tracking system <NUM> may also include a foot mounted tracking device <NUM> positioned, for example, in a shoe <NUM> of the pedestrian. The foot mounted tracking device <NUM> may include a foot mounted inertial measurement unit that tracks the foot position and/or the foot orientation in GPS denied environments to within, e.g., a certain percentage of total distance travelled. For example, the foot mounted tracking device <NUM> may track the foot position and/or orientation to within <NUM>% of the total distance traveled, which is <NUM> of error after walking/running <NUM>.

In addition, the tracking system <NUM> may include a communication device <NUM>, such as a smart telephone that is in communication with the HMD <NUM> and the foot mounted tracking device <NUM>. Moreover, the HMD <NUM> and the foot mounted tracking device <NUM> may be in direct communication with each other.

A Kalman Filtering (KF) component and/or global pose adjustment component may be used by at least one of the controller <NUM> of the HMD <NUM> and/or the foot mounted tracking device <NUM> in accordance with aspects presented herein. For example, the KF component of the controller <NUM> may be used to perform visual-inertial odometry (VIO) and/or determine foot pose (e.g., using a zero-velocity updating (ZUPT) algorithm). The global pose adjustment component may be used to align information related to VIO, the foot position, and/or recognized landmarks.

In addition, the tracking system <NUM> may use a rolling-shutter image sensor in the HMD <NUM>, and thus may have to process each individual fiducial measurement acquired by the HMD <NUM> separately at a different point in time, using a nonlinear bearings measurement model, which operates as a function of position as well as orientation. By using a global shutter imager and the much faster processing element in the controller <NUM> (e.g., Acorn Reduced Instruction Set Computing Machine (ARM) Cortex A8 at <NUM>), the HMD <NUM> and/or the foot mounted tracking device <NUM> presented herein may be able to contemporaneously capture and decode up to or more than <NUM> ORB features at frame rate. For every frame captured by the camera, the controller <NUM> may solve for pose using a modified version of the OpenCV pose recovery algorithm, which results in a measurement of the head position, the head orientation, the foot position, and/or the foot orientation that may be used to correct drift. Therefore, the head orientation may be tracked independently from position using camera pose measurements and a very simple <NUM>-state Complementary Kalman Filter (CKF) to estimate the position/orientation errors and biases.

Still referring to <FIG>, the control unit <NUM> may determine a first heading and/or position uncertainty associated with at least one of the head position or the head orientation of the pedestrian based on the at least one of the head position or the head orientation. For example, the control unit <NUM> may track the at least one of the head position or the head orientation of the pedestrian by attempting to recognize one or more visual landmarks captured by the camera and may determine the first heading and/or position uncertainty based on the tracking. In an additional aspect, the control unit <NUM> may provide map registration based on the one or more visual landmarks being recognized. Moreover, the control unit <NUM> may determine a first heading and/or position uncertainty associated with the at least one of the head position or the head orientation of the pedestrian. For example, the first heading and/or position uncertainty determined by the control unit <NUM> may be related to one or more of first yaw information associated with at least one of the head position or the head orientation, first roll information related to the associated with at least one of the head position or the head orientation, first pitch information associated with at least one of the head position or the head orientation, first up and/or down information associated with at least one of the head position or the head orientation, first forward and/or back information associated with at least one of the head position or the head orientation, and/or first left and/or right information associated with at least one of the head position or the head orientation.

Referring still to <FIG>, the foot mounted tracking device <NUM> may track at least one of a foot position or a foot orientation of the pedestrian. For example, the at least one of the foot position or the foot orientation of the pedestrian may be tracked in a GPS denied environment. In an aspect, the foot mounted tracking device <NUM> may determine a second heading and/or position uncertainty associated with the at least one of the foot position or the foot orientation of the pedestrian based on the tracking. For example, the second heading and/or position uncertainty determined by the foot mounted tracking device <NUM> may be related to second yaw information associated with at least one of the foot position or the foot orientation, second roll information associated with at least one of the foot position or the foot orientation, second pitch information associated with at least one of the foot position or the foot orientation, second up and/or down information associated with at least one of the foot position or the foot orientation, second forward and/or back information associated with at least one of the foot position or the foot orientation, and/or second left and/or right information associated with at least one of the foot position or the foot orientation.

Referring again to <FIG>, the tracking system <NUM> may determine which of the first heading and/or position uncertainty or the second heading and/or position uncertainty is smaller. For example, GPS may only be able to provide information related to absolute position, visual landmarks captured by the camera may be able to provide information related to absolute heading and/or position, the Visual-Inertial Odometry Component of the Kalman Filter may provide information related to heading and/or position which may accumulate error over time, the foot mounted tracking device <NUM> may provide information related to relative heading and/or position information and accumulate error over time. The tracking system <NUM> may be able to determine information related to absolute position and/or orientation by combining one or more of the information and/or accumulated error from one or more of the GPS, the VIO component of the Kalman Filter, and/or the camera, the foot mounted sensor component <NUM>.

In a first example embodiment, the control unit <NUM> and the foot mounted tracking device <NUM> may transmit information associated with respective heading and/or position uncertainties to a communication device <NUM>, such as a smart telephone carried by the pedestrian. In an aspect, the control unit <NUM> may transmit information related to the first heading and/or position uncertainty to the communication device <NUM> using one or more of global navigation satellite system (GNSS) signaling, Bluetooth® short-range wireless communication signaling, Wi-Fi communication signaling, long term evolution (LTE) wireless communication signaling, and/or radio-frequency identification (RFID) communication signaling, among other modes of communication. Similarly, the foot mounted tracking device <NUM> may transmit information related to the second heading and/or position information to the communication device <NUM> using one or more of GNSS signaling, Bluetooth® short-range wireless communication signaling, Wi-Fi communication signaling, LTE wireless communication signaling, and/or RFID communication signaling, among others. The communication device <NUM> may determine which of the first heading and/or position uncertainty or the second heading and/or position uncertainty is smaller. If the first heading and/or position uncertainty is determined to be smaller than the second heading and/or position uncertainty, the communication device <NUM> may transmit the information related to the first heading and/or position uncertainty to the foot mounted tracking device <NUM>. Alternatively, when the first heading and/or position uncertainty is smaller than the second heading uncertainty, the communication device <NUM> may transmit signaling to the control unit <NUM> that indicates that information related to the first heading and/or position uncertainty should be sent from the control unit <NUM> to the foot mounted tracking device <NUM>. In either case, the foot mounted tracking device <NUM> may use the first heading and/or position uncertainty to correct the at least one of the foot position or the foot orientation of the pedestrian rather than by using the second heading and/or position uncertainty.

Moreover, if the second heading and/or position uncertainty is determined to be smaller, the communication device <NUM> may transmit information related to the second heading and/or position uncertainty to the control unit <NUM>. Alternatively, when the second heading and/or position uncertainty is smaller than the first heading and/or position uncertainty, the communication device <NUM> may send signaling to the foot mounted tracking device <NUM> that indicates that information related to the second heading and/or position uncertainty should be sent from the foot mounted tracking device <NUM> to the control unit <NUM>. In either case, the control unit <NUM> may use the second heading and/or position uncertainty to correct the at least one of the head position or the head orientation, rather than by using the first heading and/or position uncertainty.

In a second example embodiment, the control unit <NUM> and the foot mounted tracking device <NUM> may transmit the information associated with their respective heading and/or position uncertainties to each other without the need for communicating with the communication device <NUM>. For example, a transfer and/or exchange of information associated with heading and/or position uncertainty between the HMD <NUM> and the foot mounted tracking device <NUM> may occur automatically when the HMD <NUM> recognizes at least a portion of the shoe <NUM>. In an aspect, the control unit <NUM> of the HMB <NUM> may transmit (e.g., automatically) information related to the first heading and/or position uncertainty to the foot mounted tracking device <NUM> using one or more of GNSS signaling, Bluetooth® short-range wireless communication signaling, Wi-Fi communication signaling, LTE wireless communication signaling, and/or RFID communication signaling, among others. Similarly, the foot mounted tracking device <NUM> may transmit (e.g., automatically) information related to the second heading and/or position information to the control unit <NUM> of the HMD <NUM> using one or more of GNSS signaling, Bluetooth® short-range wireless communication signaling, Wi-Fi communication signaling, LTE wireless communication signaling, and/or RFID communication signaling, among others. A determination with respect to which of the first heading and/or position uncertainty or the second heading and/or position uncertainty is smaller may be made by one or more of the control unit <NUM> and/or the foot mounted tracking device <NUM>. When it is determined that the first heading and/or position uncertainty is smaller than the second heading and/or position uncertainty, the foot mounted tracking device <NUM> may use the first heading and/or position uncertainty to correct the at least one of the foot position or the foot orientation of the pedestrian. Alternatively, when it is determined that the second heading and/or position uncertainty is smaller than the first heading and/or position uncertainty, the control unit <NUM> may use the second heading and/or position uncertainty to correct the at least one of the head position or the head orientation.

In this way, the system of the present disclosure is able to provide a high rate, low latency head tracking that works robustly in both indoor and outdoor environments without the need for installing any external equipment or markers in the environment. In addition, the system of the present disclosure is able to provide position and orientation track of a pedestrian in the absence of GPS data, with intermittent visual features and in the presence of magnetic interference.

<FIG> illustrates a system diagram of various features of an example HMD system <NUM> for use in accordance with aspects of the present disclosure. For example, the HMD system may be the HDM <NUM> illustrated in <FIG>.

In one aspect, the HDM system <NUM> illustrated in <FIG> may include a camera <NUM> (e.g., natural feature tracking device, a global shutter camera, and/or rolling shutter camera) that operates a mapping (e.g., a contemporaneous localization and mapping (SLAM)) algorithm, such as an ORB-SLAM (e.g., ORB indicates "Oriented FAST and Rotated BRIEF" features) while searching for known landmarks to provide absolute map registration. In one aspect, the camera <NUM> may obtain monochrome images from an image sensor such as digital night vision goggles (NVGs). Additionally or alternatively, the camera <NUM> may include a zoom with a range of <NUM> or more. The camera <NUM> may be coupled to a control unit <NUM> and a battery unit <NUM> via an interface box <NUM>. For example, the camera <NUM> may be connected to the interface box via cable <NUM>. Signals associated with images captured by the camera <NUM> (e.g., <NUM> images) may be transmitted to the interface box <NUM> that then sends the signals to the controller <NUM> via, for example, cable and/or wire <NUM> for processing. For example, controller <NUM> may process, filter, and/or alter signals received from the camera <NUM> and overlay a visual landmark (e.g., building, vehicle, hill, tree) with a symbol and/or fiducial that is projected by an image generator <NUM> onto a visor, goggle, and/or lens (not illustrated in <FIG>) through which the pedestrian is looking. For example, the image generator <NUM> may be connected to the interface box via cable and/or wire <NUM>. Additionally, the controller <NUM> may determine a first heading and/or position uncertainty associated with a head position and/or head orientation of the pedestrian, and correct the first heading and/or position uncertainty with information related to a second heading and/or position uncertainty related to a foot position and/or foot orientation, similar to as described with respect to <FIG>. The information related to the second heading and/or position uncertainty may be received, for example, from the communication device <NUM> and/or foot mounted tracking devices 304a, 304b illustrated in <FIG>.

At times the sun may be directly in the field of vision (FOV) of the camera <NUM>, which may create dynamic range problems. In order to address the potential challenges caused by sunlight, the exposure may be lowered when the sunlight is too bright, and increased when it is too dark.

Still referring to <FIG>, the controller <NUM> may include a system on module (SOM), such as an embedded computer built on a single circuit board. The SOM may include a microprocessor with RAM, input/output controllers and all other features needed for signal processing. In addition, the controller <NUM> may include a Bluetooth® wireless transmitter/receiver, a Wi-Fi transmitter/receiver, an LTE transmitter/receiver, and/or an RFID transmitter/receiver for communication with external devices, such as a smart phone or other communication device <NUM> and/or the foot mounted tracking devices 304a, 304b illustrated in <FIG>. In an aspect, the controller <NUM> may transfer information associated with the first heading and/or position uncertainty to the communication device <NUM> and/or foot mounted tracking devices 304a, 304b illustrated in <FIG>. Moreover, the controller <NUM> may receive information related to the second heading and/or position uncertainty related to a foot position and/or foot orientation from the communication device <NUM> and/or foot mounted tracking devices 304a, 304b. Additionally, the controller <NUM> may include controls and/or buttons that may used to adjust the features of the HMD system <NUM>. For example, the controls and/or buttons may enable an on/off function of the HMD system <NUM> and/or change the settings used by the camera and/or the image generator <NUM>.

In an example embodiment, the image generator <NUM> may receive signals from the controller <NUM> related to images with or without fiducial markings related to the images captured by the camera <NUM>. The signals may include correction based on the second heading and/or position uncertainty when appropriate. The image generator <NUM> may be coupled to the interface box <NUM> via, for example, cable and/or wire <NUM> (e.g., low voltage differential signaling (LVDS) cable and/or a high definition multimedia interface (HDMI) cable). The image generator <NUM> is able to then project an image based on the signal received from the controller <NUM> onto a visor, goggle, or lens (not illustrated in <FIG>).

A battery <NUM> (e.g., a Lithium (Li)-ion battery) may also be coupled to the interface box <NUM> via, for example, cable and/or wire <NUM>, and used to power one or more of the controller <NUM>, the camera <NUM>, and/or the image generator <NUM>.

<FIG> illustrates a system diagram of an example foot tracking system <NUM> for use in accordance with aspects of the present disclosure. For example, the foot tracking system <NUM> may include the shoe <NUM> with the foot mounted tracking device <NUM> illustrated in <FIG>.

In one aspect, the foot tracking system <NUM> illustrated in <FIG> may include a pair of shoes 302a, 302b each including a foot mounted tracking device 304a, 304b embedded therein. The foot mounted tracking devices 304a, 304b may include a controller (not depicted) that is able to perform zero velocity Kalman Filter updates and learn the foot shape as a fiducial. Over time, when there are few or no visual landmarks, the HMD system <NUM> will drift in heading and/or position more rapidly. The foot tracking system <NUM> may then become the system with the least amount of heading and/or position uncertainty, and the controller of the foot mounted tracking device 304a, 304b is able to send information related to the second heading and/or position uncertainty to one or more of the communication device <NUM> and/or the controller <NUM> of the HMD system <NUM> illustrated in <FIG>. The HMD system <NUM> may use the second heading and/or position uncertainty to correct the head position and/or the head orientation. Similarly, the foot tracking system <NUM> is able to receive information related to the first heading and/or position uncertainty from the HMD system <NUM> and use this information to correct for the foot position and/or foot orientation.

<FIG> is flowchart <NUM> of a method of tracking a position and orientation of a pedestrian. The method may be performed by a tracking system (e.g., tracking system <NUM> illustrated in <FIG>). It should be understood that the operations indicated with dashed lines represent operations for various aspects of the disclosure.

In block <NUM>, the tracking system is able to track, using a head mounted tracking device, at least one of a head position or a head orientation of the pedestrian. For example, referring to <FIG>, the control unit <NUM> and image generator may receive tracking data (e.g., from the camera) relating at least one of a head position or a head orientation of the pedestrian. The image generator may generate an image that is displayed on display <NUM>. In an aspect, the camera may include a natural feature tracker (e.g., hybrid optical-based inertial tracker) running a contemporaneous localization and mapping (SLAM) algorithm, such as ORB-SLAM (ORB indicates "Oriented FAST and Rotated BRIEF" features) while searching for known landmarks to provide absolute map registration at the controller.

In block <NUM>, the tracking system is able to track, using a foot mounted tracking device, at least one of a foot position or a foot orientation of the pedestrian. For example, referring to <FIG>, the tracking system <NUM> may also include a foot mounted tracking device <NUM> positioned in a shoe <NUM> of the pedestrian. The foot mounted tracking device <NUM> may include a foot mounted inertial measurement unit that tracks the foot position and/or the foot orientation in GPS denied environments to within, e.g., a certain percentage of total distance travelled. For example, the foot mounted tracking device <NUM> may track the foot position and/or orientation to within <NUM>% of the total distance traveled, which is <NUM> of error after walking/running <NUM>.

In block <NUM>, the tracking system is able to determine a first heading and/or position uncertainty associated with the at least one of the head position or the head orientation of the pedestrian. For example, referring to <FIG>, the control unit <NUM> may determine a first heading and/or position uncertainty associated with at least one of the head position or the head orientation of the pedestrian based on the at least one of the head position or the head orientation. For example, the control unit <NUM> may track the at least one of the head position or the head orientation of the pedestrian by attempting to recognize one or more visual landmarks captured by the camera and may determine the first heading and/or position uncertainty based on the tracking.

In block <NUM>, the tracking system is able to determine a second heading and/or position uncertainty associated with the at least one of the foot position or the foot orientation of the pedestrian. For example, referring to <FIG>, the foot mounted tracking device <NUM> may track at least one of a foot position or a foot orientation of the pedestrian. For example, the at least one of the foot position or the foot orientation of the pedestrian may be tracked in a GPS denied environment. In an aspect, the foot mounted tracking device <NUM> may determine a second heading and/or position uncertainty associated with the at least one of the foot position or the foot orientation of the pedestrian based on the tracking.

At block <NUM>, the tracking system is able to determine which of the first heading uncertainty or the second heading uncertainty is smaller. For example, referring to <FIG>, the control unit <NUM> and the foot mounted tracking device <NUM> may send information associated with their respective heading uncertainties to a communication device <NUM> carried by the pedestrian. The communication device <NUM> may determine which of the first heading uncertainty or the second heading and/or position uncertainty is smaller. Alternatively, the control unit <NUM> and the foot mounted tracking device <NUM> may send the information associated with their respective heading and/or position uncertainties to each other without the need for communicating with the communication device <NUM>. In this case, a determination with respect to which of the first heading and/or position uncertainty or the second heading and/or position uncertainty is smaller may be made by one or more of the control unit <NUM> and/or the foot mounted tracking device <NUM>.

In block <NUM>, the tracking device is able to transfer the first heading and/or position uncertainty to the foot mounted tracking device when it is determined that the first heading and/or position uncertainty is smaller. For example, referring to <FIG>, if the first heading and/or position uncertainty is determined to be smaller than the second heading and/or position uncertainty, the communication device <NUM> may transmit the information related to the first heading and/or position uncertainty to the foot mounted tracking device <NUM>. Alternatively, when the first heading and/or position uncertainty is smaller than the second heading and/or position uncertainty, the communication device <NUM> may transmit signaling to the control unit <NUM> that indicates that information related to the first heading and/or position uncertainty should be sent from the control unit <NUM> to the foot mounted tracking device <NUM>. Additionally or alternatively, the control unit <NUM> and the foot mounted tracking device <NUM> may transmit the information associated with their respective heading and/or position uncertainties to each other without the need for communicating with the communication device <NUM>.

In block <NUM>, the tracking system is able to transfer the second heading and/or position uncertainty to the head mounted tracking device when it is determined that the second heading and/or position uncertainty is smaller. For example, referring to <FIG>, if the second heading and/or position uncertainty is determined to be smaller, the communication device <NUM> may transmit information related to the second heading and/or position uncertainty to the control unit <NUM>. Alternatively, when the second heading and/or position uncertainty is smaller than the first heading and/or position uncertainty, the communication device <NUM> may send signaling to the foot mounted tracking device <NUM> that indicates that information related to the second heading and/or position uncertainty should be transmitted from the foot mounted tracking device <NUM> to the control unit <NUM>. Additionally or alternatively, the control unit <NUM> and the foot mounted tracking device <NUM> may transmit the information associated with their respective heading and/or position uncertainties to each other without the need for communicating with the communication device <NUM>.

In block <NUM>, the tracking system is able to correct the at least one of the foot position or the foot orientation of the pedestrian using the first heading and/or position uncertainty transferred to the foot mounted tracking device. For example, referring to <FIG>, the foot mounted tracking device <NUM> may use the first heading and/or position uncertainty to correct the at least one of the foot position or the foot orientation of the pedestrian rather than by using the second heading and/or position uncertainty when the first heading and/or position uncertainty is determined to be smaller than the second heading and/or position uncertainty.

As seen in <FIG>, in block <NUM>, the tracking system is able to correct the at least one of the head position or the head orientation of the pedestrian using the second heading and/or position uncertainty transferred to the head mounted tracking device. For example, referring to <FIG>, the control unit <NUM> may use the second heading and/or position uncertainty to correct the at least one of the head position or the head orientation rather than by using the first heading and/or position uncertainty when the second heading and/or position uncertainty is determined to be smaller than the first heading and/or position uncertainty.

In block <NUM>, the tracking system is able to provide map registration based on the one or more visual landmarks being recognized. For example, referring to <FIG>, the control unit <NUM> may provide map registration based on the one or more visual landmarks being recognized by the camera.

In this way, the tracking system of the present disclosure is able to provide a high rate, low latency head tracking that works robustly in both indoor and outdoor environments without the need for installing any external equipment or markers in the environment. In addition, the tracking system of the present disclosure is able to provide position and orientation tracking of a pedestrian in the absence of GPS data with intermittent visual features and in the presence of magnetic interference.

<FIG> is a representative data flow diagram <NUM> illustrating the data flow between different features/components in an example system. The system may be a tracking system, such as tracking system <NUM> illustrated in <FIG>. The apparatus includes a head mounted sensor component <NUM> that tracks at least one of a head position or a head orientation of the pedestrian and searches for known landmarks, a foot mounted sensor component <NUM> that tracks at least one of a foot position or a foot orientation of the pedestrian, a Kalman Filtering and global pose alignment component <NUM> that processes each individual optical feature measurement acquired by the HMD <NUM> separately at a different point in time, using a nonlinear bearings measurement model, which operates as a function of position, as well as orientation, a GPS component <NUM> that tracks a position of the pedestrian, a SLAM component <NUM> provides absolute map registration while the head mounted sensor component <NUM> searches for known landmarks, a landmark recognition component <NUM> that recognizes known landmarks based on the searches performed by the head mounted sensor component <NUM>, a cloud based network <NUM> with which the tracking system is in communication, a content component <NUM> that organizes data obtained by components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, such that the signals and data may be used in an predefined or dynamic manner by the tracking system, a graphics generator component <NUM> that generates an image based on the data received from content component <NUM>, and an AR display component <NUM> that displays a visual landmark (e.g., building, vehicle, hill, tree, etc.) with a symbol and/or fiducial that is generated by graphics generator component <NUM>.

The apparatus may include additional components that perform each of the functions in the blocks of the algorithm in the aforementioned flowchart of <FIG> and <FIG>. As such, each block in the aforementioned flowchart of <FIG> and <FIG> may be performed by a component, and the apparatus may include one or more of those components. The components may include one or more hardware components specifically configured to carry out the stated processes/algorithm, implemented by a processor, for example, configured to perform the stated processes/algorithm, stored within a computer-readable medium for implementation by a processor, or some combination thereof.

<FIG> is a representative flow diagram illustrating data flow between different aspects/components and a Kalman Filtering / Global Pose Alignment Component used in an example system. The system may be a tracking system, such as tracking system <NUM> illustrated in <FIG>, and the Kalman Filtering / Global Pose Alignment component <NUM> may be the Kalman Filtering and Global Pose Alignment Component <NUM> illustrated in <FIG>.

For example, the Kalman Filtering and Global Pose Alignment Component <NUM> may include a VIO Kalman Filter Component <NUM>, a Foot Tracking Kalman Filter Component <NUM>, and a Global Coordinate Frame Alignment Component <NUM>.

In an example embodiment, the VIO Kalman Filter Component <NUM> may receive a SLAM Component Input <NUM> and Head Mounted Sensor Component Input <NUM>. In an aspect, the SLAM Component Input <NUM> may be received from the SLAM Component <NUM> and the Head Mounted Sensor Component Input <NUM> may be received from the Head Mounted Sensor Component <NUM> illustrated in <FIG>. In an aspect, the VIO Kalman Filter Component <NUM> may apply a VIO algorithm to one or more of the received SLAM Component Input <NUM> and/or Head Mounted Sensor Component Input <NUM> to produce a signal <NUM> that is sent to the Global Coordinate Frame Alignment Component <NUM>.

In another example embodiment, the Foot Tracking Kalman Filter Component <NUM> may receive a GPS Input <NUM> and a Foot Mounted Sensor Component Input <NUM>. In an aspect, the GPS Input <NUM> may be received from the GPS <NUM> and the Foot Mounted Sensor Component Input <NUM> may be received from the Foot Mounted Sensor Component <NUM> illustrated in <FIG>. In an aspect, the VIO Kalman Filter Component <NUM> may apply a VIO algorithm to one or more of the received GPS Input <NUM> and/or Foot Mounted Sensor Component Input <NUM> to produce a signal <NUM> that is sent to the Global Coordinate Frame Alignment Component <NUM>.

Using the received signals <NUM>, <NUM>, the Global Coordinate Frame Alignment Component <NUM> may reduce and/or correct for heading and/or position uncertainty. An Inertial Measurement Unit (IMU) Feedback Signal <NUM> that includes a reduction and/or correction in heading and/or position uncertainty may be sent from the Kalman Filtering and Global Pose Alignment component <NUM> to the SLAM Component <NUM> illustrated in <FIG>. In addition, a global degree of freedom (GDOF) signal <NUM> that also includes a reduction and/or correction in heading and/or position uncertainty may be sent to Graphics Generator Component <NUM>. Additionally and/or alternatively, the Foot Tracking Kalman Filter <NUM> may send a Heading Re-alignment and/or Absolute Position Correction Signal <NUM> to the Foot Mounted Sensor Component <NUM> illustrated in <FIG>.

<FIG> presents an example system diagram of various hardware components and other features, for use in accordance with aspects presented herein. The aspects may be implemented using hardware, software, or a combination thereof and may be implemented in one or more computer systems or other processing systems. In one example, the aspects may include one or more computer systems capable of carrying out the functionality described herein, e.g., in connection with <FIG> and <FIG>. An example of such a computer system <NUM> is shown in <FIG>.

Computer system <NUM> includes one or more processors, such as processor <NUM>. The processor <NUM> is connected to a communication infrastructure <NUM> (e.g., a communications bus, cross-over bar, or network). Various software aspects are described in terms of this example computer system. After reading this description, it will become apparent to a person skilled in the relevant art(s) how to implement the aspects presented herein using other computer systems and/or architectures.

Computer system <NUM> can include a display interface <NUM> that forwards graphics, text, and other data from the communication infrastructure <NUM> (or from a frame buffer not shown) for display on a display unit <NUM>. Computer system <NUM> also includes a main memory <NUM>, preferably random access memory (RAM), and may also include a secondary memory <NUM>. The secondary memory <NUM> may include, for example, a hard disk drive <NUM> and/or a removable storage drive <NUM>, representing a floppy disk drive, a magnetic tape drive, an optical disk drive, etc. The removable storage drive <NUM> reads from and/or writes to a removable storage unit <NUM> in a well-known manner. Removable storage unit <NUM>, represents a floppy disk, magnetic tape, optical disk, etc., which is read by and written to removable storage drive <NUM>. As will be appreciated, the removable storage unit <NUM> includes a computer usable storage medium having stored therein computer software and/or data.

In alternative aspects, secondary memory <NUM> may include other similar devices for allowing computer programs or other instructions to be loaded into computer system <NUM>. Such devices may include, for example, a removable storage unit <NUM> and an interface <NUM>. Examples of such may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an erasable programmable read only memory (EPROM), or programmable read only memory (PROM)) and associated socket, and other removable storage units <NUM> and interfaces <NUM>, which allow software and data to be transferred from the removable storage unit <NUM> to computer system <NUM>.

Computer system <NUM> may also include a communications interface <NUM>. Communications interface <NUM> allows software and data to be transferred between computer system <NUM> and external devices. Examples of communications interface <NUM> may include a modem, a network interface (such as an Ethernet card), a communications port, a Personal Computer Memory Card International Association (PCMCIA) slot and card, etc. Software and data transferred via communications interface <NUM> are in the form of signals <NUM>, which may be electronic, electromagnetic, optical or other signals capable of being received by communications interface <NUM>. These signals <NUM> are provided to communications interface <NUM> via a communications path (e.g., channel) <NUM>. This path <NUM> carries signals <NUM> and may be implemented using wire or cable, fiber optics, a telephone line, a cellular link, a radio frequency (RF) link and/or other communications channels. In this document, the terms "computer program medium" and "computer usable medium" are used to refer generally to media such as a removable storage drive <NUM>, a hard disk installed in hard disk drive <NUM>, and signals <NUM>. These computer program products provide software to the computer system <NUM>. Aspects presented herein may include such computer program products.

Computer programs (also referred to as computer control logic) are stored in main memory <NUM> and/or secondary memory <NUM>. Computer programs may also be received via communications interface <NUM>. Such computer programs, when executed, enable the computer system <NUM> to perform the features presented herein, as discussed herein. In particular, the computer programs, when executed, enable the processor <NUM> to perform the features presented herein. Accordingly, such computer programs represent controllers of the computer system <NUM>.

In aspects implemented using software, the software may be stored in a computer program product and loaded into computer system <NUM> using removable storage drive <NUM>, hard drive <NUM>, or communications interface <NUM>. The control logic (software), when executed by the processor <NUM>, causes the processor <NUM> to perform the functions as described herein. In another example, aspects may be implemented primarily in hardware using, for example, hardware components, such as application specific integrated circuits (ASICs). Implementation of the hardware state machine so as to perform the functions described herein will be apparent to persons skilled in the relevant art(s).

In yet another example, aspects presented herein may be implemented using a combination of both hardware and software.

<FIG> is a diagram illustrating an example of a hardware implementation for a system <NUM> employing a processing system <NUM>. The processing system <NUM> may be implemented with an architecture that links together various circuits including one or more processors and/or components, represented by the processor <NUM>, the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, and the computer-readable medium / memory <NUM>.

The processing system <NUM> may be coupled to a display <NUM>, such as display <NUM> in <FIG>. The processing system may also be coupled to various sensors, such as HMD <NUM>, foot mounted tracking unit <NUM>, camera <NUM>, image generator <NUM>, smart phone <NUM>, etc..

The processing system <NUM> includes a processor <NUM> coupled to a computer-readable medium / memory <NUM> via bus <NUM>. The processor <NUM> is responsible for general processing, including the execution of software stored on the computer-readable medium / memory <NUM>. The software, when executed by the processor <NUM>, causes the processing system <NUM> to perform the various functions described supra for any particular apparatus and/or system. The computer-readable medium / memory <NUM> may also be used for storing data that is manipulated by the processor <NUM> when executing software. The processing system further includes at least one of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. The components may be software components running in the processor <NUM>, resident/stored in the computer readable medium / memory <NUM>, one or more hardware components coupled to the processor <NUM>, or some combination thereof. The processing system <NUM> may be a component of an AR vision-inertial pedestrian tracking system, as illustrated in <FIG>.

The system <NUM> may further include features for tracking, using a head mounted tracking device, at least one of a head position or a head orientation of the pedestrian, features for tracking, using a foot mounted tracking device, at least one of a foot position or a foot orientation of the pedestrian, features for determining a first heading and/or position uncertainty associated with the at least one of the head position or the head orientation of the pedestrian, features for determining a second heading and/or position uncertainty associated with the at least one of the foot position or the foot orientation of the pedestrian, features for determining which of the first heading and/or position uncertainty or the second heading and/or position uncertainty is smaller, features for transferring the first heading and/or position uncertainty to the foot mounted tracking device when it is determined that the first heading and/or position uncertainty is smaller, or transferring the second heading and/or position uncertainty to the head mounted tracking device when it is determined that the second heading and/or position uncertainty is smaller, wherein the features for transferring is configured to transfer the first heading and/or position uncertainty to the foot mounted tracking device by automatically transferring the first heading and/or position uncertainty to the foot mounted tracking device when the head mounted tracking device recognizes at least a portion of the foot mounted tracking device, features for correcting the at least one of the foot position or the foot orientation of the pedestrian using the first heading and/or position uncertainty transferred to the foot mounted tracking device, wherein the features for transferring is configured to transfer the second heading and/or position uncertainty to the head mounted tracking device by automatically transferring the second heading and/or position uncertainty to the head mounted tracking device when the head mounted tracking device recognizes at least a portion of the foot mounted tracking device, features for correcting the at least one of the head position or the head orientation of the pedestrian using the second heading and/or position uncertainty transferred to the head mounted tracking device, wherein the features for tracking is configured to track the at least one of the head position or the head orientation of the pedestrian by attempting to recognize one or more visual landmarks, features for providing map registration based on the one or more visual landmarks being recognized, wherein the features for tracking are configured to track the at least one of the foot position or the foot orientation of the pedestrian in a GPS denied environment, wherein the first heading and/or position uncertainty is related first yaw information associated with the at least one of the head position or the head orientation of the pedestrian, and wherein the second heading and/or position uncertainty is related to second yaw information associated with the at least one of the foot position or the foot orientation of the pedestrian. The aforementioned features may be carried out via one or more of the aforementioned components of the system <NUM> and/or the processing system <NUM> of the system <NUM> configured to perform the functions recited by the aforementioned features.

Thus, aspects may include a system for tracking a position and orientation of a pedestrian, e.g., in connection with <FIG> and <FIG>.

The system may include additional components that perform each of the functions blocks of the algorithm in the aforementioned flowcharts of <FIG> and <FIG>. As such, each block in the aforementioned flowcharts of <FIG> and <FIG> may be performed by a component and the system may include one or more of those components. The components may include one or more hardware components specifically configured to carry out the stated processes/algorithm, implemented by a processor configured to perform the stated processes/algorithm, stored within a computer-readable medium for implementation by a processor, or some combination thereof.

Thus, aspects may include a non-transitory computer-readable medium for tracking a position and orientation of a pedestrian, the non-transitory computer-readable medium having control logic stored therein for causing a computer to perform the aspects described in connection with, e.g., <FIG> and <FIG>.

While the aspects described herein have been described in conjunction with the example aspects outlined above, various alternatives, modifications, variations, improvements, and/or substantial equivalents, whether known or that are or may be presently unforeseen, may become apparent to those having at least ordinary skill in the art. Accordingly, the example aspects, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the scope of the disclosure.

Thus, the claims are not intended to be limited to the aspects shown herein, but is to be accorded the full scope consistent with the claim language. No claim element is to be construed as a means plus function unless the element is expressly recited using the phrase "means for.

Claim 1:
A method (<NUM>) for tracking a position and orientation of a pedestrian, the method comprising:
tracking (<NUM>), using a head mounted tracking device, at least one of a head position or a head orientation of the pedestrian;
tracking (<NUM>), using a foot mounted tracking device, at least one of a foot position or a foot orientation of the pedestrian;
determining (<NUM>) a first heading or position uncertainty associated with the at least one of the head position or the head orientation of the pedestrian;
determining (<NUM>) a second heading or position uncertainty associated with the at least one of the foot position or the foot orientation of the pedestrian;
determining (<NUM>) which of the first heading or position uncertainty or the second heading or position uncertainty is smaller; and
transferring (<NUM>) the first heading or position uncertainty to the foot mounted tracking device and correcting (<NUM>) the tracked foot position or the tracked foot orientation of the pedestrian using the first heading or position uncertainty when it is determined that the first heading or position uncertainty is smaller, and transferring (<NUM>) the second heading or position uncertainty to the head mounted tracking device and correcting (<NUM>) the tracked head position or the tracked head orientation of the pedestrian using the second heading or position uncertainty when it is determined that the second heading or position uncertainty is smaller.