Patent Description:
Augmented reality (AR) is rapidly changing the way people interact with computer systems and the environment. This technology is expected to broadly impact aerospace and defense. Crew members (e.g., of commercial aircraft, military aircraft, ships, and ground vehicles) typically maintain situational awareness across two distinct contexts: information presented primarily on fixed, two-dimensional computer displays and the three-dimensional external environment. Crew members transition between these two contexts by redirecting their gaze or by physically moving between a console and a window. A mental transition also occurs, e.g., as a crew member attempts to map between two-dimensional graphics and three-dimensional terrain. Size and weight constraints can limit a number of fixed displays, so some crew members may not have access to information as and where needed.

<CIT>, in accordance with its abstract, states, "Method and apparatus are disclosed for stabilizing in-vehicle projected reality. An example in-vehicle projected reality system includes headset for displaying a projected reality image. The system also includes a localizer device for determining a headset pose in the vehicle. The system also includes a computing device for receiving the headset pose, receiving vehicle inertial data, determining a stabilized image based on the received headset pose and vehicle inertial data, and providing the stabilized image to the headset".

<CIT> further states, in accordance with claim <NUM> as originally filed, "A projected reality system comprising: a sensor for collecting inertial data of a vehicle; a localizer device for determining a headset pose in the vehicle, the localizer device being positioned such that inertial forces of the localizer device correspond with the inertial data; a computing device for determining, based on the headset pose and the inertial data, a stabilized image that synchronizes with the inertial forces; and a headset for displaying the stabilized image". This document further states, in accordance with claim <NUM> as originally filed, "The projected reality system of claim <NUM>, wherein the headset pose includes a location and orientation of the headset".

<CIT> further states, in accordance with claim <NUM> as originally filed, "A method comprising: detecting inertial data of a vehicle via a sensor; detecting a headset pose of a headset for projected reality within the vehicle via a localizer device that is positioned such that inertial forces of the localizer device correspond with the inertial data; determining, via a processor, a stabilized image that synchronizes with the inertial forces based on the headset pose and inertial data; and presenting the stabilized image via the headset".

A device for augmented reality visualization includes an interface and one or more processors. The interface is configured to receive vehicle sensor data from one or more vehicle sensors coupled to a first vehicle. The interface is also configured to receive headset sensor data from one or more headset sensors coupled to an augmented reality headset. The one or more processors are configured to determine, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the first vehicle. The one or more processors are also configured to estimate a gaze target of a user of the augmented reality headset based on the headset sensor data and the orientation and the location of the augmented reality headset relative to the first vehicle. The one or more processors are further configured to generate visualization data based on the gaze target. Responsive to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the first vehicle. The first point of interest includes at least a portion of a particular route of a particular vehicle. The particular vehicle includes the first vehicle or a second vehicle. Responsive to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the first vehicle. The one or more processors are also configured to send the visualization data to a display of the augmented reality headset.

A method of augmented reality visualization includes receiving, at a device, vehicle sensor data from one or more vehicle sensors coupled to a first vehicle. The method also includes receiving, at the device, headset sensor data from one or more headset sensors coupled to an augmented reality headset. The method further includes determining, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the first vehicle. The method also includes estimating, at the device, a gaze target of a user of the augmented reality headset based on the headset sensor data and the orientation and the location of the augmented reality headset relative to the first vehicle. The method further includes generating, at the device, visualization data based on the gaze target. Responsive to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the first vehicle. The first point of interest includes at least a portion of a particular route of a particular vehicle. The particular vehicle includes the first vehicle or a second vehicle. Responsive to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the first vehicle. The method also includes sending the visualization data from the device to a display of the augmented reality headset.

A computer-readable storage device stores instructions that, when executed by one or more processors, cause the one or more processors to receive vehicle sensor data from one or more vehicle sensors coupled to a first vehicle. The instructions, when executed by the one or more processors, also cause the one or more processors to receive headset sensor data from one or more headset sensors coupled to an augmented reality headset. The instructions, when executed by the one or more processors, further cause the one or more processors to determine, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the first vehicle. The instructions, when executed by the one or more processors, also cause the one or more processors to estimate a gaze target of a user of the augmented reality headset based on the headset sensor data and the orientation and the location of the augmented reality headset relative to the first vehicle. The instructions, when executed by the one or more processors, further cause the one or more processors to generate visualization data based on the gaze target. Responsive to determining that the gaze target is inside the first vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the first vehicle. The first point of interest includes at least a portion of a particular route of a particular vehicle. The particular vehicle includes the first vehicle or a second vehicle. Responsive to determining that the gaze target is outside the first vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the first vehicle. The instructions, when executed by the one or more processors, also cause the one or more processors to send the visualization data to a display of the augmented reality headset.

A device for augmented reality visualization includes an interface and one or more processors. The interface is configured to receive vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The interface is also configured to receive headset sensor data from one or more headset sensors coupled to an augmented reality headset. The one or more processors are configured to determine, based on the vehicle sensor data, a movement of the vehicle. The one or more processors are also configured to determine, based on the headset sensor data, a movement of the augmented reality headset. The one or more processors are further configured to estimate, based on a comparison of the movement of the vehicle and the movement of the augmented reality headset, a user portion of the movement of the augmented reality headset caused by a movement of a head of a user of the augmented reality headset and not caused by the movement of the vehicle. The one or more processors are also configured to determine, based on the user portion of the movement of the augmented reality headset, an orientation and a location of the augmented reality headset relative to the vehicle. The one or more processors are further configured to estimate a gaze target of the user based on the headset sensor data and the orientation and the location of the augmented reality headset relative to the vehicle. The one or more processors are also configured to generate visualization data based on the gaze target. Responsive to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the vehicle. Responsive to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the vehicle. The one or more processors are further configured to send the visualization data to a display of the augmented reality headset.

A method of augmented reality visualization includes receiving, at a device, vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The method also includes receiving, at the device, headset sensor data from one or more headset sensors coupled to an augmented reality headset. The method further includes determining, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the vehicle. The method also includes estimating, based at least in part on the orientation and the location of the augmented reality headset relative to the vehicle, a gaze target of a user of the augmented reality headset. The method further includes generating, at the device, visualization data based on the gaze target. Responsive to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the vehicle. Responsive to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the vehicle. The method also includes sending the visualization data from the device to a display of the augmented reality headset.

A computer-readable storage device stores instructions that, when executed by a processor, cause the processor to perform operations including receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle. The operations also include receiving headset sensor data from one or more headset sensors coupled to an augmented reality headset. The operations further include estimating, based on the vehicle sensor data and the headset sensor data, a gaze target of a user of the augmented reality headset. The operations also include generating visualization data based on the gaze target. Responsive to determining that the gaze target is inside the vehicle, the visualization data includes a first visual depiction of a first point of interest that is outside the vehicle. Responsive to determining that the gaze target is outside the vehicle, the visualization data includes a second visual depiction of a second point of interest that is inside the vehicle. The operations further include sending the visualization data to a display of the augmented reality headset.

The features, functions, and advantages described herein can be achieved independently in various examples or may be combined in yet other examples, further details of which can be found with reference to the following description and drawings.

Examples described herein are directed to augmented reality visualizations. In some examples, a user in a vehicle wears an augmented reality (AR) headset. A device receives headset sensor data from one or more headset sensors coupled to the AR headset. The device also receives vehicle sensor data from one or more vehicle sensors coupled to the vehicle. A user movement estimator of the device determines a user portion of a movement of the AR headset caused by a movement of a head of the user and not caused by a movement of the vehicle. In some examples, the user turns <NUM> degrees to look in a direction of an instrument panel in the vehicle and the vehicle turns left. In some of these examples, the vehicle sensor data indicates that the vehicle has turned <NUM> degrees. The headset sensor data indicates that the AR headset has turned <NUM> degrees (<NUM> degrees due to the movement of the vehicle + <NUM> degrees due to user movement). The user movement estimator determines a user portion of the movement of the AR headset based on a comparison of the movement of the AR headset (e.g., <NUM> degrees) and the movement of the vehicle (e.g., <NUM> degrees). For example, the user movement estimator determines that the user portion indicates a <NUM> degree movement relative to the vehicle based on determining that a net of the movement of the AR headset and the movement of the vehicle is <NUM> degrees. In other examples, the user movement estimator determines that the user portion indicates no movement based on determining that the movement of the AR headset matches the movement of the vehicle.

A gaze target estimator estimates a gaze target of the user based on the user portion of the movement of the AR headset. The gaze target indicates where the user is looking at a particular time. In some examples, the gaze target estimator determines, based on the user portion of the movement of the AR headset, an orientation and location of the AR headset relative to the vehicle. The gaze target estimator determines, based on the orientation and location of the AR headset, that the user is looking in the direction of an instrument panel of the vehicle. In some examples, the headset sensor data includes image sensor data. In such examples, the gaze target estimator may perform image recognition on the image sensor data to determine that the gaze target includes a particular instrument of the instrument panel. In other examples, the gaze target can indicate that the user is looking in the direction of a location external to the vehicle (e.g., outside a window).

A visualization data generator generates visualization data for the AR headset based on the gaze target and a context at the particular time. For example, the context indicates whether the user is in the vehicle or outside the vehicle. In some examples, the context indicates a role of the user. The visualization data may differ based on context for the same gaze target. For example, the visualization data may include an indicator identifying a ship for a user looking at the ship from inside an airplane, and may include a visual depiction of a wind speed meter for a user (e.g., a paratrooper) looking at the ship from outside the airplane. In other examples, the visualization data may include a three-dimensional rotatable map with visual depictions of objects when the user gaze target is inside the vehicle and may include indicators marking locations of the objects when the user gaze target is outside the vehicle. In some examples, the visualization data may include an indicator marking a location of an object outside the vehicle although the object is not visible (e.g., behind cloud cover or another object) to the user. In some examples, the visualization data indicates at least a portion of a route of a vehicle. For example, the visualization data may include a three-dimensional rotatable map with visual depictions of routes of one or more vehicles when the user gaze target is inside the vehicle. In other examples, the visualization data may include indicators marking routes of the one or more vehicles when the user gaze target is outside the vehicle.

In some examples, the visualization data reduces the mental workload of the user when visually moving from one context to another. For example, the visualization data includes a visual depiction of an object (e.g., a ship, a portion of a route of the ship, or both) in a first context (e.g., a map displayed inside the vehicle) and includes an indicator to mark a location of the object (e.g., the ship, the portion of the route of the ship, or both) in a second context (e.g., outside the vehicle). The indicator includes at least one visual element, such as a color, an image, a video, an animation, a symbol, text, a pattern, or a combination thereof, in common with the visual depiction. Having similar visual elements for the same object helps the user detect the corresponding object in different contexts.

The figures and the following description illustrate specific examples. It will be appreciated that those skilled in the art will be able to devise various arrangements that, although not explicitly described or shown herein, embody the principles described herein and are included within the scope of the claims that follow this description. Furthermore, any examples described herein are intended to aid in understanding the principles discussed herein and are to be construed as being without limitation. As a result, this application is not limited to the specific examples described below, but by the claims.

Particular examples are described herein with reference to the drawings. In the description, common features are designated by common reference numbers throughout the drawings. As used herein, various terminology is used for the purpose of describing particular examples only and is not intended to be limiting. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some examples and plural in other examples. To illustrate, <FIG> depicts a device <NUM> including one or more processors ("processor(s)" <NUM> in <FIG>), which indicates that in some examples the device <NUM> includes a single processor <NUM> and in other examples the device <NUM> includes multiple processors <NUM>. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects related to multiple of the features are being described.

The terms "comprise," "comprises," and "comprising" are used interchangeably with "include," "includes," or "including. " Additionally, the term "wherein" is used interchangeably with the term "where. " As used herein, "exemplary" indicates an example, and should not be construed as limiting or as indicating a preference or a preferred example. As used herein, an ordinal term (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term "set" refers to a grouping of one or more elements, and the term "plurality" refers to multiple elements.

As used herein, "generating", "calculating", "using", "selecting", "accessing", and "determining" are interchangeable unless context indicates otherwise. For example, "generating", "calculating", or "determining" a parameter (or a signal) can refer to actively generating, calculating, or determining the parameter (or the signal) or can refer to using, selecting, or accessing the parameter (or signal) that is already generated, such as by another component or device. As used herein, "coupled" can include "communicatively coupled," "electrically coupled," or "physically coupled," and can also (or alternatively) include any combinations thereof. Two devices (or components) can be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled can be included in the same device or in different devices and can be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some examples, two devices (or components) that are communicatively coupled, such as in electrical communication, can send and receive electrical signals (digital signals or analog signals) directly or indirectly, such as via one or more wires, buses, networks, etc. As used herein, "directly coupled" is used to describe two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

<FIG> is a block diagram of a system <NUM> that is operable to perform augmented reality visualization. The system <NUM> includes a device <NUM> coupled, via a network <NUM>, to an augmented reality (AR) headset <NUM> and to a vehicle <NUM>. The network <NUM> includes a wired network, a wireless network, or both. The AR headset <NUM> includes (or is coupled to) one or more headset sensors <NUM>. The headset sensor <NUM> includes an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, an image sensor, a global positioning system (GPS) receiver, a beacon, or a combination thereof. The headset sensor <NUM> is configured to generate headset sensor (HS) data <NUM>. The vehicle <NUM> includes (or is coupled to) one or more vehicle sensors <NUM>. The vehicle sensor <NUM> includes a radar receiver, a sonar hydrophone, a maritime surface search radar receiver, an acoustic sensor, a seismic sensor, a ground surveillance radar receiver, a passive sonar hydrophone, a passive radar receiver, a doppler radar receiver, an accelerometer, a gyroscope, a magnetometer, an inertial measurement unit, an image sensor, a GPS receiver, a beacon, or a combination thereof. The vehicle sensor <NUM> is configured to generate vehicle sensor (VS) data <NUM>.

It should be noted that in the following description, various functions performed by the system <NUM> of <FIG> are described as being performed by certain components or modules. However, this division of components and modules is for illustration only. In other examples, a function described herein as performed by a particular component or module is divided amongst multiple components or modules. Moreover, in other examples, two or more components or modules of <FIG> are integrated into a single component or module. Each component or module illustrated in <FIG> can be implemented using hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a controller, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

The device <NUM> includes a memory <NUM>, one or more processors <NUM>, and an interface <NUM>. The interface <NUM> includes a communication interface, a network interface, an application interface, or a combination thereof. The memory <NUM> is configured to store data <NUM> that is used (e.g., generated) by the processor <NUM>. In some examples, a portion of the data <NUM> is stored in the memory <NUM> at any given time. The interface <NUM> is configured to communicate with the network <NUM>. The processor <NUM> includes a user movement estimator <NUM>, a gaze target estimator <NUM>, a route estimator <NUM>, a context estimator <NUM>, a visualization data generator <NUM>, or a combination thereof. The user movement estimator <NUM> is configured to estimate a headset movement <NUM> of the AR headset <NUM> based on the HS data <NUM> and to estimate a vehicle movement <NUM> of the vehicle <NUM> based on the VS data <NUM>. The user movement estimator <NUM> is configured to estimate a user movement <NUM> based on a comparison of the headset movement <NUM> and the vehicle movement <NUM>. The user movement <NUM> corresponds to a user portion of the headset movement <NUM> caused by a movement of a head of a user <NUM> of the AR headset <NUM> and not caused by the vehicle movement <NUM>.

The gaze target estimator <NUM> is configured to estimate a gaze target <NUM> (e.g., a locus of a gaze) of the user <NUM> based on the user movement <NUM>. In some examples, the gaze target estimator <NUM> is configured to determine, based on the user movement <NUM>, a headset orientation <NUM>, a headset location <NUM>, or both, of the AR headset <NUM> relative to the vehicle <NUM>. To illustrate, the user movement <NUM> indicates that the AR headset <NUM> moved by a particular angle (e.g., <NUM> degrees), a particular distance, or both, relative to the vehicle <NUM>. The headset orientation <NUM> indicates that the user <NUM> is looking in a particular direction relative to (e.g., <NUM> degrees from the front of) the vehicle <NUM>. The headset location <NUM> indicates a location of the AR headset <NUM> relative to (e.g., in) the vehicle <NUM>.

In some examples, the gaze target estimator <NUM> is configured to determine the gaze target <NUM> based on the headset orientation <NUM>, the headset location <NUM>, or both. For example, the gaze target estimator <NUM> determines that the gaze target <NUM> includes a particular instrument panel of the vehicle <NUM>. In other examples, the gaze target estimator <NUM> determines that the gaze target <NUM> includes a window of the vehicle <NUM>.

In some examples, the gaze target estimator <NUM> is configured to determine (e.g., refine) the gaze target <NUM> by performing image recognition on the HS data <NUM>. For example, the gaze target estimator <NUM> is configured to determine, based on the image recognition, that the gaze target <NUM> includes a particular instrument of the instrument panel. In other examples, the gaze target estimator <NUM> is configured to determine, based on the image recognition, that the gaze target <NUM> includes a particular object outside the vehicle <NUM> that is visible through the window.

The route estimator <NUM> is configured to determine one or more routes <NUM> of the vehicle <NUM>, one or more routes of one or more vehicles <NUM>, or a combination thereof. For example, the routes <NUM> include one or more planned routes <NUM>, one or more traversed routes <NUM>, one or more interpolated routes <NUM>, one or more forecasted routes <NUM>, one or more recommended routes <NUM>, or a combination thereof. For example, a vehicle is intended to travel a planned route <NUM> from an origin to a destination. The vehicle is detected as having traveled a traversed route <NUM> from a first location to a second location. In some examples, the first location includes the origin and the second location is between the origin and the destination. In other examples, in situations where the vehicle has been re-routed from the planned route <NUM> for various reasons, such as traffic, weather, obstacles, etc., the second location is not between the origin and the destination. In some examples, the VS data <NUM> (e.g., GPS data, radar data, sonar data, or a combination thereof) indicates that the vehicle is detected at an estimated location. An interpolated route <NUM> is determined for the vehicle from the second location to the estimated location. A forecasted route <NUM> of the vehicle indicates an anticipated route of the vehicle from the estimated location to a forecasted location on the planned route <NUM>. A recommended route <NUM> of the vehicle satisfies a route goal <NUM>. The route goal <NUM> includes obstacle avoidance, collision avoidance, bad weather avoidance, an area avoidance, a time goal, a fuel consumption goal, a cost goal, or a combination thereof.

The context estimator <NUM> is configured to determine a context <NUM>. For example, the context <NUM> includes a user role <NUM> of the user <NUM>, a user location context <NUM> (e.g., whether the user <NUM> is inside or outside the vehicle <NUM>), a gaze target context <NUM> (e.g., a location of the object that the user is looking at), or a combination thereof. The visualization data generator <NUM> is configured to generate visualization data <NUM> based on the gaze target <NUM> (e.g., what the user <NUM> is looking at), the context <NUM>, the route <NUM>, or a combination thereof. For example, the visualization data <NUM> includes one or more visual elements <NUM> (e.g., virtual elements) that are selected based on the gaze target <NUM>, the context <NUM>, the route <NUM>, or a combination thereof. The visualization data generator <NUM> is configured to send the visualization data <NUM> to a display of the AR headset <NUM>.

During operation, the user <NUM> activates (e.g., powers up) the AR headset <NUM>. The headset sensor <NUM> generates the HS data <NUM>. For example, the headset sensor <NUM> generates the HS data <NUM> during a first time range. To illustrate, one or more sensors of the headset sensor <NUM> generate data continuously, at various time intervals, responsive to detecting an event, or a combination thereof, during the first time range. An event can include receiving a request from the device <NUM>, receiving a user input, detecting a movement of the AR headset <NUM>, or a combination thereof.

The device <NUM> receives the HS data <NUM> from the AR headset <NUM>. For example, the AR headset <NUM> sends the HS data <NUM> to the device <NUM> continuously, at various time intervals, responsive to detecting an event, or a combination thereof. An event can include receiving a request from the device <NUM>, receiving a user input, detecting a movement of the AR headset <NUM>, detecting an update of the HS data <NUM>, or a combination thereof.

The vehicle sensor <NUM> generates the VS data <NUM>. For example, the vehicle sensor <NUM> generates the VS data <NUM> during a second time range. In some examples, the second time range is the same as, overlaps, or is within a threshold duration (e.g., <NUM> minutes) of the first time range. To illustrate, one or more sensors of the vehicle sensor <NUM> generate data continuously, at various time intervals, responsive to detecting an event, or a combination thereof, during the second time range. An event can include receiving a request from the device <NUM>, receiving a user input, detecting a movement of the vehicle <NUM>, or a combination thereof.

The device <NUM> receives the VS data <NUM> from the vehicle <NUM>. For example, the vehicle <NUM> sends the VS data <NUM> to the device <NUM> continuously, at various time intervals, responsive to detecting an event, or a combination thereof. An event can include receiving a request from the device <NUM>, receiving a user input, detecting a movement of the vehicle <NUM>, detecting an update of the VS data <NUM>, or a combination thereof.

In some examples, the device <NUM> receives, via the interface <NUM>, planned route data <NUM>, location data <NUM>, or a combination thereof, from one or more data sources <NUM>, the vehicle <NUM>, the vehicle <NUM>, or a combination thereof. The one or more data sources <NUM>, the vehicle <NUM>, the vehicle <NUM>, or a combination thereof, send the planned route data <NUM>, the location data <NUM>, or a combination thereof, to the device <NUM> continuously, at various time intervals, responsive to detecting an event, or a combination thereof. An event can include receiving a request from the device <NUM>, receiving a user input, or a combination thereof. In some examples, the one or more data sources <NUM> include a shore station, a satellite, an aircraft communications addressing and reporting system (ACARS), a blue force tracker (BFT), or a combination thereof.

In some examples, the device <NUM> receives planned route data 191A of the vehicle <NUM> from the vehicle <NUM>, via user input (e.g., from an operator of the vehicle <NUM>), or both. The planned route data 191A indicates a planned route 180A of the vehicle <NUM> from a first origin at a first time (e.g., expected time of departure) to a first destination at a second time (e.g., expected time of arrival). For example, the planned route 180A indicates that the vehicle <NUM> is expected to be at the first origin at the first time, a second location at a particular time, one or more additional locations at one or more additional times, the first destination at the second time, or a combination thereof. In some examples, the device <NUM> receives the planned route data 191A prior to the vehicle <NUM> leaving the first origin.

In some examples, the device <NUM> receives planned route data 191B via the network <NUM> from a data source 192A. The planned route data 191B indicates a planned route 180B of the vehicle <NUM> from a second origin at a first time (e.g., expected time of departure) to a second destination at a second time (e.g., expected time of arrival). For example, the planned route 180B indicates that the vehicle <NUM> is expected to be at the second origin at the first time, a second location at a particular time, one or more additional locations at one or more additional times, the second destination at the second time, or a combination thereof. The first origin may be the same as or different from the second origin or the second destination. The first destination may be the same as or different from the second origin or the second destination.

In some examples, the route estimator <NUM> sends a planned route data request to the data source 192A and receives the planned route data 191B responsive to the planned route data request. In some examples, the planned route data request indicates the planned route 180A of the vehicle <NUM>, and the data source 192A sends planned route data <NUM> of the one or more vehicles <NUM> that are expected to be within a threshold distance of the vehicle <NUM> along the planned route 180A. For example, the data source 192A determines, based on a comparison of the planned route 180A and the planned route 180B, that a vehicle 106A is expected to be within a threshold distance of the vehicle <NUM>. The data source 192A sends the planned route data 191B to the device <NUM> in response to determining that the vehicle 106A is expected to be within the threshold distance of the vehicle <NUM>. In some examples, the planned route data request indicates one or more regions associated with (e.g., passed through) the planned route 180A, and the data source 192A sends the planned route 180B in response to determining that the planned route 180B is associated with (e.g., passes through) at least one of the one or more regions.

In some examples, the device <NUM> receives location data 193A of the vehicle <NUM> from the vehicle <NUM>. The location data 193A indicates a particular location of the vehicle <NUM> detected at a particular time. In some examples, the device <NUM> sends the planned route data request indicating the particular location and the particular time to the data source 192A, and the data source 192A sends planned route data <NUM> of the one or more vehicles <NUM> that are expected to be within a threshold distance of the particular location within a threshold time of the particular time.

In some examples, the route estimator <NUM> determines a traversed route 182A of the vehicle <NUM> from a first location to the particular location. For example, the route estimator <NUM> at various time intervals receives the location data 193A from the vehicle <NUM>. To illustrate, the route estimator <NUM> receives location data 193A at a first time indicating that the vehicle <NUM> is detected at the first location, location data 193A at a second time indicating that the vehicle <NUM> is detected at a second location, location data 193A at one or more additional times indicating that the vehicle <NUM> is detected at one or more additional locations, location data 193A at the particular time indicating that the vehicle <NUM> is detected (e.g., most recently detected) at the particular location, or a combination thereof. The route estimator <NUM> determines the traversed route 182A of the vehicle <NUM> from the first location, via the second location, the one or more additional locations, or a combination thereof, to the particular location. In some examples, the traversed route 182A indicates points (and corresponding times) along a route traversed by the vehicle <NUM> from the first location to the particular location. In some examples, the traversed route 182A includes straight route segments between pairs of detected locations of the vehicle <NUM>. In some examples, the traversed route <NUM> includes a curvilinear route that consists of a series of straight route segments that appear curvilinear at a particular scale. In some examples, the first location includes the origin and the particular location is between the origin and the destination.

In some examples, the device <NUM> receives location data <NUM> of the one or more vehicle <NUM> from a data source 192B. For example, the vehicle <NUM> at various time intervals (e.g., at <NUM>-minute intervals) transmits vehicle location information of the vehicle <NUM> and the data source 192B stores the vehicle location information. To illustrate, the vehicle location information is received by a shore station that forwards the vehicle location information via a satellite to the data source 192B.

In some examples, the route estimator <NUM> determines one or more traversed routes <NUM> of the one or more vehicles <NUM> based on the location data <NUM>. For example, location data 193B indicates that a vehicle <NUM> is detected at a first location at a first time, one or more locations at one or more additional times, a particular location (e.g., a most recently detected location), at a particular time, or a combination thereof. The route estimator <NUM> determines a traversed route 182B of the vehicle <NUM> from the first location, via one or more additional locations, to the particular location. In some examples, the traversed route 182B indicates points (and corresponding times) along a route traversed by the vehicle <NUM> from the first location to the particular location. In some examples, the traversed route 182B includes straight route segments, curvilinear route segments, or a combination thereof, between pairs of detected locations of the vehicle <NUM>.

In some examples, the route estimator <NUM> determines one or more interpolated routes <NUM> of the one or more vehicles <NUM>. For example, there may be a delay between the vehicle <NUM> transmitting the vehicle location information indicating the particular location (e.g., most recently detected location) of the vehicle <NUM> and the device <NUM> receiving the vehicle location information. The vehicle <NUM> may have moved from the particular location during the delay. For example, the VS data <NUM> (e.g., radar data, sonar data, or both) indicates an estimated location of the vehicle <NUM>. In some examples, the route estimator <NUM> determines an interpolated route <NUM> of the vehicle <NUM> in response to determining that a difference between the estimated location indicated by the VS data <NUM> and the particular location (e.g., most recently detected location) indicated by the location data <NUM> is greater than a threshold distance. In some examples, determining the interpolated route <NUM> of the vehicle <NUM> includes performing an interpolation based on the particular location (e.g., most recently detected location) and the estimated location. In some examples, the interpolated route <NUM> corresponds to a shortest traversable path, a fastest traversable path, or both, between the particular location and the estimated location. In some examples, the interpolated route <NUM> corresponds to a straight-line between the particular location and the estimated location. In other examples, the interpolated route <NUM> corresponds to a non-linear path between the particular location and the estimated location. To illustrate, the interpolated route <NUM> follows a curved path (e.g., a road or a river) that the vehicle <NUM> is traveling, avoids an obstacle (e.g., goes around an island, a mountain, or a building), follows legally accessible paths (e.g., avoids restricted areas, one way streets in the wrong direction, etc.), or a combination thereof.

In some examples, the particular location indicated by the location data 193A of the vehicle <NUM> is the same as the estimated location indicated by the VS data <NUM> of the vehicle <NUM>. For example, the device <NUM> receives the location data 193A, the VS data <NUM>, or a combination thereof, from the vehicle <NUM>. The location data 193A, the VS data <NUM>, or a combination thereof, indicate the same vehicle location information (e.g., GPS coordinates) of the vehicle <NUM>.

In some examples, the route estimator <NUM> determines one or more forecasted routes <NUM> of the vehicle <NUM>, one or more forecasted routes <NUM> of the one or more vehicles <NUM>, or a combination thereof. For example, the route estimator <NUM> determines a forecasted route <NUM> from an estimated location to a forecasted location on a planned route. The forecasted route <NUM> is based on an estimated location (e.g., indicated by the VS data <NUM>), a vehicle state (e.g., indicated by the VS data <NUM>), a vehicle capability (e.g., indicated by vehicle capability data), external conditions (e.g., indicated by external conditions data), or a combination thereof. To illustrate, the route estimator <NUM> determines a forecasted route 186A of the vehicle <NUM> from an estimated location to a forecasted location on the planned route 180A of the vehicle <NUM>. The forecasted route 186A is based on the estimated location of the vehicle <NUM>, a vehicle state (e.g., a direction, a speed, acceleration, or a combination thereof) of the vehicle <NUM>, a vehicle capability (e.g., average speed, maximum speed, turn radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration) of the vehicle <NUM>, external conditions (e.g., weather conditions, wind speed, wind currents, water currents, obstacles, restricted areas), or a combination thereof. In some examples, the route estimator <NUM> determines the forecasted route 186A that the vehicle <NUM> is likely to take from the estimated location to get back to the planned route 180A given the vehicle state, the vehicle capability, the external conditions, or a combination thereof. In some examples, the route estimator <NUM> receives the vehicle capability data, the external conditions data, or a combination thereof, from the one or more data sources <NUM>, the vehicle <NUM>, or a combination thereof.

In some examples, the route estimator <NUM> determines a forecasted route 186B of a vehicle <NUM> from an estimated location to a forecasted location on a planned route 180B of the vehicle <NUM>. The forecasted route 186B is based on the estimated location of the vehicle <NUM> (e.g., indicated by the VS data <NUM>), a vehicle state (e.g., indicated by the VS data <NUM>, such as a direction, a speed, acceleration, or a combination thereof) of the vehicle <NUM>, a vehicle capability (e.g., indicated by vehicle capability data, such as average speed, maximum speed, turn radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration) of the vehicle <NUM>, the external conditions (e.g., indicated by the external conditions data, such as weather conditions, wind speed, wind currents, water currents, obstacles, restricted areas), or a combination thereof. In some examples, the route estimator <NUM> determines the forecasted route 186B that the vehicle <NUM> is likely to take from the estimated location to get back to the planned route 180B given the vehicle state, the vehicle capability, the external conditions, or a combination thereof. In some examples, the route estimator <NUM> receives the vehicle capability data, the external conditions data, or a combination thereof, from the one or more data sources <NUM>, the vehicle <NUM>, the vehicle <NUM>, or a combination thereof.

In some examples, the route estimator <NUM> determines one or more recommended routes <NUM> of the vehicle <NUM>. For example, the route estimator <NUM> determines a recommended route <NUM> from an estimated location to a recommended location on a planned route. The recommended route <NUM> satisfies a route goal <NUM>. For example, the route goal <NUM> includes obstacle avoidance, collision avoidance, bad weather avoidance, an area avoidance, a time goal, a fuel consumption goal, a cost goal, or a combination thereof. In some examples, the route goal <NUM> is based on user input, default data, a configuration setting, or a combination thereof. In some examples, the recommended route <NUM> is based on an estimated location (e.g., indicated by the VS data <NUM>) of the vehicle <NUM>, the vehicle state (e.g., indicated by the VS data <NUM>) of the vehicle <NUM>, a vehicle capability (e.g., indicated by the vehicle capability data) of the vehicle <NUM>, the external conditions (e.g., indicated by the external conditions data), the one or more forecasted routes <NUM>, the route goal <NUM>, or a combination thereof. To illustrate, the route estimator <NUM> determines a recommended route 188A of the vehicle <NUM> from an estimated location to a recommended location on the planned route 180A of the vehicle <NUM>. The recommended route 186A is based on the estimated location of the vehicle <NUM>, the vehicle state (e.g., a direction, a speed, acceleration, or a combination thereof) of the vehicle <NUM>, a vehicle capability (e.g., average speed, maximum speed, turn radius, average acceleration, maximum acceleration, average deceleration, maximum deceleration) of the vehicle <NUM>, the external conditions (e.g., weather conditions, wind speed, wind currents, water currents, obstacles, restricted areas), the one or more forecasted routes <NUM> of the one or more vehicles <NUM>, the route goal <NUM>, or a combination thereof. In some examples, the route estimator <NUM> determines the recommended route 188A of the vehicle <NUM> from the estimated location to get to the planned route 180A and satisfy the route goal <NUM> given the vehicle state, the vehicle capability, the external conditions, the routes that the one or more vehicles <NUM> are likely to take, or a combination thereof.

In some examples, the route estimator <NUM>, in response to determining that the forecasted route <NUM> fails to satisfy a route goal <NUM>, generates a recommended route <NUM> that satisfies the route goal <NUM>. The recommended route <NUM> indicates a recommended heading, a recommended speed, a recommended altitude, or a combination thereof, which is different from a forecasted heading, a forecasted speed, a forecasted altitude, or a combination thereof, indicated by the forecasted route <NUM>.

In some examples, the route estimator <NUM> detects a predicted collision between the vehicle <NUM> and the vehicle <NUM> based on a comparison of the forecasted route 186A of the vehicle <NUM> and a forecasted route 186B of the vehicle <NUM>. To illustrate, the route estimator <NUM> detects the predicted collision in response to determining that a first forecasted location of the vehicle <NUM> at a first forecasted time indicated by the forecasted route 186A is within a threshold distance of a second forecasted location of the vehicle <NUM> at a second forecasted time indicated by the forecasted route 186B, and that the first forecasted time is within a threshold duration of the second forecasted time. The route estimator <NUM>, in response to determining that the route goal <NUM> includes collision avoidance and detecting the predicted collision, generates a recommended route 188A of the vehicle <NUM> that avoids the predicted collision, generates an alert <NUM> indicating the predicted collision and the recommended route 188A, or both. For example, the recommended route 188A includes a recommended speed, a recommended direction, a recommended altitude, or a combination thereof, that is different from a forecasted speed, a forecasted direction, a forecasted altitude, or a combination thereof, indicated by the forecasted route 186A, prior to a predicted location of the predicted collision.

In some examples, the route estimator <NUM> determines multiple recommended routes <NUM> of the vehicle <NUM>. For example, the route estimator <NUM> determines the recommended route 188A that satisfies a route goal 194A (e.g., collision avoidance), a recommended route 188B that satisfies a route goal 194B (e.g., a time goal), a recommended route 188C that satisfies a route goal 194C (e.g., collision avoidance and a cost goal), or a combination thereof. In some examples, the route estimator <NUM> stores data indicating the one or more routes <NUM>, the route goal <NUM>, the alert <NUM>, the planned route data <NUM>, the location data <NUM>, or a combination thereof, in the memory <NUM>.

The context estimator <NUM> retrieves the HS data <NUM> corresponding to the first time range (e.g., <NUM>:<NUM> AM - <NUM>:<NUM> AM) from the memory <NUM>. For example, the HS data <NUM> includes sensor data that is timestamped with a time (e.g., <NUM>:<NUM>:<NUM> AM) during the first time range. The context estimator <NUM> retrieves the VS data <NUM> corresponding to the second time range (e.g., <NUM>:<NUM> AM - <NUM>:<NUM> AM) from the memory <NUM>. For example, the VS data <NUM> includes sensor data that is timestamped with a time (e.g., <NUM>:<NUM>:<NUM> AM) during the second time range. The context estimator <NUM>, in response to determining that the first time range matches (e.g., overlaps) the second time range, determines that the HS data <NUM> corresponds to the VS data <NUM>. In some examples, the context estimator <NUM> determines that the first time range matches the second time range in response to determining that the first time range is the same as the second time range. In other examples, the context estimator <NUM> determines that the first time range matches the second time range in response to determining that the first time range overlaps the second time range. In some examples, the context estimator <NUM> determines that the first time range matches the second time range in response to determining that a duration between an end of one of the first time range or the second time range and a beginning of the other of the first time range or the second time range is within a threshold duration (e.g., <NUM> seconds or <NUM> minute).

The context estimator <NUM> determines a context <NUM> based on the HS data <NUM>, the VS data <NUM>, or both. For example, the context estimator <NUM>, in response to determining that the HS data <NUM> corresponds to the VS data <NUM>, determines the context <NUM> based on the HS data <NUM> and the VS data <NUM>. In some examples, the context estimator <NUM> determines a user location context <NUM> indicating whether or not the user <NUM> is in any vehicle. In some examples, the context estimator <NUM>, in response to determining that the VS data <NUM> indicates that a tag associated with the user <NUM> is detected in proximity (e.g., less than <NUM> inches or less than <NUM>) of a first sensor that is in the vehicle <NUM>, generates the user location context <NUM> indicating that the user <NUM> is in the vehicle <NUM>. Alternatively, the context estimator <NUM>, in response to determining that the VS data <NUM> indicates that a tag associated with the user <NUM> is detected in proximity (e.g., less than <NUM> inches or less than <NUM>) of a second sensor that is outside (e.g., on an external surface of) the vehicle <NUM>, generates the user location context <NUM> indicating that the user <NUM> is not in the vehicle <NUM>. In alternative examples, the context estimator <NUM> generates the user location context <NUM> indicating that the user <NUM> is in the vehicle <NUM> in response to determining that a first user location of the user <NUM> indicated by the HS data <NUM> is within a threshold distance (e.g., <NUM> inches or less than <NUM>) of a first vehicle location of the vehicle <NUM> indicated by the VS data <NUM>, that image recognition performed on image sensor data of the HS data <NUM> matches an interior of the vehicle <NUM>, or both.

In some examples, a user <NUM> is considered to be "in" the vehicle <NUM> when a movement of the vehicle <NUM> is likely to cause a movement of the AR headset <NUM> worn by the user <NUM> independently of any movement caused by the user <NUM>. In some examples, the user <NUM> is standing on the roof of the vehicle <NUM> and is considered to be "in" the vehicle <NUM>. In other examples, the user <NUM> is standing with feet on the ground outside the vehicle <NUM> and with a head through an open window looking inside the vehicle <NUM> and the user <NUM> is considered to not be "in" the vehicle <NUM>.

The context estimator <NUM> determines a user role <NUM> of the user <NUM>. In some examples, the user role <NUM> is relatively static and is indicated by context configuration data <NUM>. In some examples, the context configuration data <NUM> is based on default data, configuration data, user input, or a combination thereof. In alternative examples, the user role <NUM> is dynamic and is based at least in part on the user location context <NUM>. To illustrate, the context configuration data <NUM> indicates that the user <NUM> has a first role inside the vehicle <NUM> and a second role outside the vehicle <NUM>.

The user movement estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is in the vehicle <NUM>, determines a user movement <NUM> based on a comparison of the HS data <NUM> and the VS data <NUM>. In some examples, the user movement estimator <NUM> determines a headset movement <NUM> indicated by the HS data <NUM>. For example, the HS data <NUM> indicates that the AR headset <NUM> moved in a particular direction, a particular distance, a particular angle of rotation, or a combination thereof, relative to an environment (e.g., geographic coordinates). The headset movement <NUM> indicates the particular direction, the particular distance, the particular angle of rotation, or a combination thereof. In some examples, the particular angle of rotation indicates yaw, pitch, roll, or a combination thereof, of a head of the user <NUM> relative to the environment. The user movement estimator <NUM> determines a vehicle movement <NUM> indicated by the VS data <NUM>. For example, the VS data <NUM> indicates that the vehicle <NUM> moved in a particular direction, a particular distance, a particular angle of rotation, or a combination thereof, relative to an environment (e.g., geographic coordinates). In some examples, the particular angle of rotation indicates yaw, pitch, roll, or a combination thereof, of the vehicle <NUM> relative to the environment. The vehicle movement <NUM> indicates the particular direction, the particular distance, the particular angle of rotation, or a combination thereof.

The user movement estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is in the vehicle <NUM>, determines the user movement <NUM> based on a comparison of the headset movement <NUM> and the vehicle movement <NUM>. For example, the user movement estimator <NUM> determines the user movement <NUM> based on a net of the headset movement <NUM> and the vehicle movement <NUM>. In some examples, the headset movement <NUM> includes a first portion caused by a movement of the user <NUM> and a second portion caused by a movement of the vehicle <NUM>. For example, the user <NUM> wearing the AR headset <NUM> moves their head (e.g., turns <NUM> degrees to the left) and the vehicle <NUM> turns (e.g., <NUM> degrees to the left). The headset movement <NUM> indicates that the AR headset <NUM> moved (e.g., <NUM> degrees) relative to the environment. The user movement <NUM> (e.g., <NUM> degrees to the left) indicates the first portion (e.g., caused by movement of a head of the user <NUM> and not caused by the vehicle movement <NUM>).

In some examples, the user movement estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is not in the vehicle <NUM> (e.g., is not in any vehicle), determines the user movement <NUM> based on the HS data <NUM> and independently of the VS data <NUM>. For example, the user movement estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is not in the vehicle <NUM> (e.g., is not in any vehicle) designates the headset movement <NUM> as the user movement <NUM>.

The user movement estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is in the vehicle <NUM>, determines a headset orientation <NUM> and a headset location <NUM>, of the AR headset <NUM> relative to the vehicle <NUM>. For example, the user movement estimator <NUM> determines the headset orientation <NUM>, the headset location <NUM>, or both, based on the user movement <NUM>. In some examples, the user movement <NUM> indicates a particular angle of rotation (e.g., a net angle of rotation) relative to the vehicle <NUM> and the user movement estimator <NUM> determines the headset orientation <NUM> based on the particular angle of rotation. To illustrate, the user movement estimator <NUM>, in response to determining that the AR headset <NUM> previously had a first headset orientation relative to (e.g., the user <NUM> was looking towards a front of) the vehicle <NUM>, determines the headset orientation <NUM> (e.g., -<NUM> degrees or <NUM> degrees) by applying the particular angle of rotation (e.g., <NUM> degrees to the left) to the first headset orientation (e.g., <NUM> degrees). In some examples, the user movement <NUM> indicates a particular distance, a particular direction, or both, relative to the vehicle <NUM>, and the user movement estimator <NUM> determines the headset location <NUM> based on the particular distance, the particular direction, or both. To illustrate, the user movement estimator <NUM>, in response to determining that the AR headset <NUM> previously had a first headset location (e.g., first coordinates) relative to the vehicle <NUM>, determines the headset location <NUM> (e.g., second coordinates) relative to the vehicle <NUM> by applying the particular distance, the particular direction, or both (e.g., coordinates delta), to the first headset location (e.g., first coordinates). The user movement estimator <NUM> stores the headset location <NUM>, the headset orientation <NUM>, or both, in the memory <NUM>.

In some examples, the HS data <NUM> indicates that the user <NUM> is looking in a first global direction (e.g., West) and the VS data <NUM> indicates that the vehicle <NUM> is oriented in a second global direction (e.g., North). The user movement estimator <NUM> determines, based on a comparison of the first global direction and the second global direction, that the user <NUM> is looking in a particular user-vehicle direction relative to (e.g., towards the left side of) the vehicle <NUM>. The headset orientation <NUM> indicates the particular user-vehicle direction.

In some examples, the HS data <NUM> indicates that the user <NUM> is located at a first global location and the VS data <NUM> indicates that a center of the vehicle <NUM> is located at a second global location. The user movement estimator <NUM> determines, based on a comparison of the first global location and the second global location, that the user <NUM> is located at a particular user-vehicle location relative to the center of the vehicle <NUM>. The headset location <NUM> indicates the particular user-vehicle location.

The gaze target estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is not in the vehicle <NUM>, determines a gaze target <NUM> based on the HS data <NUM>, the user movement <NUM>, or both. In some examples, the HS data <NUM> indicates a global location (e.g., GPS coordinates) of the user <NUM>, a global direction (e.g., a compass direction, an elevation angle, or both) that the user <NUM> is looking at, or both. In other examples, the HS data <NUM> indicates the user movement <NUM>. In some of these examples, the gaze target estimator <NUM> determines the global location, the global direction, or both, by applying the user movement <NUM> to a previous global location, a previous global direction, or both, of the user <NUM>. In some examples, the gaze target estimator <NUM> identifies the gaze target <NUM> in response to determining that map data <NUM> indicates that the gaze target <NUM> is associated with the global location, the global direction, or both. In some examples, the gaze target estimator <NUM> performs image recognition by comparing image sensor data of the HS data <NUM> with images of objects associated with the global location, global direction, or both. In some of these examples, the gaze target estimator <NUM> identifies the gaze target <NUM> in response to determining, based on the image recognition, that an image of the gaze target <NUM> matches the image sensor data.

In some examples, the gaze target estimator <NUM>, in response to determining that the user location context <NUM> indicates that the user <NUM> is in the vehicle <NUM>, determines the gaze target <NUM> based on the HS data <NUM>, the headset location <NUM>, the headset orientation <NUM>, or a combination thereof. In some examples, the gaze target <NUM> indicates what (e.g., an object) the user <NUM> is looking at. The gaze target estimator <NUM> performs image recognition on image sensor data of the HS data <NUM> (e.g., captured by a camera of the headset sensor <NUM>) to identify a particular object. To illustrate, the gaze target estimator <NUM> determines vehicle map data <NUM> indicates that a gaze target area (e.g., a particular instrument panel) corresponds to the headset location <NUM>, the headset orientation <NUM>, or both. The vehicle map data <NUM> includes images of objects (e.g., particular instruments) located in the gaze target area (e.g., the particular instrument panel) in the vehicle <NUM>. For example, the vehicle map data <NUM> includes one or more first images of a particular object. The gaze target estimator <NUM> identifies the particular object as the gaze target <NUM> in response to determining that image recognition indicates that the image sensor data matches the first images. The gaze target estimator <NUM>, in response to determining that the vehicle map data <NUM> indicates that the particular object is in the vehicle <NUM>, generates a gaze target context <NUM> indicating that the gaze target <NUM> is in the vehicle <NUM>.

In some examples, the gaze target estimator <NUM> determines that the user <NUM> is looking at a particular object that is outside the vehicle <NUM>. To illustrate, that the gaze target estimator <NUM>, in response to determining that the vehicle map data <NUM> indicates that the headset location <NUM>, the headset orientation <NUM>, or both, correspond to an open or see-through portion (e.g., a window) of the vehicle <NUM>, that the user <NUM> is looking outside. In some examples, the gaze target estimator <NUM> identifies a particular object (that is outside the vehicle <NUM>) as the gaze target <NUM> by performing image recognition. In some examples, the gaze target estimator <NUM> identifies the particular object outside the vehicle <NUM> based on a global location (e.g., geographic coordinates) and a global direction (e.g., East) of the user <NUM>. For example, the gaze target estimator <NUM>, based on map data <NUM>, identifies a global area (e.g., a section on the left side of a particular road) based on the global location and the global direction. The gaze target estimator <NUM> designates the particular object as the gaze target <NUM> in response to determining, based on image recognition, that image sensor data of the HS data <NUM> matches images of the particular object. The gaze target estimator <NUM> generates the gaze target context <NUM> indicating that the gaze target <NUM> is outside the vehicle <NUM>. In some examples, the gaze target context <NUM> indicates a location (e.g., GPS coordinates) of the gaze target <NUM>.

The visualization data generator <NUM> generates visualization data <NUM> based on the HS data <NUM>, the gaze target <NUM>, the context <NUM>, the context configuration data <NUM>, the one or more routes <NUM>, the alert <NUM>, or a combination thereof. For example, the context configuration data <NUM> indicates data corresponding to the gaze target <NUM>, the context <NUM>, or both. In some examples, the context configuration data <NUM> indicates that first data (e.g., a three-dimensional map with locations of the one or more vehicles <NUM>, a location of the vehicle <NUM>, the routes <NUM>, or a combination thereof) corresponds to the user role <NUM> (e.g., a navigator), the user location context <NUM> (e.g., inside the vehicle <NUM>), the gaze target <NUM> (e.g., an interior portion of the vehicle <NUM>), the gaze target context <NUM> (e.g., inside the vehicle <NUM>), or a combination thereof. In some examples, a first portion of the first data is relatively static (e.g., locations of landmarks in the three-dimensional map) and a second portion of the first data is dynamic (e.g., locations and routes). For example, the second portion of the first data is received from one or more systems (e.g., a radar system, a communication system, or both) of the vehicle <NUM>, generated by the route estimator <NUM>, or a combination thereof.

The visualization data generator <NUM> generates the visualization data <NUM> to include one or more visual elements <NUM> based on the first data. For example, the visualization data generator <NUM>, in response to determining that the gaze target context <NUM> indicates that the gaze target <NUM> is inside the vehicle <NUM>, generates the visualization data <NUM> to include the visual elements <NUM> that are to be displayed to users in the vehicle <NUM> having the user role <NUM> (e.g., navigator) and looking in a direction of the particular object (e.g., the particular instrument) of the vehicle <NUM>.

In some examples, the visual elements <NUM> include a first visual depiction (e.g., a virtual representation) of a point of interest (POI) <NUM> (e.g., a vehicle <NUM>) that is located outside the vehicle <NUM>, a second visual indicator indicating a location of a POI <NUM> (e.g., the particular instrument) that is located inside the vehicle <NUM>, a visual depiction of the alert <NUM>, or a combination thereof. As used herein, "POI" refers to an object, a landmark, a location, a person, a vehicle, at least a portion of a route of a vehicle, data, or a combination thereof, that is to be indicated in the visualization data <NUM>. In some examples, the context configuration data <NUM> indicates the objects, the landmarks, the locations, the persons, the vehicles, the route portions, or a combination thereof, that are POIs.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual depiction <NUM> (e.g., a virtual representation) of the POI <NUM> (e.g., a vehicle <NUM>) that is outside the vehicle <NUM>, a visual indicator <NUM> that indicates a location of the POI <NUM> (e.g., a particular instrument) that is inside the vehicle <NUM>, or both. The visual depiction <NUM>, the visual indicator <NUM>, or both, include one or more visual elements. In some examples, a visual element includes a symbol, a shape, a fill, a label, an image, an animation, a video, text, a pattern, or a combination thereof. In some examples, a visual element indicates information (e.g., an identifier, specifications, remaining fuel, passenger information, cargo information, location, or a combination thereof) of a corresponding POI (e.g., the POI <NUM> or the POI <NUM>). In some examples, a visual element includes a selectable option associated with the corresponding POI (e.g., the POI <NUM> or the POI <NUM>). For example, the user <NUM> can select a selectable option associated with the POI <NUM>, e.g., to communicate with the vehicle <NUM>.

In some examples, the visual depiction <NUM> is generated for display inside the vehicle <NUM> of a virtual representation of the POI <NUM> (e.g., the vehicle <NUM>) that is outside the vehicle <NUM>. In some examples, the visual elements <NUM> include visual depictions of multiple POIs outside the vehicle <NUM> and the visual elements <NUM> are generated to display, inside the vehicle <NUM>, relative locations of the multiple POIs. For example, the visual depiction <NUM> of the POI <NUM> (e.g., the vehicle <NUM>) is generated to be displayed twice as far inside the vehicle <NUM> from a first visual depiction of a first POI (e.g., the vehicle <NUM>) than from a second visual depiction of a second POI (e.g., a hill) if the POI <NUM> is twice as far from the first POI than from the second POI outside the vehicle <NUM>.

In some examples, the visual indicator <NUM> is generated for display by the AR headset <NUM> to indicate a location of the POI <NUM> (e.g., the particular instrument) within the vehicle <NUM> when the POI <NUM> is in a field of view of the user <NUM> wearing the AR headset <NUM>. For example, the visualization data generator <NUM> determines, based on the vehicle map data <NUM>, the HS data <NUM>, or both, the location of the POI <NUM> inside the vehicle <NUM>. To illustrate, the visualization data generator <NUM> performs image recognition on image sensor data of the HS data <NUM> and determines a location of the POI <NUM> in a field of view of the user <NUM> of the AR headset <NUM>. The visualization data generator <NUM> generates the visual indicator <NUM> such that the visual indicator <NUM> indicates the location of the POI <NUM> in the field of view of the user <NUM>. For example, the visual indicator <NUM> is overlaid proximate to (e.g., within an inch of or within <NUM> of) the POI <NUM> in the field of view of the user <NUM>.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include one or more visual depictions of one or more POIs (e.g., one or more portions of one or more routes of the vehicle <NUM>) that are outside the vehicle <NUM>. For example, the visual elements <NUM> include a visual depiction <NUM> (e.g., a virtual representation) of a first POI (e.g., one or more portions of the planned route 180B), a visual depiction <NUM> of a second POI (e.g., one or more portions of the traversed route 182B), a visual depiction <NUM> of a third POI (e.g., one or more portions of the interpolated route 184B), a visual depiction <NUM> of a fourth POI (e.g., one or more portions of the forecasted route 186B), or a combination thereof.

The visual depictions <NUM>, <NUM>, <NUM>, and <NUM> include one or more visual elements. In some examples, a visual element includes a symbol, a shape, a fill, a label, an image, an animation, a video, text, or a combination thereof. In some examples, a visual element indicates information (e.g., an identifier, remaining fuel, a location, a time, or a combination thereof) of a corresponding POI (e.g., a portion of a route). For example, the visual element includes a time of a location (e.g., a planned location, a traversed location, an interpolated location, or a forecasted location) of the vehicle <NUM> along a corresponding route (e.g., the planned route 180B, the traversed route 182B, the interpolated route 184B, or the forecasted route 186B). In some examples, a visual element includes a selectable option associated with the corresponding POI (e.g., the portion of route). For example, the user <NUM> can select a selectable option associated with the indicated route, e.g., to communicate with the vehicle <NUM>. In some examples, the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> include at least one common element (e.g., the same arrowhead) to indicate that the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> are related (e.g., to the vehicle <NUM>). In some examples, the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> include at least one different element (e.g., the line dash type) to indicate that the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> correspond to portions of different types of routes (e.g., planned, traversed, interpolated, or forecasted).

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual depiction <NUM> (e.g., a virtual representation) of a POI (e.g., the vehicle <NUM>) and one or more visual depictions of one or more POIs (e.g., one or more portions of one or more routes of the vehicle <NUM>) that are outside the vehicle <NUM>. For example, the visual elements <NUM> include a visual depiction <NUM> (e.g., a virtual representation) of a first POI (e.g., one or more portions of the planned route 180A), a visual depiction <NUM> of a second POI (e.g., one or more portions of the traversed route 182A), a visual depiction <NUM> of a third POI (e.g., one or more portions of the forecasted route 186A), or a combination thereof.

The visual depictions <NUM>, <NUM>, <NUM>, and <NUM> include one or more visual elements. In some examples, a visual element indicates information (e.g., an identifier, remaining fuel, a location, a time, or a combination thereof) of a corresponding POI (e.g., the vehicle <NUM> or a portion of a route). In some examples, a visual element includes a selectable option associated with the corresponding POI (e.g., the portion of route, the vehicle <NUM>, or both). For example, the user <NUM> can select a selectable option associated with the forecasted route 186A (e.g., the indicated route) to set course of the vehicle <NUM> to the forecasted route 186A. In some examples, the visual depictions <NUM>, <NUM>, and <NUM> include at least one common element (e.g., the same arrowhead) to indicate that the visual depictions <NUM>, <NUM>, and <NUM> are related (e.g., to the vehicle <NUM>). In some examples, at least one common element of the visual depictions <NUM>, <NUM>, and <NUM> is different from at least one common element of the visual depictions <NUM>, <NUM>, <NUM>, and <NUM>. In some examples, the visual depiction <NUM> and the visual depiction <NUM> include at least a first common element (e.g., a first line dash type) to indicate a common route type (e.g., a planned route), the visual depiction <NUM> and the visual depiction <NUM> include at least a second common element (e.g., a second line dash type) to indicate a common route type (e.g., a traversed route), the visual depiction <NUM> and the visual depiction <NUM> include at least a third common element (e.g., a third line dash type) to indicate a common route type (e.g., a forecasted route), or a combination thereof.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual depiction <NUM> (e.g., a virtual representation) of a POI (e.g., one or more portions of the recommended route 188A), a visual depiction <NUM> of the alert <NUM>, or both. In some examples, the visual elements <NUM> include multiple depictions of multiple recommended routes.

The visual depiction <NUM> includes one or more visual elements. In some examples, a visual element includes a selectable option associated with the corresponding POI (e.g., the portion of route, the vehicle <NUM>, or both). For example, the user <NUM> can select a selectable option associated with the recommended route 188A (e.g., the indicated route) to set course of the vehicle <NUM> to the recommended route 188A. In some examples, the visual depiction <NUM> includes at least one common element (e.g., the same arrowhead) of the visual depictions <NUM>, <NUM>, and <NUM> to indicate that the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> are related (e.g., to the vehicle <NUM>). In some examples, at least one common element of the visual depictions <NUM>, <NUM>, <NUM>, and <NUM> is different from at least one common element of the visual depictions <NUM>, <NUM>, <NUM>, and <NUM>.

Returning to <FIG>, the visualization data generator <NUM>, in response to determining that the gaze target <NUM> is outside the vehicle <NUM>, generates the visualization data <NUM> to include the visual elements <NUM> that are to be displayed to users in the vehicle <NUM> having the user role <NUM> (e.g., navigator) and looking in a direction of the particular object (e.g., the vehicle <NUM>) outside the vehicle <NUM>. In some examples, the visual elements <NUM> include a first visual indicator indicating a location of the POI <NUM> (e.g., the vehicle <NUM>) that is located outside the vehicle <NUM>, a second visual depiction of the POI <NUM> (e.g., the particular instrument) that is located inside the vehicle <NUM>, a visual depiction of the alert <NUM>, or a combination thereof.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual depiction <NUM> of the POI <NUM> (e.g., the particular instrument) that is inside the vehicle <NUM>, a visual indicator <NUM> that indicates a location of the POI <NUM> (e.g., the vehicle <NUM>) that is outside the vehicle <NUM>, or both. The visual depiction <NUM>, the visual indicator <NUM>, or both, include one or more visual elements.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual indicator <NUM> that indicates a location of a first POI (e.g., one or more portions of the planned route 180B), a visual indicator <NUM> that indicates a location of a second POI (e.g., one or more portions of the traversed route 182B), a visual indicator <NUM> that indicates a location of a third POI (e.g., one or more portions of the interpolated route 184B), and a visual indicator <NUM> that indicates a location of a fourth POI (e.g., one or more portions of the forecasted route 186B). The visual indicator <NUM>, the visual indicator <NUM>, the visual indicator <NUM>, the visual indicator <NUM>, or a combination thereof, include one or more visual elements.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual indicator <NUM> that indicates a location of a first POI (e.g., one or more portions of the planned route 180A) and a visual indicator <NUM> that indicates a location of a second POI (e.g., one or more portions of the traversed route 182A). The visual indicator <NUM>, the visual indicator <NUM>, or both, include one or more visual elements. In some examples, the visual elements <NUM> include visual indicators corresponding to one or more portions of routes that are within a field of view of the user <NUM>. In some examples, including the example illustrated in <FIG>, portions of the planned route 180A and portions of the traversed route 182A are within a field of view of the user <NUM> and the forecasted route 186A is not within a field of view of the user <NUM>.

In some examples, including the example illustrated in <FIG>, the visual elements <NUM> include a visual depiction <NUM> of the alert <NUM>, a visual indicator <NUM> that indicates a location of a first POI (e.g., one or more portions of the forecasted route 186A) and a visual indicator <NUM> that indicates a location of a second POI (e.g., one or more portions of the recommended route 188A). The visual depiction <NUM>, visual indicator <NUM>, the visual indicator <NUM>, or a combination thereof, include one or more visual elements. In some examples, including the example illustrated in <FIG>, portions of the forecasted route 186A and portions of the recommended route 188A are within a field of view of the user <NUM>, and the planned route 180A and the traversed route 182A are not within a field of view of the user <NUM>. In some examples, the visual elements <NUM> include visual indicators of portions of multiple recommend routes <NUM> that are within a field of view of the user <NUM>.

In some examples, the visualization data generator <NUM> generates the visualization data <NUM> to reduce a mental workload of the user <NUM> in transitioning between an output of the AR headset <NUM> corresponding to the inside of the vehicle <NUM> and an output of the AR headset <NUM> corresponding to the outside of the vehicle <NUM>. For example, the visual depiction <NUM> includes at least one visual element in common with the visual indicator <NUM>. In some examples, including the example of <FIG>, the common visual elements include a shape and a fill (e.g., a diagonal fill). In other examples, the visual depiction <NUM> includes at least one visual element in common with the visual indicator <NUM>. In some examples, including the example of <FIG>, the common visual elements include a shape and text (e.g., Wind Speed). In some examples, the visual depiction <NUM> includes at least one visual element in common with the visual indicator <NUM>. In some examples, including the example of <FIG>, the common visual elements include an arrowhead and a dashed line type. The common visual elements reduce a mental workload of the user <NUM> in identifying visual depictions and corresponding visual indicators as the user <NUM> transitions between looking at a gaze target that is inside the vehicle <NUM> and a gaze target that is outside the vehicle <NUM>.

The visualization data generator <NUM> sends the visualization data <NUM> to a display of the AR headset <NUM>. The AR headset <NUM> generates, based on the visualization data <NUM>, an AR output indicating the visual elements <NUM>. In some examples, the AR output can be manipulated based on user input. For example, the user <NUM> can rotate, zoom into, or zoom out of a three-dimensional map. In some examples, the three-dimensional map includes visual depictions of moving objects in low-visibility terrain and the user <NUM> (e.g., a navigator) can use the three-dimensional map to account for the moving objects and plan a path for the vehicle <NUM> through the terrain.

The system <NUM> thus enables determining a user portion of a movement of an AR headset that is caused by a movement of a head of a user of the AR headset and not caused by a movement of a vehicle that the user is in. The system <NUM> also enables visualization data to include visual elements that are selected based on context. The visual elements can be displayed in a field of view of the user, enabling the user to access relevant information without having to transition between contexts and without having to have access to fixed displays. When the user does transition between contexts, a mental workload associated with the transition is reduced.

Referring to <FIG>, a system operable to perform augmented reality visualization is shown and generally designated <NUM>. In the system <NUM>, the processor <NUM> is integrated in an AR headset <NUM>. For example, the visualization data generator <NUM> generates the visualization data <NUM> based at least in part on the HS data <NUM> generated by the AR headset <NUM> (e.g., the headset sensor <NUM>), the VS data <NUM> received from the vehicle <NUM> (e.g., the vehicle sensor <NUM>), the planned route data <NUM>, the location data <NUM>, or a combination thereof, as described with reference to <FIG>.

Integrating one or more of the user movement estimator <NUM>, the gaze target estimator <NUM>, the route estimator <NUM>, the context estimator <NUM>, or the visualization data generator <NUM> in the AR headset <NUM> reduces a number of devices used for augmented reality visualization as compared to the system <NUM> of <FIG>. Alternatively, integrating one or more of the user movement estimator <NUM>, the gaze target estimator <NUM>, the route estimator <NUM>, the context estimator <NUM>, or the visualization data generator <NUM> in the device <NUM> of <FIG> that is distinct from the AR headset <NUM> enables the AR headset <NUM> to offload some of the augmented reality visualization tasks and reduces resource consumption (e.g., memory, processing cycles, or both) of the AR headset <NUM>. It should be understood that particular devices configured to perform particular operations are provided as illustrative examples. In other examples, one or more of the operations described herein may be performed by one or more devices.

<FIG> illustrates an example of a method <NUM> of augmented reality visualization. In some examples, the method <NUM> is performed by the user movement estimator <NUM>, the gaze target estimator <NUM>, the route estimator <NUM>, the context estimator <NUM>, the visualization data generator <NUM>, the processor <NUM>, the interface <NUM>, the device <NUM> of <FIG>, the AR headset <NUM> of <FIG>, or a combination thereof.

The method <NUM> includes receiving vehicle sensor data from one or more vehicle sensors coupled to a vehicle, at <NUM>. For example, the interface <NUM> of <FIG> receives the VS data <NUM> from the vehicle sensor <NUM>, as described with reference to <FIG>. The vehicle sensor <NUM> is coupled to the vehicle <NUM> of <FIG>.

The method <NUM> also includes receiving headset sensor data from one or more headset sensors coupled to an augmented reality headset, at <NUM>. For example, the interface <NUM> of <FIG> receives the HS data <NUM> from the headset sensor <NUM>, as described with reference to <FIG>. The headset sensor <NUM> is coupled to the AR headset <NUM> of <FIG>.

The method <NUM> further includes determining, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the vehicle, at <NUM>. For example, the gaze target estimator <NUM> of <FIG> determines, based on the VS data <NUM> and the HS data <NUM>, the headset orientation <NUM> and the headset location <NUM> of the AR headset <NUM> relative to the vehicle <NUM>, as described with reference to <FIG>.

In some examples, the method <NUM> includes determining, based on the vehicle sensor data, a movement of the vehicle. For example, the user movement estimator <NUM> of <FIG> determines, based on the VS data <NUM>, the vehicle movement <NUM> of the vehicle <NUM>, as described with reference to <FIG>.

The method <NUM> also includes determining, based on the headset sensor data, a movement of the augmented reality headset. For example, the user movement estimator <NUM> of <FIG> determines, based on the HS data <NUM>, a headset movement <NUM> of the AR headset <NUM>, as described with reference to <FIG>.

The method <NUM> further includes estimating, based on a comparison of the movement of the vehicle and the movement of the augmented reality headset, a user portion of the movement of the augmented reality headset caused by a movement of a head of a user of the augmented reality headset and not caused by the movement of the vehicle. For example, the user movement estimator <NUM> of <FIG> estimates, based on a comparison of the vehicle movement <NUM> and the headset movement <NUM>, a user movement <NUM> (e.g., a user portion of the headset movement <NUM>) caused by a movement of a head of a user <NUM> of the AR headset <NUM> and not caused by the vehicle movement <NUM>, as described with reference to <FIG>. The headset orientation <NUM> and the headset location <NUM> are determined based on the user movement <NUM>, as described with reference to <FIG>.

The method <NUM> also includes estimating, based at least in part on the orientation and the location of the augmented reality headset relative to the vehicle, a gaze target of a user of the augmented reality headset, at <NUM>. For example, the gaze target estimator <NUM> of <FIG> determines, based at least in part on the headset orientation <NUM> and the headset location <NUM>, the gaze target <NUM> of the user <NUM> of the AR headset <NUM>, as described with reference to <FIG>.

In some examples, the method <NUM> includes determining, based on the headset sensor data, whether the user is in the vehicle. For example, the context estimator <NUM> of <FIG> determines, based on the HS data <NUM>, whether the user <NUM> is in the vehicle <NUM>, as described with reference to <FIG>. The gaze target estimator <NUM>, responsive to the determination that the user <NUM> is in the vehicle <NUM>, estimates the gaze target <NUM> based on the headset orientation <NUM> and the headset location <NUM>.

The method <NUM> further includes generating visualization data based on the gaze target, at <NUM>. For example, the visualization data generator <NUM> of <FIG> generates the visualization data <NUM> based on the gaze target <NUM>, as described with reference to <FIG>. Responsive to determining that the gaze target <NUM> is inside the vehicle <NUM>, the visualization data <NUM> includes the visual depiction <NUM> of the POI <NUM> that is outside the vehicle <NUM>, as described with reference to <FIG>. Responsive to determining that the gaze target <NUM> is outside the vehicle <NUM>, the visualization data <NUM> includes the visual depiction <NUM> of the POI <NUM> that is inside the vehicle <NUM>, as described with reference to <FIG>.

The method <NUM> further includes sending the visualization data to a display of the augmented reality headset, at <NUM>. For example, the visualization data generator <NUM> of <FIG> sends the visualization data <NUM> to a display of the AR headset <NUM>, as described with reference to <FIG>.

The method <NUM> thus enables determining a user portion of a movement of an AR headset that is caused by a movement of a head of a user of the AR headset and not caused by a movement of a vehicle that the user is in. The method <NUM> also enables visualization data to include visual elements that are selected based on context, e.g., whether the gaze target is inside the vehicle or outside the vehicle. The visual elements can be displayed in a field of view of the user, enabling the user to access relevant information without having to transition between contexts and without having to have access to fixed displays. When the user does transition between contexts, a mental workload associated with the transition is reduced.

The method <NUM> includes receiving vehicle sensor data from one or more vehicle sensors coupled to a first vehicle, at <NUM>. For example, the interface <NUM> of <FIG> receives the VS data <NUM> from the vehicle sensor <NUM>, as described with reference to <FIG>. The vehicle sensor <NUM> is coupled to the vehicle <NUM> of <FIG>.

The method <NUM> further includes determining, based on the vehicle sensor data and the headset sensor data, an orientation and a location of the augmented reality headset relative to the first vehicle, at <NUM>. For example, the gaze target estimator <NUM> of <FIG> determines, based on the VS data <NUM> and the HS data <NUM>, the headset orientation <NUM> and the headset location <NUM> of the AR headset <NUM> relative to the vehicle <NUM>, as described with reference to <FIG>.

In some examples, the method <NUM> includes determining, based on the vehicle sensor data, a movement of the first vehicle. For example, the user movement estimator <NUM> of <FIG> determines, based on the VS data <NUM>, the vehicle movement <NUM> of the vehicle <NUM>, as described with reference to <FIG>.

The method <NUM> further includes estimating, based on a comparison of the movement of the first vehicle and the movement of the augmented reality headset, a user portion of the movement of the augmented reality headset caused by a movement of the head of a user of the augmented reality headset and not caused by the movement of the first vehicle. For example, the user movement estimator <NUM> of <FIG> estimates, based on a comparison of the vehicle movement <NUM> and the headset movement <NUM>, a user movement <NUM> (e.g., a user portion of the headset movement <NUM>) caused by a movement of the head of a user <NUM> of the AR headset <NUM> and not caused by the vehicle movement <NUM>, as described with reference to <FIG>. The headset orientation <NUM> and the headset location <NUM> are determined based on the user movement <NUM>, as described with reference to <FIG>.

The method <NUM> also includes estimating a gaze target of a user of the augmented reality headset based on the headset sensor data and the orientation and the location of the augmented reality headset relative to the first vehicle, at <NUM>. For example, the gaze target estimator <NUM> of <FIG> determines the gaze target <NUM> of the user <NUM> of the AR headset <NUM> based on the HS data <NUM> and the headset orientation <NUM> and the headset location <NUM>, as described with reference to <FIG>.

In some examples, the method <NUM> includes determining, based on the headset sensor data, whether the user is in the first vehicle. For example, the context estimator <NUM> of <FIG> determines, based on the HS data <NUM>, whether the user <NUM> is in the vehicle <NUM>, as described with reference to <FIG>. The gaze target estimator <NUM>, responsive to the determination that the user <NUM> is in the vehicle <NUM>, estimates the gaze target <NUM> based on the headset orientation <NUM> and the headset location <NUM>.

The method <NUM> further includes generating visualization data based on the gaze target, at <NUM>. For example, the visualization data generator <NUM> of <FIG> generates the visualization data <NUM> based on the gaze target <NUM>, as described with reference to <FIG>. Responsive to determining that the gaze target <NUM> is inside the vehicle <NUM>, the visualization data <NUM> includes the visual depiction <NUM> of a first POI (e.g., one or more portions of the planned route 180B), the visual depiction <NUM> of a second POI (e.g., one or more portions of the planned route 180A), or both, that are outside the vehicle <NUM>, as described with reference to <FIG> and <FIG>. The first POI includes at least a portion of a particular route of a vehicle <NUM>. The second POI includes at least a portion of a particular route of a vehicle <NUM>. Responsive to determining that the gaze target <NUM> is outside the vehicle <NUM>, the visualization data <NUM> includes the visual depiction <NUM> of the POI <NUM> that is inside the vehicle <NUM>, as described with reference to <FIG>.

The method <NUM> thus enables visualization data to include visual elements that are selected based on context, e.g., whether the gaze target is inside the vehicle or outside the vehicle. The visual elements indicate one or more routes of the vehicle, another vehicle, or both. The visual elements can be displayed in a field of view of the user, enabling the user to access relevant information without having to transition between contexts and without having to have access to fixed displays. When the user does transition between contexts, a mental workload associated with the transition is reduced.

<FIG> is an illustration of a block diagram of a computing environment <NUM> including a computing device <NUM> configured to support examples of computer-implemented methods and computer-executable program instructions (or code). For example, the computing device <NUM>, or portions thereof, is configured to execute instructions to initiate, perform, or control one or more operations described with reference to <FIG>. In some examples, the computing device <NUM> corresponds to the device <NUM>, the AR headset <NUM>, the vehicle <NUM> of <FIG>, the AR headset <NUM> of <FIG>, or a combination thereof.

The computing device <NUM> includes the processor <NUM>. The processor <NUM> is configured to communicate with system memory <NUM>, one or more storage devices <NUM>, one or more input/output interfaces <NUM>, a transceiver <NUM>, one or more communications interfaces <NUM>, or a combination thereof. The system memory <NUM> includes volatile memory devices (e.g., random access memory (RAM) devices), nonvolatile memory devices (e.g., read-only memory (ROM) devices, programmable read-only memory, and flash memory), or both. The system memory <NUM> stores an operating system <NUM>, which may include a basic input/output system for booting the computing device <NUM> as well as a full operating system to enable the computing device <NUM> to interact with users, other programs, and other devices. The system memory <NUM> stores system (program) data <NUM>. In some examples, the memory <NUM> of <FIG> includes the system memory <NUM>, the one or more storage devices <NUM>, or a combination thereof. In some examples, the system (program) data <NUM> includes the data <NUM> of <FIG>. In some examples, the data <NUM> includes (e.g., indicates) the alert <NUM>, the visualization data <NUM>, the context configuration data <NUM>, the route goal <NUM>, the map data <NUM>, the headset orientation <NUM>, the vehicle map data <NUM>, the gaze target <NUM>, the context <NUM>, the one or more routes <NUM>, the headset location <NUM>, the user movement <NUM>, the vehicle movement <NUM>, the headset movement <NUM>, the VS data <NUM>, the HS data <NUM>, the planned route data <NUM>, the location data <NUM>, or a combination thereof.

The system memory <NUM> includes one or more applications <NUM> executable by the processor <NUM>. As an example, the one or more applications <NUM> include instructions executable by the processor <NUM> to initiate, control, or perform one or more operations described with reference to <FIG>. To illustrate, the one or more applications <NUM> include instructions executable by the processor <NUM> to initiate, control, or perform one or more operations described with reference to the user movement estimator <NUM>, the gaze target estimator <NUM>, the route estimator <NUM>, the context estimator <NUM>, the visualization data generator <NUM>, or a combination thereof.

The processor <NUM> is configured to communicate with one or more storage devices <NUM>. For example, the one or more storage devices <NUM> include nonvolatile storage devices, such as magnetic disks, optical disks, or flash memory devices. In some examples, the storage devices <NUM> include both removable and non-removable memory devices. The storage devices <NUM> are configured to store an operating system, images of operating systems, applications, and program data. In some examples, the system memory <NUM>, the storage devices <NUM>, or both, include tangible computer-readable media. In some examples, one or more of the storage devices <NUM> are external to the computing device <NUM>.

The processor <NUM> is configured to communicate with one or more input/output interfaces <NUM> that enable the computing device <NUM> to communicate with one or more input/output devices <NUM> to facilitate user interaction. The processor <NUM> is configured to detect interaction events based on user input received via the input/output interfaces <NUM>. The processor <NUM> is configured to communicate with devices or controllers <NUM> via the one or more communications interfaces <NUM>. For example, the one or more communications interfaces <NUM> include the interface <NUM> of <FIG> and the devices or controllers <NUM> include the AR headset <NUM>, the headset sensor <NUM>, the vehicle <NUM>, the vehicle sensor <NUM>, the one or more data sources <NUM>, or a combination thereof. In some examples, a non-transitory computer-readable storage medium (e.g., the system memory <NUM>) includes instructions that, when executed by a processor (e.g., the processor <NUM>), cause the processor to initiate, perform, or control operations. The operations include one or more operations described with reference to <FIG>.

Although one or more of <FIG> illustrate systems, apparatuses, and/or methods according to the teachings herein, the application is not limited to these illustrated systems, apparatuses, and/or methods. One or more functions or components of any of <FIG> as illustrated or described herein may be combined with one or more other portions of another of <FIG>. For example, one or more elements of the method <NUM> of <FIG>, one or more elements of the method <NUM> of <FIG>, or a combination thereof, may be performed in combination with other operations described herein. Accordingly, no single example described herein should be construed as limiting and examples described herein may be suitably combined without departing from the teachings of the application. As an example, one or more operations described with reference to <FIG> may be optional, may be performed at least partially concurrently, and/or may be performed in a different order than shown or described.

Examples described above are illustrative and not part of the invenion, as defined by the present set of claims.

Claim 1:
A method (<NUM>) of augmented reality visualization, the method (<NUM>) comprising:
receiving vehicle sensor data (<NUM>) from one or more vehicle sensors (<NUM>, <NUM>) coupled to a first vehicle (<NUM>); and
receiving headset sensor data (<NUM>) from one or more headset sensors (<NUM>, <NUM>) coupled to an augmented reality headset (<NUM>, <NUM>, <NUM>);
determining, based on the vehicle sensor data (<NUM>) and the headset sensor data (<NUM>), an orientation (<NUM>) and a location (<NUM>) of the augmented reality headset (<NUM>, <NUM>, <NUM>) relative to the first vehicle (<NUM>);
estimating a gaze target (<NUM>) of a user (<NUM>) of the augmented reality headset (<NUM>, <NUM>, <NUM>) based on the headset sensor data (<NUM>) and the orientation (<NUM>) and the location (<NUM>) of the augmented reality headset (<NUM>, <NUM>, <NUM>) relative to the first vehicle (<NUM>);
generating visualization data (<NUM>) based on the gaze target (<NUM>), wherein, responsive to determining that the gaze target (<NUM>) is inside the first vehicle (<NUM>), the visualization data (<NUM>) includes a first visual depiction (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of a first point of interest that is outside the first vehicle (<NUM>), wherein the first point of interest includes at least a portion of a particular route (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of a particular vehicle (<NUM>, <NUM>), wherein the particular vehicle (<NUM>, <NUM>) includes the first vehicle (<NUM>) or a second vehicle (<NUM>), and wherein, responsive to determining that the gaze target (<NUM>) is outside the first vehicle (<NUM>), the visualization data (<NUM>) includes a second visual depiction (<NUM>) of a second point of interest that is inside the first vehicle (<NUM>); and
sending the visualization data (<NUM>) to a display of the augmented reality headset (<NUM>, <NUM>, <NUM>).