Patent Description:
This document describes techniques, apparatuses, and systems for vehicle localization using map and vision data. For example, this document describes a localization system that can obtain a map centerline point and vision centerline point of a lane of a roadway in which a vehicle is traveling. The map centerline point is added to a map trail, the vision centerline point is added to a vision trail, and the map trail and the vision trail are maintained as a vehicle trail. The localization system can also obtain the position of the vehicle using positioning data. The localization system can then compare the map centerline point and the vision centerline point to generate a lateral correction and longitudinal correction relative to the vehicle's position. The lateral and longitudinal corrections are used to generate a corrected position, which includes the steps of determining a length of the vehicle trail from the position of the host vehicle; determining whether one or more gaps in the vehicle trail are greater than a gap threshold value, respectively; in response to the trail length being greater than a trail length threshold and the one or more gaps being less than the gap threshold value, identifying the vehicle trail as valid and applying the lateral correction and the longitudinal correction to the position of the host vehicle to provide a corrected position of the host vehicle, the trail length threshold being adjusted based on at least one of a speed of the host vehicle or a curvature of the roadway; or in response to the trail length being less than the trail length threshold or the one or more gaps being greater than the gap threshold value, identifying the vehicle trail as invalid and waiting to apply the lateral correction and the longitudinal correction until the vehicle trail is valid. The vehicle can then be operated in the roadway based on the corrected position. In this way, the described localization system can provide accurate vehicle localization that addresses potential drift caused by lapses in or inaccurate positioning data. The corrected vehicle localization using map and vision data allows the operation of assisted-driving and autonomous-driving systems at higher speeds and on roadways with tighter curves.

This document also describes other operations of the above-summarized systems, techniques, apparatuses, and other methods set forth herein, as well as means for performing these methods.

According to an embodiment, the map centerline point, the vision centerline point, and the position of the host vehicle are obtained in a map coordinate system.

According to an embodiment, the method further comprises: transforming the map centerline point and the vision centerline point from the map coordinate system to a vehicle coordinate system, the vehicle coordinate system being relative to the position of the host vehicle and a heading of the host vehicle.

According to an embodiment, obtaining the vision centerline point of the lane comprises: determining, using the vision data, a left lane line and a right lane line of the lane of the roadway; and determining the vision centerline point of the lane as a lateral centerline point between the left lane line and the right lane line.

According to an embodiment, obtaining the map centerline point of the lane comprises: determining, using the map database, two nearest database centerline points for the lane to the position of the host vehicle; and interpolating the two nearest database centerline points to obtain the map centerline point, the map centerline point having a longitudinal position along the roadway approximately equal to the longitudinal position of the vision centerline point.

According to an embodiment, the vision trail does not include vision centerline points that are located longitudinally ahead of the host vehicle along the roadway.

According to an embodiment, the position of the host vehicle is obtained from at least one of a Global Positioning System (GPS), a Global Navigation Satellite System (GNSS), or an inertial measurement unit (IMU).

According to an embodiment, the method further comprises: obtaining a heading of the host vehicle; generating, based on the comparison of the map centerline point and the vision centerline point, a heading correction for the host vehicle; and applying the heading correction to the heading of the host vehicle to provide a corrected heading of the host vehicle.

According to an embodiment, comparing the map centerline point and the vision centerline point comprises applying a least-squares fitting algorithm to the map centerline point and the vision centerline point.

According to an embodiment, the lateral correction and the longitudinal correction are obtained in a vehicle coordinate system and the method further comprises: transforming the lateral correction and the longitudinal correction to a map coordinate system.

A computer-readable storage media comprises computer-executable instructions that, when executed, cause a processor in a host vehicle to perform the above method or any one of the embodiments.

A system comprises a processor configured to perform the above method or any one of the embodiments.

This Summary introduces simplified concepts for vehicle localization using map and vision data, further described below in the Detailed Description and Drawings. This Summary is not intended to identify essential features of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter.

Details of one or more aspects of vehicle localization using map and vision data are described in this document regarding the following figures. The same numbers are used throughout the drawings to reference like features and components:.

Vehicle localization is an important aspect of assisted-driving and autonomous-driving systems. Assisted-driving and autonomous-driving systems can provide route planning, lane centering, automatic lane changing, and autonomous driving functionalities. These functionalities, however, require accurate localization of the host vehicle to satisfy performance and safety requirements. Positioning with satellite-based systems (e.g., GPS, GNSS) can achieve the necessary accuracy under ideal conditions. Such systems do not provide the required accuracy in all environments (e.g., blocked line of sight to satellites from overpasses or skyscrapers). In such situations, estimates of the vehicle's position drift (e.g., up to two-meter errors), which result in unacceptable accuracy for many ADAS functionalities.

Some vehicle-based systems fuse map data and vision data to correct vehicle localization. These systems often use complex algorithms (e.g., simultaneous localization and mapping (SLAM), Kalman filters, 3D image processing) to correlate map data with vision data. Other systems use blending cubic formulas to link map data to vision data which requires significant computational overhead that is generally unavailable in vehicles.

In contrast, this document describes computationally efficient vehicle localization using map and vision data. For example, this document discloses a localization system that obtains a map centerline point and vision centerline point of a roadway lane. The localization system also determines the position of the vehicle using positioning data. The localization system can then compare the map centerline point and the vision centerline point to generate a lateral and longitudinal correction relative to the vehicle's position. The lateral and longitudinal corrections are applied to the vehicle's position to generate a corrected position. The described system can provide accurate vehicle localization that addresses potential drift caused by lapses in or insufficient data from positioning systems. This improved vehicle localization allows the operation of assisted-driving and autonomous-driving systems at higher speeds and on roadways with tighter curves.

This section describes just one example of how the described techniques and systems can perform vehicle localization using map and vision data. This document describes other examples and implementations.

<FIG> illustrates an example road environment <NUM> in which a localizer <NUM> can perform vehicle localization using map and vision data according to techniques described in this disclosure. <FIG> illustrates the localizer <NUM> as part of a system (not shown) implemented within a vehicle <NUM>. Although presented as a car, vehicle <NUM> can represent other motorized vehicles (e.g., a motorcycle, a bus, a tractor, a semi-trailer truck, or construction equipment). In general, manufacturers can mount or install the localizer <NUM> in any moving platform traveling on the roadway.

Vehicle <NUM> is traveling along a roadway. Although presented as a road (e.g., a highway) with lanes and lane markers in <FIG>, the roadway can be any type of designated travel routes for a vehicle, including, for example, virtual water lanes used by ships and ferries, virtual air lanes used by unmanned aerial vehicles (UAVs) and other aircraft, train tracks, tunnels, or virtual underwater lanes.

The localizer <NUM> obtains vision centerline points <NUM> and map centerline points <NUM> associated with the roadway. The array of vision centerline points <NUM> represents the lateral center of the respective lane as determined by one or more vision sensors (e.g., cameras). The array of map centerline points <NUM> represents the lateral center of the respective lane as stored in a map database (e.g., a high-definition (HD) map database). The map centerline points <NUM> can include respective geographic locations (e.g., latitude and longitude coordinates) provided in sequence according to the desired direction of vehicle travel on sections of the roadway. Each vision centerline point <NUM> and map centerline point <NUM> can include latitude and longitude coordinates in a map coordinate system.

The roadway includes one or more lanes, with the lanes represented by the vision centerline points <NUM>, the map centerline points <NUM>, lane segments <NUM>, and lane segment groups (LSGs) <NUM>. The lane segments <NUM> represent respective portions of a roadway lane. For example, the lane segments <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM> represent respective portions of the current lane in which vehicle <NUM> is traveling. One or more lane segments <NUM> with the same travel direction are included in an LSG <NUM>. The LSGs <NUM> are generally respective portions of a group of lanes in the same travel direction that do not split with unchanging lane markers. For example, the LSG <NUM>-<NUM> includes the lane segments <NUM>-<NUM> and <NUM>-<NUM>. The LSG <NUM>-<NUM> includes the lane segments <NUM>-<NUM> and <NUM>-<NUM>. The LSG <NUM>-<NUM> includes the lane segments <NUM>-<NUM> and <NUM>-<NUM>. Each of the LSGs <NUM> may include a plurality of lines (e.g., vectors of points, lane markers). In some implementations, each of the LSGs <NUM> may include a predetermined origin. The origins can be centered laterally in the respective LSGs <NUM> and at the beginning of each LSG <NUM>. The locations of the origins relative to the respective LSGs <NUM> may vary without departing from the scope of this disclosure.

In the depicted environment <NUM>, one or more sensors (not illustrated) are mounted to or integrated within the vehicle <NUM>. The sensors include vision sensors (e.g., cameras) and position sensors that provide vision data and position data, respectively, to the localizer <NUM>. The position sensors can include GPS and/or GNSS systems or inertial measurement units (IMUs). The localizer <NUM> can also obtain map data stored locally or remotely in a map database. The localizer <NUM> uses the map data and vision data to provide accurate vehicle positioning to assisted-driving and autonomous-driving systems of the vehicle <NUM> (e.g., for lane centering or autonomous driving). For example, the localizer can obtain the vision centerline points <NUM> and the map centerline points <NUM> for the roadway on which the vehicle <NUM> is traveling. The localizer <NUM> compares the vision centerline points <NUM> and the map centerline points <NUM> to generate lateral and longitudinal corrections relative to the position of the vehicle <NUM>, which is obtained based on positioning data. The localizer <NUM> applies the lateral and longitudinal corrections to the vehicle's position to generate a corrected position of the vehicle. In this way, the localizer <NUM> uses map and vision data to provide more accurate vehicle positioning, allowing assisted-driving and autonomous-driving systems to avoid drift in positioning systems and smoothly operate curved roadways at higher speeds.

<FIG> illustrates vehicle software components utilized to perform vehicle localization using map and vision data according to techniques described in this disclosure. The vehicle <NUM> includes one or more processors <NUM>, computer-readable storage media (CRM) <NUM>, one or more communication components <NUM>, and one or more vehicle-based systems <NUM>. The vehicle <NUM> can also include one or more sensors (e.g., a camera, a radar system, a global positioning system (GPS), a global navigation satellite system (GNSS), a lidar system, an inertial measurement unit (IMU)) to provide input data to the localizer <NUM> and the vehicle-based systems <NUM>.

The processor <NUM> can include, as non-limiting examples, a system on chip (SoC), an application processor (AP), an electronic control unit (ECU), a central processing unit (CPU), or a graphics processing unit (GPU). The processor <NUM> may be a single-core processor or a multiple-core processor implemented with a homogenous or heterogenous core structure. The processor <NUM> may include a hardware-based processor implemented as hardware-based logic, circuitry, processing cores, or the like. In some aspects, functionalities of the processor <NUM> and other components of the localizer <NUM> are provided via integrated processing, communication, or control systems (e.g., an SoC), which may enable various operations of the vehicle <NUM> in which the system is embodied.

The CRM <NUM> described herein excludes propagating signals. The CRM <NUM> may include any suitable memory or storage device such as random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), non-volatile RAM (NVRAM), read-only memory (ROM), or Flash memory useable to store device data (not illustrated), map data <NUM>, position data <NUM>, and vision data <NUM> of a data manager <NUM>.

The processor <NUM> executes computer-executable instructions stored within the CRM <NUM> to perform the techniques described herein. For example, the processor <NUM> can execute the data manager <NUM> to process and access the position data <NUM> or cause the localizer <NUM> to perform vehicle localization using the map data <NUM> and the vision data <NUM>.

The data manager <NUM> includes the map data <NUM>, the position data <NUM>, and the vision data <NUM>. The data manager <NUM> can store the map data <NUM>, process updated map data received from a remote source, and retrieve portions of the map data <NUM> for the localizer <NUM>. In the depicted system, the data manager <NUM> is illustrated as located on or within the vehicle <NUM>. In other implementations, the data manager <NUM> or another implementation of the data manager <NUM> can be located remote from vehicle <NUM> (e.g., in the cloud or on a remote computer system) and provide the map data <NUM> or a subset of the map data <NUM> to the vehicle <NUM> and the localizer <NUM>.

The map data <NUM> can include lane geometry data, including the map centerline points <NUM>, the lane segments <NUM>, and the LSGs <NUM>. For example, the map data <NUM> can include lane geometry data for a predetermined proximity around a current position or pose of the vehicle <NUM> or along a navigation route. Lane geometry data for each lane of the environment <NUM> can include an array of points for a lateral (e.g., x) and longitudinal (e.g., y) position of the map centerline points <NUM> in a global or map coordinate system and an offset for the distance traveled by the vehicle <NUM> in a particular lane segment <NUM> and/or LSG <NUM>. The lane geometry data can also include other details associated with roadways (e.g., curvature, traffic control devices, stop bars, localization data, and three-dimensional data). The map data <NUM> can be stored locally or received from a remote source or database that provides lane geometry data for sections of the roadways.

The position data <NUM> provides a pose or position with a lateral (e.g., x) and longitudinal (e.g., y) position in a global or map coordinate system and a heading of the vehicle <NUM>. The position data <NUM> can be generated from sensor data from satellite systems (e.g., GPS, GNSS) and/or motion sensors (e.g., IMUs).

The vision data <NUM> can include the vision centerline points <NUM> and be generated from sensor data of vision sensors (e.g., cameras). The data manager <NUM> can determine a lateral offset of the vehicle <NUM> relative to a center of the lane based on the left and right lane lines detected from the vision data <NUM>. The lateral offset can provide a lateral (e.g., x) and longitudinal (e.g., y) distance of the vehicle <NUM> from a vision centerline point <NUM>, which can be set at a longitudinal distance ahead of the vehicle <NUM> based on its current speed. The vision centerline point <NUM> can be obtained in the map coordinate system. The data manager <NUM> can also apply different logic or algorithms to determine the center line when both lane lines are not visible (e.g., at lane mergers).

Similarly, the processor <NUM> can execute the localizer <NUM> to accurately localize the vehicle <NUM>. The localizer <NUM> can include a position module <NUM>, a lane crawler <NUM>, and a fusion module <NUM>. The position module <NUM> can apply a position model to the position data <NUM> to maintain an estimated pose of the vehicle <NUM> when satellite system data is intermittent. The position module <NUM> can be designed to avoid step-function changes in the estimated pose or heading of the vehicle <NUM>, which can negatively impact the vehicle-based system <NUM>. The position module <NUM> can provide the pose and heading to the lane crawler <NUM>.

The lane crawler <NUM> determines the location of the vehicle <NUM> on or within the map data <NUM> based on the estimated pose or heading provided by the position module <NUM>. The lane crawler <NUM> can output a lateral offset of the vehicle <NUM> to the centerline for the current lane of travel. The lateral offset can be determined as a longitudinal distance ahead of the vehicle based on its speed to match that of the data manager <NUM> for processing the vision data <NUM>.

The fusion module <NUM> can compare a trail or series of the vision centerline points <NUM> to a series or trail of the map centerline points <NUM> to determine a fused trail. The vision centerline points <NUM> and the map centerline points <NUM> are initially in the map coordinate system. The data manager <NUM> or the fusion module <NUM> can transform the trail points from map coordinates into a vehicle coordinate system. The fused trail is obtained by using a transformation to best fit (e.g., a least error fit) a match between the two trails, resulting in a corrected pose to localize the vehicle <NUM>. The corrected pose includes a lateral correction, a longitudinal correction, and/or a heading or rotation correction. The lateral correction can represent a value in the vehicle coordinate system to adjust the pose of the vehicle <NUM> in a lateral direction (e.g., along the x-axis of the vehicle coordinate system). The longitudinal correction can represent a value in the vehicle coordinate system to adjust the pose of the vehicle <NUM> in a longitudinal direction (e.g., along the y-axis of the vehicle coordinate system). The rotation correction can represent a value in radians (or degrees) to correct the heading of the vehicle <NUM>. The resulting lateral correction, longitudinal correction, and rotation correction are transformed into the map coordinate system to apply them to the current vehicle pose estimate and determine the corrected pose.

The fusion module <NUM> can apply several rules to maintain a stable fused trail. The vision centerline points <NUM> and the map centerline points <NUM> are generally sampled on a periodic distance of travel (e.g., every six meters). The vision centerline points <NUM> and the map centerline points <NUM> include synchronized samples of the lateral offset of the centerline relative to the estimated pose of the vehicle <NUM>. Trail points are skipped or not sampled for conditions where either the vision centerline points <NUM> or the map centerline points <NUM> are invalid (e.g., during a lane change where the centerline switches to the new lane). The fusion module <NUM> can also set a minimum valid trail length for determining a corrected pose. In addition, the fusion module <NUM> can allow a configurable gap of trail samples to maintain corrections during transient events (e.g., lane changes, lane mergers).

The communication components <NUM> can include a vehicle-based system interface <NUM>. The vehicle-based system interface <NUM> can transmit data over a communication network of the vehicle <NUM> between various components of the vehicle <NUM> or between components of the vehicle <NUM> and external components. For example, when the data manager <NUM> and the localizer <NUM> are integrated within the vehicle <NUM>, the vehicle-based system interface <NUM> may facilitate data transfer therebetween. When a portion of the data manager <NUM> is remote to the vehicle <NUM>, the vehicle-based system interface <NUM> may facilitate data transfer between the vehicle <NUM> and a remote entity that has the data manager <NUM>. The communication components <NUM> can also include a sensor interface (not illustrated) to relay measurement data from sensors as input to the data manager <NUM>, the vehicle-based systems <NUM>, or other components of the vehicle <NUM>.

The vehicle-based system interface <NUM> can transmit the corrected pose to the vehicle-based systems <NUM> or another component of the vehicle <NUM>. In general, the corrected pose and corrected heading provided by the vehicle-based system interface <NUM> is in a format usable by the vehicle-based systems <NUM>.

The vehicle-based systems <NUM> can use the corrected pose and corrected heading data from the localizer <NUM> to operate the vehicle <NUM> on the roadway. The vehicle-based systems <NUM> can include an assisted-driving system and an autonomous-driving system (e.g., a Traffic-Jam Assist (TJA) system, Lane-Centering Assist (LCA) system, L3/L4 Autonomous Driving on Highways (L3/L4) system). Generally, the vehicle-based systems <NUM> require accurate localization data (e.g., an accuracy of plus or minus ten centimeters) to satisfy performance requirements. As described in greater detail below, the localizer <NUM> expands the instances in which sufficiently accurate localization data is provided to the vehicle-based systems <NUM>. The localizer <NUM> uses the vision detection of lane lines to correct the position data <NUM> by efficiently using a trail of fused data points. In this way, the localizer <NUM> can handle lane changes, tight curves in a roadway, and poor satellite reception.

The vehicle-based systems <NUM> may move the vehicle <NUM> to a particular location on the roadway while operating the vehicle <NUM> based on the corrected pose provided by the localizer <NUM>. The autonomous-driving system can also move the vehicle <NUM> to a specific location on the roadway to avoid collisions with objects detected by other systems (e.g., a radar system, a lidar system) on the vehicle <NUM> and move the vehicle <NUM> back to the original navigation route.

<FIG> illustrates an example conceptual diagram <NUM> indicating how vision centerline points <NUM> and map centerline points <NUM> are managed as inputs to vehicle localization. The localizer <NUM> obtains and manages the vision centerline points <NUM> and the map centerline points <NUM> to generate a vehicle trail.

As the vehicle <NUM> is traveling along a roadway, the localizer <NUM> obtains a vision left lane line <NUM> and a vision right lane line <NUM>. The vision left lane line <NUM> and the vision right lane line <NUM> indicate the current lateral offset of the left lane line and the right lane line, respectively, from a vehicle pose <NUM>. The lateral offsets are provided as a vehicle-coordinate-system value. The localizer <NUM> can set the lateral offsets relative to the vehicle pose <NUM> based on vehicle speed and steering angle.

The vehicle pose <NUM> represents an estimated position of the vehicle <NUM> in the map coordinate system based on the position data <NUM>. As described above, the position data <NUM> can be obtained from GPS sensors, GNSS sensors, IMU sensors, or wheel encoder systems. The vehicle pose <NUM> also includes a heading (e.g., in radians) to allow transforming points between the map coordinate system and the vehicle coordinate system.

The localizer <NUM> uses a vision centerline point <NUM>-<NUM>, which represents the vision centerline point <NUM> for the current vehicle pose, to determine a vision centerline lateral offset from the vehicle pose. The vision centerline lateral offset is translated to the map coordinate system to determine a longitudinal (e.g., x) and lateral (e.g., y) coordinate to be sampled for addition to the vehicle trail. Because the localizer <NUM> does not focus on processing lane lines in front of the vehicle <NUM>, the described vehicle localization is robust to errors in vision processing lane line trajectories.

The localizer <NUM> maintains the vision centerline points <NUM> as vision trail points. The vision centerline points <NUM> are generated from the vision centerline lateral offset. The vision trail points are maintained in the map coordinate system and generally evenly spaced based on the distance traveled by the vehicle <NUM>. For example, the vision centerline points <NUM> can be sampled based on the vehicle <NUM> traveling a specified distance as opposed to at a particular clock interval.

The localizer <NUM> also maintains the map centerline points <NUM> as map trail points. The map centerline points <NUM> are sparsely spaced in the map coordinate system with their density generally inversely proportional to the curvature of the roadway. The localizer <NUM> determines interpolated map centerline points <NUM> from the map centerline points <NUM>. In contrast to the map centerline points <NUM>, the interpolated map centerline points <NUM> are generally evenly spaced in the map coordinate system and paired with a corresponding vision centerline point <NUM>.

The localizer <NUM> uses an interpolated map centerline point <NUM>-<NUM>, which represents the interpolated map centerline point <NUM> for the current vehicle pose <NUM>, to determine a map centerline lateral offset from the vehicle pose. The longitudinal position of the interpolated map centerline point <NUM>-<NUM> is determined to correspond to the longitudinal position of the vision centerline point <NUM>-<NUM>. The lateral position of the interpolated map centerline point <NUM>-<NUM> is determined by interpolating between the two nearest map centerline points <NUM>. The map centerline lateral offset is in or translated to the map coordinate system and paired with the associated vision centerline point <NUM>-<NUM>.

The localizer <NUM> maintains the paired vision centerline points <NUM> and interpolated map centerline points <NUM> as the vehicle trail. In this way, the localizer <NUM> uses a single trail of sampled centerline points from map data and vision data to localize the vehicle. As the vehicle <NUM> travels along a roadway, the vehicle trail points accumulate. The localizer <NUM> can analyze the length of and gaps in the vehicle trail to determine whether the vehicle trail is valid for determining a corrected vehicle position. The localizer <NUM> can determine if the length of the vehicle trail is less than a length threshold. The length threshold can be a configurable distance. For example, the length threshold can be dynamically adjusted based on vehicle speed and roadway curvature to enable shorter trails on roadway sections with curves and longer trails on straight roadway sections. The localizer <NUM> can consider a vehicle trail shorter than the length threshold as invalid for position correction and can wait to apply a lateral or longitudinal correction until the vehicle trail is valid. If the vehicle trail is longer than the length threshold, then the vehicle trail is sufficient to enable the corrected position determination.

The localizer <NUM> can also maintain a list of filtered vehicle trail points, which indicate when a trail point is skipped based on filtering criteria. Trail points can be filtered when there are transients in the vision centerline points <NUM> (e.g., during a lane change). The localizer <NUM> can identify the vehicle trail as invalid if a gap in the vehicle trail exceeds a gap threshold and can wait to apply a lateral or longitudinal correction until the vehicle trail is valid. The vehicle trail can exceed the gap threshold due to persistent issue causing trail points to be filtered out (e.g., temporary loss or blocking of lane line markers).

<FIG> illustrates an example conceptual diagram <NUM> of a software method to localize a vehicle using map and vision data according to techniques described in this disclosure. In particular, the conceptual diagram <NUM> illustrates a software method to manage trail points and apply pose or position corrections. The localizer <NUM> of <FIG> and <FIG> or another component can perform the software method illustrated in the conceptual diagram <NUM>.

The localizer <NUM> can perform the software method at a cyclic rate to determine a pose correction and localize the vehicle <NUM>. For example, the localizer <NUM> can have a cycle time for the conceptual diagram <NUM> of no more than twenty milliseconds to process the map data <NUM>, the position data <NUM>, and the vision data <NUM>.

At <NUM>, the localizer <NUM> determines lateral errors and longitudinal errors associated with map centerline points <NUM> and vision centerline points <NUM>. The localizer <NUM> can determine the lateral errors relative to the vehicle pose <NUM>. The lateral errors can be in units of meters. In performing the software method, the localizer <NUM> determines a pose correction that minimizes the lateral and longitudinal errors between the map data <NUM> and the vision data <NUM>. The localizer <NUM> maintains the trail of vision centerline points <NUM> and the trail of map centerline points <NUM> and interpolated map centerline points <NUM> to infer the shape of the roadway and minimize both the lateral and longitudinal errors. The localizer <NUM> can use an Nth order polynomial equation to represent the centerline of a road segment starting from the vehicle pose <NUM>, with the lateral error representing the first coefficient or offset from the vehicle pose <NUM>.

At <NUM>, the localizer <NUM> determines a change or derivative in the lateral errors and longitudinal errors associated with the map centerline points <NUM>. In particular, the localizer <NUM> determines the change in the map vision error between the current and previous values. This change value is used to detect transient conditions where a significant deviation between the map and vision centerlines exist. The localizer <NUM> primarily uses the change value to detect lane changes where the vision centerline lateral offset changes sign (e.g., from negative to positive).

At <NUM>, the localizer <NUM> determines the distance traveled by the vehicle <NUM>. The localizer <NUM> determines the distance traveled since the last sampling of map and vision lateral offsets to generate a trail point. The distance traveled can be based on the Euclidean distance between the current estimated vehicle pose <NUM> (e.g., from the position data <NUM>) and the corrected vehicle pose for the most recent trail point.

At <NUM>, the localizer <NUM> determines a recapture distance based on a configurable distance between trail points (e.g., six meters when traveling at a speed of <NUM> miles per hour). The configurable distance can be extended to allow skipping trail points under transient conditions (e.g., lane changes). Operation <NUM> is described in greater detail with respect to <FIG>.

At <NUM>, the localizer <NUM> captures a trail point when the vehicle <NUM> travels a distance greater than the recapture distance. Operation <NUM> to create a trail point is described in greater detail with respect to <FIG> and <FIG>.

At <NUM>, the localizer <NUM> can reset the trail under specific conditions where sensor data is invalid or significant transients occur. In such situations, the trail can be considered invalid and reset.

At <NUM>, the localizer <NUM> applies a correction to the vehicle pose <NUM>. When the trail is valid for both length and gaps, the localizer <NUM> applies the pose correction to the vehicle pose <NUM> to generate the corrected vehicle pose. In this way, the localizer <NUM> can use the map data <NUM> and the vision data <NUM> to generate a map trail and a vision trail and localize the vehicle <NUM>.

<FIG> illustrates an example conceptual diagram <NUM> of a software method to determine a recapture distance as part of vehicle localization using map and vision data. In particular, the conceptual diagram <NUM> illustrates a software method to perform operation <NUM> (e.g., determine recapture distance) of <FIG>. The localizer <NUM> of <FIG> and <FIG> or another component can perform the software method illustrated in the conceptual diagram <NUM>.

At <NUM>, the localizer <NUM> determines whether the lateral errors or the change in lateral errors (e.g., from operations <NUM> and <NUM> of <FIG>) exceeds a configurable threshold. If the lateral errors or the change in lateral errors exceed the configurable threshold, then the conditions are not valid for capturing a trail point during the transient condition (e.g., during a lane change).

At <NUM>, if an invalid condition exists, the localizer <NUM> determines a new recapture distance by adding a configurable transient gap distance (e.g., in meters) to the current distance traveled by the vehicle <NUM>.

At <NUM>, the localizer <NUM> ends the software method to determine the recapture distance and returns to the software method described with respect to <FIG>.

At <NUM>, if an invalid condition does not exist, the localizer <NUM> determines whether the vision data <NUM> is valid or sufficient for determining the vision centerline point <NUM> based on both the left lane line and the right lane line. If the vision data <NUM> is valid, the localizer <NUM> proceeds to operation <NUM> and ends the software method to determine the recapture distance.

At <NUM>, if the vision data <NUM> is not valid, the localizer <NUM> determines a new recapture distance by adding a configurable vision gap distance (e.g., in meters) to the current distance traveled by the vehicle <NUM>.

<FIG> illustrates an example conceptual diagram <NUM> of a software method to capture a trail point as part of vehicle localization using map and vision data. In particular, the conceptual diagram <NUM> illustrates a software method to perform operation <NUM> (e.g., capture trail point) of <FIG>. The localizer <NUM> of <FIG> and <FIG> or another component can perform the software method illustrated in the conceptual diagram <NUM>.

At <NUM>, the localizer <NUM> determines whether the vehicle <NUM> has traveled beyond the sampling distance (e.g., six meters for highway speeds) since the most-recent trail point. If so, the data at this vehicle pose is a candidate for recording as a trail point.

At <NUM>, if the vehicle <NUM> has not traveled past the sampling distance, the localizer <NUM> ends the software method to capture the trail point and returns to the software method described with respect to <FIG>. At <NUM>, if the vehicle <NUM> has traveled past the sampling distance, the localizer <NUM> resets the last trail point distance to allow for the next sample.

At <NUM>, the localizer <NUM> determines whether the vehicle <NUM> has traveled past the recapture distance. The localizer <NUM> can set the recapture distance as the last distance traveled by default. If transient or error conditions exist, the localizer <NUM> can adjust the recapture distance. If the vehicle <NUM> has not traveled past the recapture distance, the localizer <NUM> proceeds to operation <NUM> and ends the software method to capture the trail point.

If the vehicle <NUM> has traveled past the recapture distance, the localizer <NUM> samples the data and creates a trail point. At <NUM>, the localizer <NUM> transforms the vision lateral offset to map coordinates. The vision lateral offset is generally in the vehicle coordinate system (e.g., relative to the vehicle <NUM>) and the localizer <NUM> translates and rotates it into the map coordinate system.

At <NUM>, the localizer <NUM> transforms the map lateral offset to map coordinates. The map lateral offset represents the offset of the vehicle <NUM> from the centerline interpolated between map centerline points <NUM>. The localizer <NUM> translates and rotates the map lateral offset into the map coordinate system.

At <NUM>, the localizer <NUM> creates a fusion trail point that stores the vision lateral offset and map lateral offset in map coordinates. The distance traveled is also stored to support determining distances and managing the trail length. Each fusion trail point is added to a queue to maintain the fused trail in memory.

At <NUM>, the localizer <NUM> updates the fused trail by managing the trail length and determining a match between the map and vision data. The operation <NUM> to update the trail is described in greater detail with respect to <FIG>.

<FIG> illustrates an example conceptual diagram <NUM> of a software method to update the trail as part of vehicle localization using map and vision data. In particular, the conceptual diagram <NUM> illustrates a software method to perform operation <NUM> (e.g., update trail) of <FIG>. The localizer <NUM> of <FIG> and <FIG> or another component can perform the software method illustrated in the conceptual diagram <NUM>.

At <NUM>, the localizer <NUM> adds the trail point from the fusion data to the end of the vector or array of trail points.

At <NUM>, the localizer <NUM> truncates the trail. When the distance of the oldest trail point (e.g., the first trail point in the vector or array of trail points) is beyond the configurable trail distance, then the localizer <NUM> removes the oldest trail point from the vector or array of trail points.

At <NUM>, the localizer <NUM> determines whether the trail is valid. In particular, the localizer <NUM> determines whether the trail length is longer than a configurable value and the gap between the two most recent trail points is less than another configurable value. If both conditions are satisfied, the trail is considered valid.

At <NUM>, if the localizer <NUM> determines that the trail is not valid, the localizer <NUM> ends the software method to update the trail and returns to the software method described with respect to <FIG>.

At <NUM>, if the localizer <NUM> determines that the trail is valid, the localizer <NUM> determines a least error point match between the most recent vision centerline point <NUM> and the map centerline point <NUM>. The localizer <NUM> compares the vision centerline point <NUM> to the map centerline point <NUM> to determine a transformation to minimize the error between the map data and vision data.

<FIG> illustrates an example flowchart as an example process to perform vehicle localization using map and vision data. Flowchart <NUM> is shown as sets of operations (or acts) performed, but not necessarily limited to the order or combinations in which the operations are shown herein. Further, one or more of the operations may be repeated, combined, or reorganized to provide other methods. In portions of the following discussion, reference may be made to localizer <NUM> of <FIG> and entities detailed therein, references to which are made for example only. The techniques are not limited to performance by one entity or multiple entities.

At <NUM>, a map centerline point and vision centerline point of a lane of a roadway in which a host vehicle is traveling are obtained. For example, the localizer <NUM> can obtain a vision centerline point <NUM> and a map centerline point <NUM> for the vehicle <NUM>. The vision centerline point <NUM> can be obtained from the vision data <NUM> of a vision-based system of the vehicle <NUM>. The map centerline point <NUM> can be obtained from a map database that includes the map data <NUM>. The vision centerline point and the map centerline point <NUM> can be obtained in a map coordinate system.

The localizer <NUM> can obtain the vision centerline point <NUM> by determining, using the vision data <NUM>, a left lane line and a right lane line of the lane of the roadway. The vision centerline point <NUM> can then be determined as a lateral centerline point between the left lane line and the right lane line.

The localizer <NUM> can obtain the map centerline point <NUM> by determining, using the map data <NUM> in the map database, two nearest database centerline points for the lane to the vehicle's position. The localizer <NUM> can interpolate the two nearest database centerline points to obtain the map centerline point <NUM>. The map centerline point <NUM> has a longitudinal position along the roadway approximately equal to the longitudinal position of the vision centerline point <NUM>.

At <NUM>, a position of the host vehicle can be obtained. For example, the localizer <NUM> can obtain a position of the vehicle <NUM> using the position data <NUM>. The position data <NUM> can be obtained from at least one of a Global Positioning System (GPS), a Global Navigation Satellite System (GNSS), or an inertial measurement unit (IMU). The vehicle's position can be obtained in the map coordinate system. The localizer <NUM> can also use the position data <NUM> to obtain the vehicle' s heading.

The localizer <NUM> can transform the map centerline point <NUM> and the vision centerline point <NUM> from the map coordinate system to a vehicle coordinate system. The vehicle coordinate system is relative to the vehicle's position and heading.

At <NUM>, a comparison of the map centerline point and the vision centerline point can be performed to generate a lateral correction and longitudinal correction relative to the position of the host vehicle. For example, the localizer <NUM> can compare the map centerline point <NUM> and the vision centerline point <NUM> to generate a lateral correction and a longitudinal correction relative to the position of the vehicle <NUM>. The localizer <NUM> can match the map centerline point <NUM> and the vision centerline point <NUM> by applying a least-squares fitting algorithm to the map centerline point <NUM> and the vision centerline point <NUM>.

The lateral correction and longitudinal correction can be obtained in a vehicle coordinate system. The localizer can transform the lateral correction and the longitudinal correction to the map coordinate system. The localizer <NUM> can also generate, based on the map between the map centerline point <NUM> and the vision centerline point <NUM>, a heading correction for the vehicle <NUM>.

At <NUM>, the lateral correction and the longitudinal correction can be applied to the position of the host vehicle to provide a corrected position of the host vehicle. For example, the localizer <NUM> can apply (e.g., add or subtract) the lateral correction and the longitudinal correction to the position of the vehicle <NUM> to generate a correction position. The localizer <NUM> can also apply the heading correction to the vehicle's heading to provide a corrected heading.

The localizer <NUM> can add the vision centerline point <NUM> to a vision trail. The vision trail does not include vision centerline points <NUM> longitudinally ahead of the vehicle <NUM> along the roadway. The localizer <NUM> can also add the map centerline point <NUM> to a map trail. The localizer <NUM> can maintain the map trail and the vision trail in the data manager <NUM> or another component.

At <NUM>, the host vehicle can be operated in the roadway based on the corrected position. For example, the localizer <NUM> can provide the corrected position to a vehicle-based system <NUM> that uses the corrected position to localize and operate the vehicle <NUM> in the roadway.

Claim 1:
A method comprising:
obtaining a map centerline point (<NUM>) and a vision centerline point (<NUM>) of a lane of a roadway in which a host vehicle (<NUM>) is traveling, the map centerline point (<NUM>) being obtained from a map database, the vision centerline point (<NUM>) being obtained from vision data of a vision-based system;
adding the map centerline point (<NUM>) to a map trail;
adding the vision centerline point (<NUM>) to a vision trail;
maintaining the map trail and the vision trail as a vehicle trail;
obtaining a position of the host vehicle (<NUM>);
comparing the map centerline point (<NUM>) and the vision centerline point (<NUM>) to generate a lateral correction and a longitudinal correction relative to the position of the host vehicle (<NUM>);
determining a length of the vehicle trail from the position of the host vehicle (<NUM>);
determining whether one or more gaps in the vehicle trail are greater than a gap threshold value, respectively;
in response to the trail length being greater than a trail length threshold and the one or more gaps being less than the gap threshold value, identifying the vehicle trail as valid and applying the lateral correction and the longitudinal correction to the position of the host vehicle (<NUM>) to provide a corrected position of the host vehicle (<NUM>), the trail length threshold being adjusted based on at least one of a speed of the host vehicle (<NUM>) or a curvature of the roadway; or
in response to the trail length being less than the trail length threshold or the one or more gaps being greater than the gap threshold value, identifying the vehicle trail as invalid and waiting to apply the lateral correction and the longitudinal correction until the vehicle trail is valid; and
operating, based on the corrected position, the host vehicle (<NUM>) in the roadway.