Patent Description:
In <CIT> an RF fingerprinting methodology is generalized to include non-RF related factors. For each fingerprinted tile, there is an associated distance function between two fingerprints (the training fingerprint and the test fingerprint) from within that tile which may be a linear or non-linear combination of the deltas between multiple factors of the two fingerprints. The distance function for each tile is derived from a training dataset corresponding to that specific tile, and optimized to minimize the total difference between real distances and predicted distances. Upon receipt of an inference request, a result is derived from a combination of the fingerprints from the training dataset having the least distance per application of the distance function. Likely error for the tile is also determined to ascertain whether to rely on other location methods.

<CIT> discloses that a GPS sensor senses a current position of a vehicle. A map database stores map data. A controller obtains road information of a road ahead of the vehicle from the map data based on the sensed current position of the vehicle. Then, the controller measures a distance from the vehicle to a predetermined object, which serves as a sensing subject, on the road when the obtained road information is predetermined road information. Then, the controller outputs the measured distance as control information of the vehicle.

<NPL> proposes a model of combining unscented Kalman Filter (UKF) and Back Propagation (BP) neural network algorithms for Inertial Navigation System (INS) errors compensation. UKF is an implementation of Kalman Filter (KF) with great performance and used to ensure the high accuracy when Global Positioning System (GPS) is available. BP is a most widely used method of training multi-layer Feed-Forward Artificial Neural Networks (FFANNs). On the basis of enough training, it can predict INS position error when GPS signal is blocked.

<CIT> proposes a location detection system that performs a process that uses data from a satellite navigation sensor in conjunction with data from a second sensor to determine the accuracy of location estimates provided by the satellite navigation sensor. The system uses the satellite navigation sensor to determine location estimates from at a first time and a second time. The system also uses data from the second sensor to determine a third location estimate, which represents another estimate of the system's location at the second time. The system uses the three location estimates to determine whether the second location estimate satisfies an accuracy condition. If the accuracy condition is satisfied, then the second location estimate may be provided as input to a process.

In <NPL>, multipath error on Global Navigation Satellite System (GNSS) signals in urban environments is characterized with the help of Light Detection and Ranging (LiDAR) measurements. For this purpose, LiDAR equipment and Global Positioning System (GPS) receiver implementing a multipath estimating architecture were used to collect data in an urban environment. This paper demonstrates how GPS and LiDAR measurements can be jointly used to model the environment and obtain robust receivers. Multipath amplitude and delay are estimated by means of LiDAR feature extraction and multipath mitigation architecture. The results show the feasibility of integrating the information provided by LiDAR sensors and GNSS receivers for multipath mitigation.

There is a need for an approach for predicting sensor error (location sensor error).

A method, an apparatus and a non-transitory computer-readable storage medium according to the invention are set out in the appended claims.

The embodiments of the invention are illustrated by way of example in the figures of the accompanying drawings:.

Examples of a method, apparatus, and computer program for predicting sensor error are disclosed. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the invention.

<FIG> is a diagram of a system capable of predicting sensor error, according to one embodiment. As discussed above, the various embodiments described herein relate broadly to autonomous driving, and specifically to vehicle positioning using sensor data. However, it is contemplated that the embodiments are also applicable to any other type of positioning application (e.g., device positioning) and that as illustrative examples which are not embodiments they are applicable as well to any other type of sensor data (e.g., collected sensors for detecting any other physical attribute or characteristic) for which error is to be predicted. With respect to vehicle localization, in order to accurately position a vehicle <NUM>, a class of robotics or automated techniques called localizing can be adopted. For example, during localization, the vehicle position and/or heading direction (e.g., a vehicle pose) can be obtained from various sensors of the vehicle <NUM>. As shown in <FIG>, the vehicle <NUM> can be equipped with a variety of sensors including but not limited to location sensors <NUM> (e.g., configured to process signals from positioning satellites <NUM> - e.g., a Global Positioning System (GPS) satellite), and other sensors <NUM> (e.g., camera sensor, LiDAR sensor, RADAR sensor, etc.) to assist in correctly localizing the vehicle <NUM> on a map <NUM>.

Accurately determining the vehicle <NUM>'s location on the map enables planning of a route, both on fine and coarse scales. On a coarse scale, navigation maps (e.g., a digital map provided from a geographic database <NUM>) allow vehicles <NUM> to know what roads to use to reach a particular destination. However, on a finer scale, digital maps allow vehicles <NUM> to know what lanes to be in and when to make lane changes (e.g., lane-level localization). Knowing this information is important for planning an efficient and safe route. This is because, in complicated driving situations, the vehicle <NUM> may need to execute maneuvers quickly (e.g., lane changes), sometimes before the situations are obvious to the driver or passenger.

With respect to lane localization and also generally with respect to autonomous driving, high accuracy and real-time localization of vehicles <NUM> is needed. Traditionally, most vehicle navigation systems have accomplished this localization using GPS-based location sensors <NUM>, which generally provide a real-time location with a <NUM>% confidence interval of <NUM> meters. However, in complicated urban environments, reflection of GPS signals can further increase this error, such that one's location may be off by as much as <NUM> meters. In other words, the challenge with raw-sensor readings such as those from GPS or equivalent is that systematic errors, stemming from multipath reflection in areas such as urban canyons, cause inaccurate readings from the location sensor <NUM>. Given that the width of many lanes is <NUM>-<NUM> meters, this accuracy is not sufficient to properly localize the vehicle <NUM> (e.g., an autonomous vehicle) so that it can make safe route planning decisions. While sensor fusion using other sensors <NUM>, such as inertial sensors (INS) or inertial measurement units (IMUs) can increase the accuracy of localization by taking into account vehicle movement, the systematic errors in urban canyons or other similar terrain features that result in sensor interference can result in incorrectly positioning the vehicle <NUM> by as much as several blocks away from its true location.

In general, a localization accuracy of around <NUM> or better is needed for safe driving in many areas (e.g., safe autonomous driving). Traditionally, in order to solve the technical issue of error-prone GPS location data in challenge terrain (e.g., dense urban canyons), two GPS sensors can be used to compute a differential sensor reading that accounts for systematic biases or sensor error. However, maintaining multiple location sensors to support differential sensor readings can increase overhead costs (e.g., in terms of both technical resources and monetary costs).

To address these technical challenges, the system <NUM> of <FIG> introduces a fully automated capability to predict the existence of sensor error at a location being evaluated using a machine learning model <NUM> (e.g., a trained neural network or equivalent) of a mapping platform <NUM>. The machine learning model <NUM> processes sensor readings from other sensors of the vehicle <NUM> and optionally also map data representing the location to calculate a predicted sensor error for a location sensor <NUM> of the vehicle <NUM>. In one embodiment, the predicted sensor error can be used as sensor error priors for localization, for instance, when using sensor fusion. As shown in the example of <FIG>, the system <NUM> can leverage the predicted sensor-error priors to ensure faster convergence of the localizer, and consequently quicker vehicle positioning relative to the map. As described above, sensor fusion uses multiple types of sensor data (e.g., LiDAR, camera images, etc.) in addition to GPS sensor data to attempt to localize a vehicle <NUM>. A respective search space (e.g., an area corresponding to a predicted location of the vehicle) for each type of sensor is then calculated. Convergence then refers to identifying a common location among the different search spaces that is the most likely to represent the true location of the vehicle <NUM>.

The example of <FIG> illustrates a first scenario <NUM> in which a vehicle <NUM> is traveling on a road <NUM> that is next to a building <NUM> that can potentially cause multipath issues for the location sensor (e.g., GPS sensor) of the vehicle <NUM>. In this first scenario <NUM>, the system <NUM> determines a search space <NUM> for localizing the vehicle <NUM> without considering any predicted sensor priors. In contrast, in the second illustrated scenario <NUM>, the system <NUM> predicts sensor error priors or sensor error according to the various embodiments described herein. In other words, sensor data (e.g., collected from vehicle sensors other than the location sensor, such as not limited to camera, LiDAR, etc.) are input to the trained machined learning model <NUM> to obtain the GPS-sensor error as output from the model <NUM>. The GPS-sensor error is then subsequently used as map error priors that speed-up convergence of a localizer by reducing the search space <NUM> (e.g., reduced with respect to the search space <NUM> of the first scenario <NUM>), enabling faster vehicle positioning. In this case, the search space <NUM> can be reduced based at least in part on training the machine learning model <NUM> to take into account environmental structures (e.g., the presence of the building <NUM>) when predicting error sensor priors. For example, the reduction of the search space <NUM> can be the machine learning model <NUM> "learning" (e.g., via ground truth training data) that vehicles <NUM> do not normally occupy the same physical space as buildings or other similar structures.

In one embodiment, the trained machine learning model <NUM> can be deployed on the server side in the mapping platform <NUM> and/or locally at the vehicle <NUM> in a mapping module <NUM> over a communication network. Accordingly, within the system <NUM>, the mapping platform <NUM> and/or the mapping module <NUM> can perform the functions related to predicting sensor error using machine learning. As shown in <FIG>, the mapping platform <NUM> and/or mapping module <NUM> include one or more components for predicting sensor error according to various embodiments described herein. It is contemplated that the functions of these components may be combined or performed by other components of equivalent functionality. In this embodiment, the mapping platform <NUM> and/or the mapping module <NUM> include a feature extraction module <NUM>, a training module <NUM>, an error prediction module <NUM>, a localizer <NUM>, and a machine learning model <NUM>. The above presented modules and components of the mapping platform <NUM> and/or the mapping module <NUM> can be implemented in hardware, firmware, software, or a combination thereof. Though depicted as separate entities in <FIG>, it is contemplated that the mapping platform <NUM> and/or the mapping module <NUM> may be implemented as a module of any of the components of the system <NUM> (e.g., vehicle <NUM>, services platform <NUM>, any of the services 127a-<NUM> of the services platform <NUM>, etc.). In another embodiment, one or more of the modules <NUM>-<NUM> and/or the machine learning model <NUM> may be implemented as a cloud-based service, local service, native application, or combination thereof. The functions of the mapping platform <NUM> and/or the mapping module <NUM> are discussed with respect to <FIG> below.

<FIG> is a flowchart of a process for training the machine learning model <NUM> to predict sensor error, according to one embodiment. In various embodiments, the mapping platform <NUM> and/or the mapping module <NUM> (e.g., alone or in combination) may perform one or more portions of the process <NUM> and may be implemented in, for instance, a chip set including a processor and a memory as shown in <FIG>. As such, the mapping platform <NUM>, mapping module <NUM>, and/or any of their component modules can provide means for accomplishing various parts of the process <NUM>, as well as means for accomplishing embodiments of other processes described herein in conjunction with other components of the system <NUM>. In addition, embodiments describing functions/actions related to either of the mapping platform <NUM> or the mapping module <NUM> individually is equally applicable to the other. Although the process <NUM> is illustrated and described as a sequence of steps, its contemplated that various embodiments of the process <NUM> may be performed in any order or combination and need not include all of the illustrated steps.

In step <NUM>, the feature extraction module <NUM> collects ground truth sensor error data for a geographic region, wherein the ground truth sensor data comprises a set of training features extracted from sensor data labeled with ground truth sensor error values. More specifically, in one embodiment, for collection of training data or ground truth data, the mapping platform <NUM> leverages resources such as but not limited to previously generated digital map data (e.g., high definition (HD) map data stored in the geographic database <NUM>) as well as probe or trajectory data collected from vehicles <NUM> that have traversed the geographic areas of interest (e.g., dense urban regions).

<FIG> is a diagram illustrating an example of a vehicle <NUM> equipped with sensors to support the collection of training or ground truth data for machine learning of sensor error, according to one embodiment. As shown, the vehicle <NUM> is equipped with a location sensor <NUM> (e.g., a GPS receiver) and other sensors such as but not limited to a camera sensor <NUM> and LiDAR sensor <NUM>. As the vehicle <NUM> travels in the an area being surveyed, the vehicle <NUM> initiates the capture of location data from the location sensor <NUM>, image data from the camera sensor <NUM>, and three-dimensional mesh data from the LiDAR sensor <NUM>. The location data (e.g., vehicle pose data comprising location and/or direction) can be collected with typical consumer-grade location sensors (e.g., a single GPS receiver versus multiple receivers that generate differential GPS readings) that are susceptible to potential systematic errors. As discussed above, the systematic errors can be caused by multipath reflections from structures, buildings, terrain, etc. (e.g., structure <NUM> as shown in <FIG>) The captured sensor data can then be timestamped with the collection time to generate a data record representing the capture data. For example, the data record (e.g., a probe point) can include but is not limited to the following data fields: <time of collection>, <location/vehicle pose data>, <image data>, <LiDAR mesh data>. As these vehicles drive and collect probe data (e.g., sensor data) in the geographic areas of interest, the probe or sensor data can be collected by the feature extraction module <NUM> of the mapping platform <NUM> to use as training data.

To address these limitations, the feature extraction module <NUM> can automatically generate ground truth sensor error values using more compute-intensive localizers that can perform more accurate sensor fusion (e.g., fusion of the pose data, image data, mesh data, and/or any other collected sensor data) to localize the corresponding collection vehicle with greater accuracy. Because compute-intensive localizers use sensor fusion algorithms that require significant computational resources and time, these types of localizers are generally executed in a batch or offline mode (e.g., on a cloud-based cluster), as opposed for use in real-time applications. Accordingly, in one embodiment, the raw sensor data is first collected. Then, for each drive in the region or area interest, the feature extraction module <NUM> can run the compute-intensive localizer in, for instance, a grid-like pattern for each vehicle pose point to identify the corrected vehicle pose. In one embodiment, the computing resources needed by the compute-intensive localizer can vary with the grid-size (e.g., smaller grid sizes with more grid cells require more compute resources). Accordingly, the grid-size can be specified based on the available computing resources. By searching and computing offsets in this pattern, the feature extraction module <NUM> can identify or select the grid location associated with the highest probability of being the true location of the collection of the vehicle. The pose-offset between the sensed vehicle pose and the corrected vehicle pose can then be used to represent the ground truth sensor error for the corresponding probe or data point.

In one embodiment, feature extraction module <NUM> can receive the sensor data already labeled or annotated with corresponding ground truth sensor error values for the probe points in the data. The true locations of each sampling point are known (e.g., location measure using more accurate GPS sensors, manual observations or logging by the drivers, etc.) and the offset between the known locations and the sensed locations at each probe point is computed to represent the ground truth sensor error value. However, determining ground truth values through manual annotation or through higher accuracy location sensors (e.g., using differential GPS location sensors, etc.) can be cost or resource prohibitive.

In one embodiment where the target sensor being evaluated is a location sensor, this location data labeled with ground truth sensor error values represents one sensor stream of the training data set. In an illustrative example which is not an embodiment, if other types of sensors were the target sensor being evaluated, the sensor stream would be the output of the target sensor labeled with its respective known or true error value. Other sensors than the target sensor (e.g., LiDAR, camera, etc.) would represent other data streams of the training data set.

In one embodiment, the training or ground truth data set can include streams from other data sources such as digital map data (e.g., HD maps of the geographic database <NUM>). For example, with respect to location sensors, the structural features (e.g., buildings, structures, etc.) or terrain can have potential correlation with sensor error (e.g., structures causing multipath interference of GPS sensors and thereby reducing their accuracy). Accordingly, in one embodiment, the feature extraction module <NUM> can query the geographic database <NUM> for map data corresponding to the respective location of the probe points in the training data set. The map data then represents yet another data stream. In yet another embodiment, other types of data can also be included the training data set such as but not limited to weather data, sensor type, sensor manufacturer, vehicle characteristics, etc..

After compiling the data streams into the training data set, in one embodiment, the feature extraction module <NUM> can determine or retrieve relevant features (e.g., characteristics, attributes, properties, etc.) of the compiled training or ground truth data. As used herein, relevant refers to any feature that has an effect or correlation with sensor error with respect to the target sensor. For example, when the target sensor is a location or GPS sensor, features indicating the presence of structures capable of causing multipath interference can potentially be relevant. The feature extraction module <NUM>, for instance, can process image data and/or structure data to determine the presence of buildings, structures, terrain, etc. The characteristics of any detected structures or buildings can be a feature extracted for training the machine learning model <NUM>. Similar data on structures can be extracted from other data streams such as the digital map data by identifying where the map data indicates the presence of any structures, buildings, terrain, etc. within a threshold distance of the probe point location. For example, the digital map data can include three-dimensional (3D) models of nearby structures or buildings, that can be used as input features for training the machine learning model <NUM>.

In one embodiment, the feature extraction process also comprises converting the feature data into a format suitable for input into the machine learning model <NUM>. For example, the features or data items can be converted into an input vector or matrix for training the by the machine learning model <NUM>. Other examples of feature conversion can include but is not limited to: converting a text label to a Boolean flag; converting text labels to categorical labels; converting dates/times to a standardized format; normalizing or converting the extracted feature data into a common taxonomy or dictionary of terms; etc..

As illustrated above, the training or ground truth data may include any number of features or characteristics of the raw sensor data and related information. However, some of the features may not correlate well or at all with sensor error of the target sensor. Including such features in ground truth training data, therefore, would not increase or contribute to the predictive power of the machine learning model <NUM>. Accordingly, in one embodiment, the feature extraction module <NUM> further processes the ground truth data to extract or select one or more training features. In one embodiment, the feature extraction module <NUM> can use any statistical method known in the art (e.g., Principal Component Analysis (PCA) and Univariate Selection) to select the best correlated features to predict or classify the sensor error. In other words, the feature extraction module <NUM> extracts the training features from the ground truth data by first determining a set of candidate features. The mapping platform <NUM> then selects the training features from among the set of candidate features based on a calculated correlation of the candidate features to predicting sensor error.

In step <NUM>, the mapping platform <NUM> trains the machine learning model <NUM> using the ground truth sensor data to calculate a predicted sensor error from a set of input features. For example, the set of input features is extracted from sensor data subsequently collected from a geographic location for which the predicted sensor error for a target sensor is to be calculated (e.g., as described with respect to <FIG> below). In one embodiment, the training module <NUM> can train the machine learning model <NUM> (e.g., a neural network, support vector machine, or equivalent) by obtaining a feature vector or matrix comprising the selected training features from the feature extraction module <NUM>. During the training process, the training module <NUM> feeds the feature vectors or matrices of the training data set (e.g., the ground truth data) into the machine learning model <NUM> to compute a predicted sensor error. The training module <NUM> then compares the predicted sensor error to the ground truth sensor error values of the ground truth training data set. Based on this comparison, the training module <NUM> computes an accuracy of the predictions or classifications for the initial set of model parameters. If the accuracy or level of performance does not meet a threshold or configured level, the training module <NUM> incrementally adjusts the model parameters until the machine learning model <NUM> generates predictions at the desired level of accuracy with respect to the predicted sensor error. In other words, the "trained" machine learning model <NUM> is a model whose parameters are adjusted to make accurate predictions with respect to the ground truth data. The trained machine learning model <NUM> can then be used as according to the embodiments described below in <FIG>.

<FIG> is a flowchart of a process for using the trained machine learning model <NUM> to predict sensor error, according to one embodiment. In various embodiments, the mapping platform <NUM> and/or the mapping module <NUM> (e.g., alone or in combination) may perform one or more portions of the process <NUM> and may be implemented in, for instance, a chip set including a processor and a memory as shown in <FIG>. As such, the mapping platform <NUM>, mapping module <NUM>, and/or any of their component modules can provide means for accomplishing various parts of the process <NUM>, as well as means for accomplishing embodiments of other processes described herein in conjunction with other components of the system <NUM>. In addition,
embodiments describing functions/actions related to either of the mapping platform <NUM> or the mapping module <NUM> individually is equally applicable to the other. Although the process <NUM> is illustrated and described as a sequence of steps, its contemplated that various embodiments of the process <NUM> may be performed in any order or combination and need not include all of the illustrated steps.

The process <NUM> assumes the availability of a trained machine learning model <NUM> such as that generated using the process <NUM> of <FIG> or equivalent. For example, the machine learning model <NUM> can be a pretrained model (e.g., neural network, etc.) that regresses sensor errors based on a combination of sensor inputs, map data (e.g., 3D models of the environment), and other relevant features. After training, the machine learning model <NUM> can be deployed on the server side (e.g., as a cloud-based component) or locally at a client device/entity. For example, when deploying the trained machine learning model <NUM> to predict sensor error for localization, the machine learning model <NUM> can be deployed to the mapping module <NUM> of the vehicle <NUM> or equivalent component. In one embodiment, deployment comprises instantiating an instance of the trained machine learning model <NUM> in the mapping module <NUM>, where the trained machine learning model <NUM> can be used in an online manner. In addition or alternatively, deployment can include using the trained machine learning model <NUM> to precompute sensor-error priors (e.g., predicted sensor error values) corresponding to specific reference locations. The sensor-error priors can then be provided to the localizer <NUM> (e.g., in the vehicle <NUM>) to improve localization.

<FIG> and <FIG> are diagrams illustrating examples of deploying a machine learning model to a vehicle to predict sensor error, according to one embodiment. <FIG> illustrates an example architecture <NUM> in which the machine learning model <NUM> is instantiated on a network component (e.g., the mapping platform <NUM>). In this way, the processing needed by the machine learning model <NUM> is provided on the server side, where computing resources (e.g., processing power, memory, storage, etc.) is generally greater than at a local component (e.g., the vehicle <NUM>).

Under the architecture <NUM>, an OEM platform <NUM> (e.g., operated by automobile manufacturer) collects sensor data observations from vehicles as they travel in a road network. The OEM platform <NUM> sends the observations to the mapping platform <NUM> (e.g., typically operated by a map or other service provider) for ingestion and processing. The mapping platform <NUM> (e.g., where the trained machine learning model <NUM> is instantiated) then processes the received sensor observations using the machine learning model <NUM> to predict sensor error according to the embodiments described herein. In one embodiment, the predicted sensor error data can then be fused with map attribute information to produce a data layer of sensor-error priors <NUM> that correlates error priors to locations in the digital map (e.g., HD map of the geographic database <NUM>). The mapping platform <NUM> can then publish the sensor-error priors <NUM> for delivery to the vehicle <NUM> (or localizer <NUM> of the vehicle <NUM>) either directly or through the OEM platform <NUM>.

<FIG> illustrates an alternative architecture <NUM> in which no sensor error priors are delivered to the vehicle <NUM> or the localizer <NUM> of the vehicle <NUM>. Instead, the trained machine learning model <NUM> is instantiated at a local component or system of a vehicle <NUM> (e.g., the mapping module <NUM>) traveling on a road network. In this way, the local component uses the machine learning model <NUM> to provide a local prediction and correction of sensor error (e.g., sensor error predictions <NUM>) based on locally collected sensor and/or map data. In one use case, the local prediction of sensor error is used to localize a vehicle while operating in an autonomous driving mode.

As shown, to enable this architecture <NUM>, the mapping platform <NUM> trains the machine learning model <NUM> as previously described in the process <NUM>. The mapping platform <NUM> can then deliver the trained machine learning model <NUM> to the vehicle <NUM> either directly or through the OEM platform <NUM>. A local system or component of the vehicle <NUM> then executes an instance of the trained machine learning model <NUM> to make sensor error predictions locally at the vehicle <NUM>. In this way, the vehicle is able detect or map physical dividers on the segments on which it is traveling when a physical divider overlay is not available or when the vehicle does not have communications to network-side components such as the mapping platform <NUM> as it travels. In one embodiment, as new training data is collected, an updated trained machine learning model <NUM> can be delivered to the vehicle <NUM> as needed, periodically, etc..

In one embodiment, when deployed, new sensor readings or data (e.g., camera images or LiDAR data captured in a region of interest) can be input to the trained machine learning model <NUM> (e.g., a neural network) that is used to predict sensor error (e.g., GPS or location sensor error) for such configuration of the environment as present in the sensor readings (e.g., image, LiDAR data, etc.). In step <NUM> of the process <NUM>, the error prediction module <NUM> receives sensor data from at least one sensor, the sensor data collected at a geographic location. The sensor data includes a stream corresponding to readings from sensors of the vehicle <NUM> (camera and/or LiDAR).

In step <NUM>, the error prediction module <NUM> interacts with the feature extraction module <NUM> to extract a set of input features from the sensor data (e.g., features related to structures or configuration of the environment at the geographic location where the data was collected) and, optionally, map data representing the geographic location (e.g., 3D models of structures in the area), and/or any other relevant feature as discussed above with respect to training the machine learning model <NUM>. The set of input features includes one or more attributes of the one or more structures and, optionally, one or more other attributes of the sensor data indicative of the one or more structures. Also as described above, feature extraction further comprises converting the features of the sensor data into format suitable for input into the trained machine learning model <NUM> (e.g., a feature vector/matrix).

In step <NUM>, the error prediction module <NUM> processes the set of input features using the trained machine learning model <NUM> to calculate a predicted sensor error of a target sensor operating at the geographic location. In other words, the machine learning model <NUM> regresses the predicted sensor error using features of the sensor data that were correlated to ground truth sensor error determined during model training. For example, in the use case where the sensor error is GPS-sensor error, the predicted error can be subsequently used as map priors that speed-up convergence of the localizer <NUM>, enabling faster vehicle positioning.

Returning to <FIG>, in one embodiment, the mapping platform <NUM> has connectivity over a communication network <NUM> to the services platform <NUM> (e.g., an OEM platform) that provides one or more services <NUM> (e.g., sensor data collection services). By way of example, the services <NUM> may also be other third-party services and include mapping services, navigation services, travel planning services, notification services, social networking services, content (e.g., audio, video, images, etc.) provisioning services, application services, storage services, contextual information determination services, location-based services, information-based services (e.g., weather, news, etc.), etc. In one embodiment, the services platform <NUM> uses the output (e.g. sensor error predictions or sensor-error priors) of the machine learning model <NUM> to provide services such as navigation, mapping, other location-based services, etc..

In one embodiment, the mapping platform <NUM> may be a platform with multiple interconnected components and may include multiple servers, intelligent networking devices, computing devices, components and corresponding software for predicting sensor error. In addition, it is noted that the mapping platform <NUM> may be a separate entity of the system <NUM>, a part of the one or more services <NUM>, a part of the services platform <NUM>, or included within the vehicle <NUM> (e.g., as a mapping module <NUM>).

In one embodiment, content providers 129a-<NUM> (collectively referred to as content providers <NUM>) may provide content or data (e.g., including geographic data, sensor data, etc.) to the geographic database <NUM>, the mapping platform <NUM>, the services platform <NUM>, the services <NUM>, and the vehicle <NUM>. The content provided may be any type of content, such as map content, textual content, audio content, video content, image content, etc. In one embodiment, the content providers <NUM> may provide content that may aid in predicting sensor error. In one embodiment, the content providers <NUM> may also store content associated with the geographic database <NUM>, mapping platform <NUM>, services platform <NUM>, services <NUM>, and/or vehicle <NUM>. In another embodiment, the content providers <NUM> may manage access to a central repository of data, and offer a consistent, standard interface to data, such as a repository of the geographic database <NUM>.

By way of example, the mapping module <NUM> can be any type of embedded system, mobile terminal, fixed terminal, or portable terminal including a built-in navigation system, a personal navigation device, mobile handset, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system (PCS) device, personal digital assistants (PDAs), audio/video player, digital camera/camcorder, positioning device, fitness device, television receiver, radio broadcast receiver, electronic book device, game device, or any combination thereof, including the accessories and peripherals of these devices, or any combination thereof. It is also contemplated that the mapping module <NUM> can support any type of interface to the user (such as "wearable" circuitry, etc.). In one embodiment, the mapping module <NUM> may be associated with the vehicle <NUM> or be a component part of the vehicle <NUM>.

In one embodiment, the vehicle <NUM> is configured with various sensors for generating or collecting vehicular sensor data, related geographic/map data, etc. In one embodiment, the sensed data represent sensor data associated with a geographic location or coordinates at which the sensor data was collected. In this way, the sensor data can act as observation data that can be aggregated into location-aware training and evaluation data sets. By way of example, the sensors may include a RADAR system, a LiDAR system, a global positioning sensor for gathering location data (e.g., GPS), a network detection sensor for detecting wireless signals or receivers for different short-range communications (e.g., Bluetooth, Wi-Fi, Li-Fi, near field communication (NFC) etc.), temporal information sensors, a camera/imaging sensor for gathering image data, an audio recorder for gathering audio data, velocity sensors mounted on steering wheels of the vehicles, switch sensors for determining whether one or more vehicle switches are engaged, and the like.

Other examples of sensors of the vehicle <NUM> may include light sensors, orientation sensors augmented with height sensors and acceleration sensor (e.g., an accelerometer can measure acceleration and can be used to determine orientation of the vehicle), tilt sensors to detect the degree of incline or decline of the vehicle along a path of travel, moisture sensors, pressure sensors, etc. In a further example embodiment, sensors about the perimeter of the vehicle <NUM> may detect the relative distance of the vehicle from a physical divider, a lane or roadway, the presence of other vehicles, pedestrians, traffic lights, potholes and any other objects, or a combination thereof. In one scenario, the sensors may detect weather data, traffic information, or a combination thereof. In one embodiment, the vehicle <NUM> may include GPS or other satellite-based receivers to obtain geographic coordinates from satellites for determining current location and time. Further, the location can be determined by visual odometry, triangulation systems such as A-GPS, Cell of Origin, or other location extrapolation technologies. In yet another embodiment, the sensors can determine the status of various control elements of the car, such as activation of wipers, use of a brake pedal, use of an acceleration pedal, angle of the steering wheel, activation of hazard lights, activation of head lights, etc..

In one embodiment, the communication network <NUM> of system <NUM> includes one or more networks such as a data network, a wireless network, a telephony network, or any combination thereof. It is contemplated that the data network may be any local area network (LAN), metropolitan area network (MAN), wide area network (WAN), a public data network (e.g., the Internet), short range wireless network, or any other suitable packet-switched network, such as a commercially owned, proprietary packet-switched network, e.g., a proprietary cable or fiber-optic network, and the like, or any combination thereof. In addition, the wireless network may be, for example, a cellular network and may employ various technologies including enhanced data rates for global evolution (EDGE), general packet radio service (GPRS), global system for mobile communications (GSM), Internet protocol multimedia subsystem (IMS), universal mobile telecommunications system (UMTS), etc., as well as any other suitable wireless medium, e.g., worldwide interoperability for microwave access (WiMAX), Long Term Evolution (LTE) networks, code division multiple access (CDMA), wideband code division multiple access (WCDMA), wireless fidelity (Wi-Fi), wireless LAN (WLAN), Bluetooth®, Internet Protocol (IP) data casting, satellite, mobile ad-hoc network (MANET), and the like, or any combination thereof.

By way of example, the mapping platform <NUM>, services platform <NUM>, services <NUM>, vehicle <NUM>, and/or content providers <NUM> communicate with each other and other components of the system <NUM> using well known, new or still developing protocols. In this context, a protocol includes a set of rules defining how the network nodes within the communication network <NUM> interact with each other based on information sent over the communication links. The protocols are effective at different layers of operation within each node, from generating and receiving physical signals of various types, to selecting a link for transferring those signals, to the format of information indicated by those signals, to identifying which software application executing on a computer system sends or receives the information. The conceptually different layers of protocols for exchanging information over a network are described in the Open Systems Interconnection (OSI) Reference Model.

<FIG> is a diagram of a geographic database, according to one embodiment. In one embodiment, the geographic database <NUM> includes geographic data <NUM> used for (or configured to be compiled to be used for) mapping and/or navigation-related services. In one embodiment, geographic features (e.g., two-dimensional or three-dimensional features) are represented using polygons (e.g., two-dimensional features) or polygon extrusions (e.g., three-dimensional features). For example, the edges of the polygons correspond to the boundaries or edges of the respective geographic feature. In the case of a building, a two-dimensional polygon can be used to represent a footprint of the building, and a three-dimensional polygon extrusion can be used to represent the three-dimensional surfaces of the building. It is contemplated that although various embodiments are discussed with respect to two-dimensional polygons, it is contemplated that the embodiments are also applicable to three-dimensional polygon extrusions. Accordingly, the terms polygons and polygon extrusions as used herein can be used interchangeably.

In one embodiment, the geographic database <NUM> includes high resolution or high definition (HD) mapping data that provide centimeter-level or better accuracy of map features. For example, the geographic database <NUM> can be based on Light Detection and Ranging (LiDAR) or equivalent technology to collect billions of 3D points and model road surfaces, structures, buildings, terrain, and other map features down to the number lanes and their widths. In one embodiment, the HD mapping data capture and store details such as the slope and curvature of the road, parking spots, lane markings, roadside objects such as sign posts, including what the signage denotes, etc. By way of example, the HD mapping data enable highly automated vehicles to precisely localize themselves on the road, and to determine road attributes (e.g., learned speed limit values) to at high accuracy levels.

In one embodiment, geographic features (e.g., two-dimensional or three-dimensional features) are represented using polygons (e.g., two-dimensional features) or polygon extrusions (e.g., three-dimensional features). For example, the edges of the polygons correspond to the boundaries or edges of the respective geographic feature. In the case of a building, a two-dimensional polygon can be used to represent a footprint of the building, and a three-dimensional polygon extrusion can be used to represent the three-dimensional surfaces of the building. It is contemplated that although various embodiments are discussed with respect to two-dimensional polygons, it is contemplated that the embodiments are also applicable to three-dimensional polygon extrusions. Accordingly, the terms polygons and polygon extrusions as used herein can be used interchangeably. In one embodiment, the following terminology applies to the representation of geographic features in the geographic database <NUM>.

"Node" - A point that terminates a link.

"Line segment" - A straight line connecting two points.

"Link" (or "edge") - A contiguous, non-branching string of one or more line segments terminating in a node at each end.

"Shape point" - A point along a link between two nodes (e.g., used to alter a shape of the link without defining new nodes).

"Oriented link" - A link that has a starting node (referred to as the "reference node") and an ending node (referred to as the "non-reference node").

"Simple polygon" - An interior area of an outer boundary formed by a string of oriented links that begins and ends in one node. In one embodiment, a simple polygon does not cross itself.

"Polygon" - An area bounded by an outer boundary and none or at least one interior boundary (e.g., a hole or island). In one embodiment, a polygon is constructed from one outer simple polygon and none or at least one inner simple polygon. A polygon is simple if it just consists of one simple polygon, or complex if it has at least one inner simple polygon.

In one embodiment, the geographic database <NUM> follows certain conventions. For example, links do not cross themselves and do not cross each other except at a node. Also, there are no duplicated shape points, nodes, or links. Two links that connect each other have a common node. In the geographic database <NUM>, overlapping geographic features are represented by overlapping polygons. When polygons overlap, the boundary of one polygon crosses the boundary of the other polygon. In the geographic database <NUM>, the location at which the boundary of one polygon intersects they boundary of another polygon is represented by a node. In one embodiment, a node may be used to represent other locations along the boundary of a polygon than a location at which the boundary of the polygon intersects the boundary of another polygon. In one embodiment, a shape point is not used to represent a point at which the boundary of a polygon intersects the boundary of another polygon.

As shown, the geographic database <NUM> includes node data records <NUM>, road segment or link data records <NUM>, POI data records <NUM>, sensor error records <NUM>, other records <NUM>, and indexes <NUM>, for example. More, fewer or different data records can be provided. In one embodiment, additional data records (not shown) can include cartographic ("carto") data records, routing data, and maneuver data. In one embodiment, the indexes <NUM> may improve the speed of data retrieval operations in the geographic database <NUM>. In one embodiment, the indexes <NUM> may be used to quickly locate data without having to search every row in the geographic database <NUM> every time it is accessed. For example, in one embodiment, the indexes <NUM> can be a spatial index of the polygon points associated with stored feature polygons.

In exemplary embodiments, the road segment data records <NUM> are links or segments representing roads, streets, or paths, as can be used in the calculated route or recorded route information for determination of one or more personalized routes. The node data records <NUM> are end points corresponding to the respective links or segments of the road segment data records <NUM>. The road link data records <NUM> and the node data records <NUM> represent a road network, such as used by vehicles, cars, and/or other entities. Alternatively, the geographic database <NUM> can contain path segment and node data records or other data that represent pedestrian paths or areas in addition to or instead of the vehicle road record data, for example.

The road/link segments and nodes can be associated with attributes, such as geographic coordinates, street names, address ranges, speed limits, turn restrictions at intersections, and other navigation related attributes, as well as POIs, such as gasoline stations, hotels, restaurants, museums, stadiums, offices, automobile dealerships, auto repair shops, buildings, stores, parks, etc. The geographic database <NUM> can include data about the POIs and their respective locations in the POI data records <NUM>. The geographic database <NUM> can also include data about places, such as cities, towns, or other communities, and other geographic features, such as bodies of water, mountain ranges, etc. Such place or feature data can be part of the POI data records <NUM> or can be associated with POIs or POI data records <NUM> (such as a data point used for displaying or representing a position of a city).

In one embodiment, the geographic database <NUM> can also include sensor error records <NUM> for storing predicted sensor error data, sensor error priors, and/or related data. The predicted data, for instance, can be stored as attributes or data records of a sensor error data layer or overlay of the geographic database <NUM>, which fuses with the predicted attributes with map attributes or features. In one embodiment, the sensor error records <NUM> can be associated with segments of a road link (as opposed to an entire link). It is noted that the segmentation of the road for the purposes of sensor error prediction can be different than the road link structure of the geographic database <NUM>. In other words, the segments can further subdivide the links of the geographic database <NUM> into smaller segments (e.g., of uniform lengths such as <NUM>-meters). In this way, sensor error can be predicted and represented at a level of granularity that is independent of the granularity or at which the actual road or road network is represented in the geographic database <NUM>. In one embodiment, the sensor error records <NUM> can be associated with one or more of the node records <NUM>, road segment records <NUM>, and/or POI data records <NUM>; or portions thereof (e.g., smaller or different segments than indicated in the road segment records <NUM>) to provide greater localization accuracy/speed and provide for safer autonomous operation of vehicles. In this way, the predicted sensor error data stored in the sensor error records <NUM> can also be associated with the characteristics or metadata of the corresponding record <NUM>, <NUM>, and/or <NUM>.

In one embodiment, the geographic database <NUM> can be maintained by the content provider <NUM> in association with the services platform <NUM> (e.g., a map developer). The map developer can collect geographic data to generate and enhance the geographic database <NUM>. There can be different ways used by the map developer to collect data. These ways can include obtaining data from other sources, such as municipalities or respective geographic authorities. In addition, the map developer can employ field personnel to travel by vehicle along roads throughout the geographic region to observe features (e.g., physical dividers, OPPO, VRU, etc.) and/or record information about them, for example. Also, remote sensing, such as aerial or satellite photography, can be used.

For example, geographic data is compiled (such as into a platform specification format (PSF) format) to organize and/or configure the data for performing navigation-related functions and/or services, such as route calculation, route guidance, map display, speed calculation, distance and travel time functions, and other functions, by a navigation device, such as by the vehicle <NUM>, for example. The navigation-related functions can correspond to vehicle navigation, pedestrian navigation, or other types of navigation. The compilation to produce the end user databases can be performed by a party or entity separate from the map developer. For example, a customer of the map developer, such as a navigation device developer or other end user device developer, can perform compilation on a received geographic database in a delivery format to produce one or more compiled navigation databases.

The processes described herein for predicting sensor error may be advantageously implemented via software, hardware (e.g., general processor, Digital Signal Processing (DSP) chip, an Application Specific Integrated Circuit (ASIC), Field Programmable Gate Arrays (FPGAs), etc.), firmware or a combination thereof. Such exemplary hardware for performing the described functions is detailed below.

<FIG> illustrates a computer system <NUM> upon which an embodiment of the invention may be implemented. Computer system <NUM> is programmed (e.g., via computer program code or instructions) to predict sensor error as described herein and includes a communication mechanism such as a bus <NUM> for passing information between other internal and external components of the computer system <NUM>. Information (also called data) is represented as a physical expression of a measurable phenomenon, typically electric voltages, but including, in other embodiments, such phenomena as magnetic, electromagnetic, pressure, chemical, biological, molecular, atomic, sub-atomic and quantum interactions. For example, north and south magnetic fields, or a zero and non-zero electric voltage, represent two states (<NUM>, <NUM>) of a binary digit (bit). Other phenomena can represent digits of a higher base. A superposition of multiple simultaneous quantum states before measurement represents a quantum bit (qubit). A sequence of one or more digits constitutes digital data that is used to represent a number or code for a character. In some embodiments, information called analog data is represented by a near continuum of measurable values within a particular range.

A processor <NUM> performs a set of operations on information as specified by computer program code related to predicting sensor error. The computer program code is a set of instructions or statements providing instructions for the operation of the processor and/or the computer system to perform specified functions. The code, for example, may be written in a computer programming language that is compiled into a native instruction set of the processor. The code may also be written directly using the native instruction set (e.g., machine language). The set of operations include bringing information in from the bus <NUM> and placing information on the bus <NUM>. The set of operations also typically include comparing two or more units of information, shifting positions of units of information, and combining two or more units of information, such as by addition or multiplication or logical operations like OR, exclusive OR (XOR), and AND. Each operation of the set of operations that can be performed by the processor is represented to the processor by information called instructions, such as an operation code of one or more digits. A sequence of operations to be executed by the processor <NUM>, such as a sequence of operation codes, constitute processor instructions, also called computer system instructions or, simply, computer instructions. Processors may be implemented as mechanical, electrical, magnetic, optical, chemical or quantum components, among others, alone or in combination.

Computer system <NUM> also includes a memory <NUM> coupled to bus <NUM>. The memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, stores information including processor instructions for predicting sensor error. Dynamic memory allows information stored therein to be changed by the computer system <NUM>. RAM allows a unit of information stored at a location called a memory address to be stored and retrieved independently of information at neighboring addresses. The memory <NUM> is also used by the processor <NUM> to store temporary values during execution of processor instructions. The computer system <NUM> also includes a read only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information, including instructions, that is not changed by the computer system <NUM>. Some memory is composed of volatile storage that loses the information stored thereon when power is lost. Also coupled to bus <NUM> is a non-volatile (persistent) storage device <NUM>, such as a magnetic disk, optical disk or flash card, for storing information, including instructions, that persists even when the computer system <NUM> is turned off or otherwise loses power.

Information, including instructions for predicting sensor error, is provided to the bus <NUM> for use by the processor from an external input device <NUM>, such as a keyboard containing alphanumeric keys operated by a human user, or a sensor. A sensor detects conditions in its vicinity and transforms those detections into physical expression compatible with the measurable phenomenon used to represent information in computer system <NUM>. Other external devices coupled to bus <NUM>, used primarily for interacting with humans, include a display device <NUM>, such as a cathode ray tube (CRT) or a liquid crystal display (LCD), or plasma screen or printer for presenting text or images, and a pointing device <NUM>, such as a mouse or a trackball or cursor direction keys, or motion sensor, for controlling a position of a small cursor image presented on the display <NUM> and issuing commands associated with graphical elements presented on the display <NUM>. In some embodiments, for example, in embodiments in which the computer system <NUM> performs all functions automatically without human input, one or more of external input device <NUM>, display device <NUM> and pointing device <NUM> is omitted.

Computer system <NUM> also includes one or more instances of a communications interface <NUM> coupled to bus <NUM>. Communication interface <NUM> provides a one-way or two-way communication coupling to a variety of external devices that operate with their own processors, such as printers, scanners and external disks. In general the coupling is with a network link <NUM> that is connected to a local network <NUM> to which a variety of external devices with their own processors are connected. For example, communication interface <NUM> may be a parallel port or a serial port or a universal serial bus (USB) port on a personal computer. In some embodiments, communications interface <NUM> is an integrated services digital network (ISDN) card or a digital subscriber line (DSL) card or a telephone modem that provides an information communication connection to a corresponding type of telephone line. In some embodiments, a communication interface <NUM> is a cable modem that converts signals on bus <NUM> into signals for a communication connection over a coaxial cable or into optical signals for a communication connection over a fiber optic cable. As another example, communications interface <NUM> may be a local area network (LAN) card to provide a data communication connection to a compatible LAN, such as Ethernet. For wireless links, the communications interface <NUM> sends or receives or both sends and receives electrical, acoustic or electromagnetic signals, including infrared and optical signals, that carry information streams, such as digital data. For example, in wireless handheld devices, such as mobile telephones like cell phones, the communications interface <NUM> includes a radio band electromagnetic transmitter and receiver called a radio transceiver. In certain embodiments, the communications interface <NUM> enables connection to the communication network <NUM> for predicting sensor error.

<FIG> illustrates a chip set <NUM> upon which an embodiment of the invention may be implemented. Chip set <NUM> is programmed to predict sensor error as described herein and includes, for instance, the processor and memory components described with respect to <FIG> incorporated in one or more physical packages (e.g., chips). By way of example, a physical package includes an arrangement of one or more materials, components, and/or wires on a structural assembly (e.g., a baseboard) to provide one or more characteristics such as physical strength, conservation of size, and/or limitation of electrical interaction. It is contemplated that in certain embodiments the chip set can be implemented in a single chip.

The processor <NUM> and accompanying components have connectivity to the memory <NUM> via the bus <NUM>. The memory <NUM> includes both dynamic memory (e.g., RAM, magnetic disk, writable optical disk, etc.) and static memory (e.g., ROM, CD-ROM, etc.) for storing executable instructions that when executed perform the inventive steps described herein to predict sensor error. The memory <NUM> also stores the data associated with or generated by the execution of the inventive steps.

<FIG> is a diagram of exemplary components of a mobile terminal (e.g., handset) capable of operating in the system of <FIG>, according to one embodiment. Generally, a radio receiver is often defined in terms of front-end and back-end characteristics. The front-end of the receiver encompasses all of the Radio Frequency (RF) circuitry whereas the back-end encompasses all of the base-band processing circuitry. Pertinent internal components of the telephone include a Main Control Unit (MCU) <NUM>, a Digital Signal Processor (DSP) <NUM>, and a receiver/transmitter unit including a microphone gain control unit and a speaker gain control unit. A main display unit <NUM> provides a display to the user in support of various applications and mobile station functions that offer automatic contact matching. An audio function circuitry <NUM> includes a microphone <NUM> and microphone amplifier that amplifies the speech signal output from the microphone <NUM>. The amplified speech signal output from the microphone <NUM> is fed to a coder/decoder (CODEC) <NUM>.

The MCU <NUM> receives various signals including input signals from the keyboard <NUM>. The keyboard <NUM> and/or the MCU <NUM> in combination with other user input components (e.g., the microphone <NUM>) comprise a user interface circuitry for managing user input. The MCU <NUM> runs a user interface software to facilitate user control of at least some functions of the mobile station <NUM> to predict sensor error. The MCU <NUM> also delivers a display command and a switch command to the display <NUM> and to the speech output switching controller, respectively. Further, the MCU <NUM> exchanges information with the DSP <NUM> and can access an optionally incorporated SIM card <NUM> and a memory <NUM>. In addition, the MCU <NUM> executes various control functions required of the station. The DSP <NUM> may, depending upon the implementation, perform any of a variety of conventional digital processing functions on the voice signals. Additionally, DSP <NUM> determines the background noise level of the local environment from the signals detected by microphone <NUM> and sets the gain of microphone <NUM> to a level selected to compensate for the natural tendency of the user of the mobile station <NUM>.

Claim 1:
A computer-implemented method for predicting sensor error comprising:
receiving (<NUM>) first sensor data from at least one sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), the first sensor data collected at a geographic location;
extracting (<NUM>) a set of input features from the first sensor data, wherein the extracting (<NUM>) of the set of input features comprises processing the first sensor data to determine one or more environmental structures (<NUM>, <NUM>) at the geographic location; and
processing (<NUM>) the set of input features using a machine learning model (<NUM>) to calculate a predicted sensor error (<NUM>) of a target sensor operating at the geographic location, the target sensor being a location sensor configured to process signals from positioning satellites,
wherein the set of input features includes one or more attributes of the one or more environmental structures (<NUM>, <NUM>) at the geographic location,
wherein the at least one sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is different from the target sensor (<NUM>, <NUM>, <NUM>), and wherein the at least one sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) includes a LiDAR sensor (<NUM>), a camera sensor (<NUM>), or a combination of a LiDAR sensor (<NUM>) and a camera sensor (<NUM>), and
wherein the machine learning model (<NUM>) has been trained (<NUM>) on ground truth sensor error data to use the set of input features to calculate the predicted sensor error (<NUM>), the ground truth sensor error data comprising a set of training features extracted from second sensor data labeled with ground truth sensor error values, the second sensor data labeled with ground truth sensor error values including location data from a location sensor configured to process signals from positioning satellites, image data from a camera sensor, and three-dimensional mesh data from a LiDAR sensor for a plurality of probe points, a true location of each of the probe points being known, and the ground truth sensor error values being represented by an offset between the true location and a sensed location at each of the probe points.