Pose error estimation and localization using static features

The position and/or pose of a vehicle is determined in real time. An observed position and an observed pose of a vehicle are determined. A reference image is generated based on the observed position and the observed pose. The reference image comprises one or more reference static features. A captured image and the reference image are implicitly compared. Based on a result of the comparison, a correction to the observed position, the observed pose, or both is determined.

TECHNOLOGICAL FIELD

An example embodiment relates generally to localization. In particular, an example embodiment generally relates to localization for enabling autonomous and/or assisted driving.

BACKGROUND

Autonomous driving requires high accuracy, real-time localization of vehicles. Currently, most vehicle navigation has been accomplished using a global positioning system (GPS), which provides a real-time location with a 95% confidence interval of 7.8 meters, according to the US government. However, in complicated urban environments, reflection in GPS signals can further increase this error, such that a determined location may be off by as much as 30 meters. Given that the width of many lanes is only 3 to 4 meters, this accuracy is not sufficient to properly localize an autonomous vehicle so that it can make safe route planning decisions. Other sensors, such as inertial measurement units (IMUs) can increase the accuracy of localization by taking into account vehicle movement, but these sensors tend to drift and still do not provide sufficient accuracy for localization. In general, the industry recognizes that a localization accuracy of around 10 cm is desired for autonomous driving in many areas.

BRIEF SUMMARY

At least some example embodiments allow for an accurate, real time (or near real time) determination of a location and/or pose of a vehicle. In particular, methods, apparatus, and computer program products are provided in accordance with an example embodiment in order to provide accurate, real time (or near real time) determinations of a vehicle's position and/or pose. An observed position and an observed pose of a vehicle is determined. In certain embodiments, a captured image is captured by an image capturing device disposed on the vehicle. A reference image is generated based on the observed position and pose. The reference image comprises one or more reference static features. The reference image and the captured image are implicitly compared. Based on a result of the comparison, a correction to the observed position, the observed pose, or both is determined by the pose error network.

In accordance with an example embodiment, a method is provided that comprises determining an observed position and an observed pose of a vehicle. In certain example embodiments, a captured image is captured by an imaging capturing device onboard the vehicle. The method further comprises generating a reference image based on the observed position and pose. The reference image comprises one or more reference static features. The method further comprises implicitly comparing the reference image and the captured image. The method further comprises, based on a result of the comparison, determining, by a pose error network, a correction to the observed position, the observed pose, or both.

In at least some example embodiments, the reference image comprises a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose. In example embodiments, the captured image comprises multi-channel color information. In an example embodiment, the observed position is based on a position determined by a global positioning system (GPS), a position determined by an inertial measurement unit (IMU), or a combination thereof. In an example embodiment, the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

In certain example embodiments, the method further comprises selecting and/or querying one or more reference static features from a plurality of reference static features stored in association with a digital database, library, and/or map tile of a digital map based on the observed position, observed pose, or both. The method may further comprise accessing reference static feature information corresponding to the selected reference static features from the map tile; and generating a three dimensional feature map based on the accessed reference static feature information; and generating the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

In an example embodiment, the method further comprises capturing at least one second captured image that substantially overlaps with the captured image. In an example embodiment, the method further comprises identifying one or more static features within the captured image; and identifying one or more static features within a second captured image. The one or more static features identified within the second captured image comprises at least one static feature that was identified in the captured image. In an example embodiment, the method further comprises generating three dimensional information corresponding to at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image; and generating a three dimensional image based at least in part on the three dimensional information. The method further comprises determining a correction to the observed position, the observed pose, or both. The correction may be determined by the pose error network based on based on implicitly comparing the generated three dimensional image and the reference image.

In accordance with an example embodiment, an apparatus is provided that comprises at least one processor, at least one memory storing computer program code, and a pose error network, with the at least one memory and the computer program code configured to, with the processor, cause the apparatus to at least determine an observed position and an observed pose of a vehicle. In certain embodiments, the at least one memory and the computer program code are configured to, with the processor, cause the capture of a captured image using an image capturing device on board the vehicle. The at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to at least generate a reference image based on the observed position and pose. The reference image comprises one or more reference static features. The at least one memory and the computer program code are configured to, with the processor, cause the apparatus to at least implicitly compare the reference image and the captured image and, based on a result of the comparison, determine, by the pose error network, a correction to the observed position, the observed pose, or both.

In an example embodiment, the reference image is a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose. In example embodiments, the captured image comprises multi-channel color information. In an example embodiment, the observed position is based on a position determined by a global positioning system (GPS), a position determined by an inertial measurement unit (IMU), or a combination thereof. In certain example embodiments, the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

In an example embodiment, the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to at least select one or more reference static features from a plurality of reference static features stored in association with a map tile of a digital map based on the observed position, observed pose, or both. The at least one memory and the computer program code may be further configured to, with the processor, cause the apparatus to at least access reference static feature information corresponding to the selected reference static features from the map tile; and generate a three dimensional feature map based on the accessed reference static feature information; and generate the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

In an example embodiment, the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to capture at least one second captured image that substantially overlaps with the captured image. In an example embodiment, the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to identify one or more static features within the captured image; and identify one or more static features within the second captured image. The one or more static features identified within the second captured image comprises at least one static feature that was identified in the captured image. In an example embodiment, the at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to generate three dimensional information corresponding to at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image; and generate a three dimensional image based at least in part on the three dimensional information. The at least one memory and the computer program code are further configured to, with the processor, cause the apparatus to determine, by the pose error network, a correction to the observed position, the observed pose, or both. The correction may be determined based on implicitly comparing the generated three dimensional image and the reference image.

In accordance with an example embodiment, a computer program product is provided that comprises at least one non-transitory computer-readable storage medium having computer-executable program code instructions stored therein with the computer-executable program code instructions comprising program code instructions configured to determine an observed position and an observed pose of a vehicle. In certain embodiments, the program code instructions comprise program code instructions configured to cause the capture of a captured image using an image capturing device on board the vehicle. The computer-executable program code instructions further comprise program code instructions configured to generate a reference image based on the observed position and pose. The reference image comprises one or more reference static features. The computer-executable program code instructions further comprise program code instructions configured to implicitly compare the reference image and the captured image and, based on a result of the comparison, determine, by a pose error network, a correction to the observed position, the observed pose, or both.

In an example embodiment, the reference image is a binary map that is a two dimensional projection of a three dimensional feature map at the observed position and observed pose. In example embodiments, the captured image comprises multi-channel color information. In certain example embodiments, the observed position is based on a position determined by a global positioning system (GPS), a position determined by an inertial measurement unit (IMU), or a combination thereof. In an example embodiment, the reference image is generated based on reference static feature information that comprises ground truth information for one or more reference static features.

In certain example embodiments, the computer-executable program code instructions further comprise program code instructions configured to select one or more reference static features from a plurality of reference static features stored in association with a map tile of a digital map based on the observed position, observed pose, or both. The computer-executable program code instructions further comprise program code instructions configured to access reference static feature information corresponding to the selected reference static features from the map tile; and generate a three dimensional feature map based on the accessed reference static feature information; and generate the reference image, wherein the reference image is a two dimensional projection of the three dimensional feature map based on the observed position, observed pose, or both.

In an example embodiment, the computer-executable program code instructions further comprise program code instructions configured to cause the capture of at least one second captured image that substantially overlaps with the captured image; identify one or more static features within the captured image; and identify one or more static features within the second captured image. The one or more static features identified within the second captured image comprises at least one static feature that was identified in the captured image. In an example embodiment, the computer-executable program code instructions further comprise program code instructions configured to generate three dimensional information corresponding to at least one static feature identified in both the captured image and the second captured image based at least in part on the captured image and the second captured image; and generate a three dimensional image based at least in part on the three dimensional information. The computer-executable program code instructions further comprise program code instructions configured to determine, by the pose error network, a correction to the observed position, the observed pose, or both. The correction may be determined based on implicitly comparing the generated three dimensional image and the reference image.

In accordance with yet another example embodiment of the present invention, an apparatus is provided that comprises means for determining an observed position and an observed pose of a vehicle. In certain embodiments, the apparatus comprises means for capturing a captured image. The apparatus further comprises means for generating a reference image based on the observed position and observed pose. The reference image comprises one or more reference static features. The apparatus further comprises means for implicitly comparing the reference image to the captured image. The apparatus comprises means for, based on a result of the comparison, determining, by a pose error network, a correction to the observed position, the observed pose, or both.

DETAILED DESCRIPTION

I. General Overview

Methods, apparatus and computer program products are provided in accordance with an example embodiment in order to provide efficient and accurate localization in real time or near real time. For example, the position and pose of a vehicle and/or a vehicle apparatus may be determined. Certain example embodiments are configured to provide efficient and accurate localization by determining a correction to an observed position and/or pose that is determined based on Global Positioning System (GPS), inertial measurement unit(s) (IMU), other position determining systems, or a combination thereof. In at least some example embodiments, the correction to the observed position and/or pose is determined based on an implicit comparison of a captured image comprising multi-channel color information/data and a binary two dimensional projection generated based on the observed position and/or pose by a trained deep net/neural network. For example, an observed position and/or pose may be determined. Based on a database of ground truth features, a reference image may be generated. Meanwhile, an image may be captured. Based on the reference image, features within the captured image may be identified. Based on the difference in position and/or pose of the features within the reference image and the features identified in the captured image, a correction to the observed position and/or pose is determined.

FIG. 1provides an illustration of an example system that can be used in conjunction with various embodiments of the present invention. As shown inFIG. 1, the system may include one or more vehicle apparatuses20, one or more remote apparatuses10, one or more networks50, and/or the like. In various embodiments, the vehicle apparatus20may be an in vehicle navigation system, vehicle control system, a mobile computing device, and/or the like. For example, a vehicle apparatus20may be an in vehicle navigation system mounted within and/or be on-board a vehicle5such as a motor vehicle, non-motor vehicle, automobile, car, scooter, truck, van, bus, motorcycle, bicycle, Segway, golf cart, and/or the like. In various embodiments, the vehicle apparatus20may be a smartphone, tablet, personal digital assistant (PDA), and/or other mobile computing device. In another example, the vehicle apparatus20may be a vehicle control system configured to autonomously drive a vehicle5, assist in control of a vehicle5, and/or the like.

In an example embodiment, a remote apparatus10may comprise components similar to those shown in the example remote apparatus10diagrammed inFIG. 2A. In an example embodiment, the remote apparatus10is configured to provide map updates, reference image information/data, and/or the like to the vehicle apparatus20. In an example embodiment, a vehicle apparatus20may comprise components similar to those shown in the example vehicle apparatus20diagrammed inFIG. 2B. In various embodiments, the remote apparatus10may be located remotely from the vehicle apparatus20. Each of the components of the system may be in electronic communication with, for example, one another over the same or different wireless or wired networks50including, for example, a wired or wireless Personal Area Network (PAN), Local Area Network (LAN), Metropolitan Area Network (MAN), Wide Area Network (WAN), cellular network, and/or the like. In some embodiments, a network40may comprise the automotive cloud, digital transportation infrastructure (DTI), radio data system (RDS)/high definition (HD) radio or other digital radio system, and/or the like. For example, a vehicle apparatus20may be in communication with a remote apparatus10via the network40. For example, the vehicle apparatus20may communicate with the remote apparatus10via a network, such as the Cloud. For example, the Cloud may be a computer network that provides shared computer processing resources and data to computers and other devices connected thereto. For example, the vehicle apparatus20may be configured to receive one or more map tiles of a digital map from the remote apparatus10.

In an example embodiment, as shown inFIG. 2B, the vehicle apparatus20may comprise a processor22, memory24, a communications interface26, a user interface28, one or more location sensors30(e.g., a location sensor such as a GPS sensor; IMU sensors, and/or the like), one or more image capturing devices (e.g., camera(s); two dimensional (2D) and/or three dimensional (3D) light detection and ranging (LiDAR)(s); long, medium, and/or short range radio detection and ranging (RADAR); ultrasonic sensors; electromagnetic sensors; (near-) infrared (IR) cameras, 3D cameras, 360° cameras and/or the like) and/or other sensors that enable the vehicle apparatus20to determine one or more features of the corresponding vehicle's5surroundings, and/or other components configured to perform various operations, procedures, functions or the like described herein. In at least some example embodiments, the memory24is non-transitory and may store information/data corresponding to one or more parameters, features, and/or characteristics of the image capturing device32. The memory24may further store information/data that identifies the correspondence between (a) the position of the vehicle5and the position of the location sensor30, (b) the position of the location sensor30and the image capturing device32, (c) the pose of the vehicle5and the field of view of the image capturing device32, and/or the like. In an example embodiment, the vehicle apparatus may further comprise a pose error network34that has been trained to determine a difference in pose between two images comprising similar features. For example, the pose error network34may be trained to compare an image captured by the image capturing device32to a reference image that is generated based on a pose of the vehicle5observed by the location sensors30and determine the error in the observed pose based thereon. In example embodiments, the pose error network34may operate on a central processing unit (CPU) or a graphics processing unit (GPU) of the vehicle apparatus20.

Similarly, as shown inFIG. 2A, the remote apparatus10may comprise a processor12, memory14, a user interface18, a communications interface16, and/or other components configured to perform various operations, procedures, functions or the like described herein. Certain example embodiments of the vehicle apparatus20and the remote apparatus10are described in more detail below with respect toFIGS. 2A and 2B.

II. Example Operation

In at least some example embodiments, an implicit comparison of a captured image comprising multi-channel color information/data and a binary two dimensional projection generated based on the observed position and/or pose by a trained deep net/neural network is used to determine a correction to an observed position and/or pose of a vehicle5and/or vehicle apparatus20onboard the vehicle5. For example, one or more location sensors30of the vehicle apparatus20may be used to determine an observed position and/or pose of a vehicle5and/or vehicle apparatus20. In an example embodiment, the observed position (or location which is used interchangeably herein) may comprise geospatial coordinates (e.g., latitude and longitude, and/or the like) that are expected to represent the current physical location of the vehicle5and/or the vehicle apparatus20on board the vehicle5. In an example embodiment, the observed pose may comprise an angle describing the direction the vehicle5is expected to be currently facing and/or heading with respect to a reference direction. For example, the observed pose may comprise the expected angle between a reference direction (e.g., North and/or the like) and the direction in which the front of the vehicle5is directed.

A reference image may be accessed and/or generated based on the observed position and/or pose by the remote apparatus10and/or the vehicle apparatus20. For example, the reference image may be a binary map and/or the like generated based on reference static feature information/data corresponding to one or more ground truth features located near the observed position and/or pose. For example, the reference image may provide an expected view of one or more reference static features from the observed position and/or pose.

Before, after, and/or simultaneous to determining the observed position and/or pose and/or accessing and/or generating the reference image, an image capturing device32of the vehicle apparatus20may capture one or more images. The pose error network34may compare the captured image and the reference image. Based on the comparison of the captured image to the reference image, a correction to the observed position and/or pose is determined. Therefore, a direct pose regression may be performed to determine a correction to the observed position and/or pose without requiring explicit detection of features within the captured image. Thus, an accurate corrected position and/or pose for the vehicle5and/or vehicle apparatus20on board the vehicle may be determined in real time (or near real time). Moreover, while described in conjunction with real time or near real time application, some example embodiments may be implemented in other scenarios.

Training the Pose Error Network

FIG. 3provides a flowchart illustrating a process of training the pose error network34. In an example embodiment, the remote apparatus10may train a pose error network and the weights and/or parameters of the trained pose error network may be provided to the pose error network34of the vehicle apparatus20. Starting at block302, one or more known images and the corresponding pose information/data are accessed. A known image is an image having known pose information/data indicating the pose from which the image was captured associated therewith. For example, the ground truth pose may be known for the known image. For example, the apparatus10may access one or more known images and the corresponding pose information/data. For example, the apparatus10may comprise means, such as processor12, memory14, communications interface16, and/or the like, for accessing known images and the corresponding pose information/data.

At block304, one or more training images are generated based on a known image and a simulated pose error. For example, the remote apparatus10may generate one or more training images providing a view that is similar to the known image, but including a simulated pose error. For example, the remote apparatus10may comprise means, such as the processor12and/or the like, for generating one or more training images. For example, a training image may be generated with a simulated pose error incorporated therein. In an example embodiment, a training image is not generated. Rather the known image is used as the training image and a synthetic error is introduced to the known pose information/data for the known image.

At block306, one or more training images are provided to the pose error network for training. For example, the remote apparatus10may provide one or more training images to the pose error network for training. For example, the remote apparatus10may comprise means, such as the processor12, communications interface16, and/or the like, for providing one or more training images to the pose error network for training. In an example embodiment, the corresponding known images may also be provided to the pose error network for training. In an example embodiment, a corresponding comparison image is provided to the pose error network for training. The comparison image is a two dimensional projection of a three dimensional feature map from the known pose of the known image. In example embodiments wherein the known image is used as the training image and synthetic error is introduced into the known pose information/data, the comparison image may be generated based on the synthetic error. In an example embodiment, the pose error network is trained using a supervised training technique.

At block308, the pose error network may be trained by comparing a training image and the corresponding known image and/or the corresponding comparison image and determining the relative difference in pose (e.g., the simulated pose error). In example embodiments, the pose error network may implicitly compare the training image and the corresponding known image and/or the corresponding comparison image. For example, the pose error network may comprise one or more nodes for comparing a training image and the corresponding known image and/or corresponding comparison image. In example embodiments, the comparison image may be a binary projection. The training image may be a multi-modal image. For example, the training image and the known image may include red, green, blue; cyan, magenta, yellow, black; and/or other multi-channel color scheme information/data. Based on the result of the comparison, the pose error network may determine the relative pose difference between the training image and the known image/comparison image. It may then be determined if the determined relative pose difference is equivalent and/or approximately equivalent to the known simulated pose error for the training image.

The process of providing a training image and the corresponding known image/comparison image to the pose error network for comparison and relative pose difference determination may be continued until the pose error network has been sufficiently trained. For example, the pose error network may be trained until the network weights and/or parameters have converged. Once the network weights and/or parameters have converged, they may be provided (e.g., transmitted), for example by the remote apparatus10(e.g., using the communications interface16), to one or more vehicle apparatuses20(e.g., to the communications interface26). The network weights and/or parameters may then be used by the pose error network34of the vehicle apparatus20.

Determining a Correction to the Observed Position and/or Pose

FIG. 4provides a flowchart illustrating a process that may be completed, for example, by the vehicle apparatus20, to provide an accurate position and/or pose of a vehicle5and/or vehicle apparatus20on board the vehicle5. Starting at block102, an observed position and/or pose of the vehicle5and/or vehicle apparatus20is determined. For example, the vehicle apparatus20may determine an observed position and/or pose of the vehicle5and/or vehicle apparatus20. For example, the vehicle apparatus20may comprise means, such as location sensor(s)30(e.g., GPS and/or IMU sensors) and/or the like, for determining an observed position and/or pose of the vehicle5and/or the vehicle apparatus20onboard the vehicle5. In an example embodiment, the observed position (or location which is used interchangeably herein) may comprise geospatial coordinates (e.g., latitude and longitude, and/or the like) that are expected to represent the current physical location of the vehicle5and/or the vehicle apparatus20on board the vehicle5. In an example embodiment, the observed pose may be comprise an angle describing the direction the vehicle5is expected to be currently facing and/or heading with respect to a reference direction. For example, the observed pose may comprise the expected angle between a reference direction (e.g., North and/or the like) and the direction in which the front of the vehicle5is directed.

At block104, a reference image may be generated and/or accessed. For example, the vehicle apparatus20may generate and/or access a reference image. For example, the vehicle apparatus20may comprise means, such as processor22and/or the like, for generating and/or accessing a reference image. For example, the vehicle apparatus20may access static reference information/data from a static feature database or library. The static feature information/data may then be used to generate the reference image. For example, the reference image may be generated based on the observed pose. For example, the reference image may be generated to simulate an image taken at the observed location and at the observed pose. For example, the static feature information/data may be used to generate and/or render a three dimensional feature map, a two dimensional projection of the three dimensional feature map may be generated and/or rendered from the perspective of the observed location and/or pose to provide the reference image. In an example embodiment, one or more parameters, features, or characteristics of the image capture device32may be taken into account when generating the reference image. For example, parameters or characteristics relating to the lens, shutter speed, aperture, light sensitivity, and/or the like of the image capture device32may be accounted for when generating the reference image. For example, one or more parameters or characteristics of the image capture device32that may cause warping of an image captured by the image capture device32may be taken into account with a generating the two dimensional projection of the reference image from the three dimensional feature map.

In an example embodiment, the reference image information/data may comprise information/data corresponding to a plurality of reference static features. The reference image information/data may comprise three dimensional geometry information/data for each of the reference static features, location and/or orientation information/data for each of the reference static features, and/or the like. For example, the reference static feature information/data may comprise an array describing the size, three dimensional shape, location, orientation, and/or the like of the reference static feature. In an example embodiment, the reference static features are ground truth features. For example, the location, orientation, and/or three dimensional geometry information/data corresponding to a reference static feature may be determined through direct observation. Co-pending U.S. patent application Ser. No. 15/355,727, filed Nov. 18, 2016 and titled “Detection Of Invariant Features For Localization,” provides an example of a static feature information/data library, database, or repository, and is incorporated by reference herein in its entirety.

In an example embodiment, the static feature information/data may be stored in association with a digital map. For example, the remote apparatus10may determine and provide a library, database, or repository of static feature information/data. For example, the static feature information/data corresponding to one or more reference static features may be used to generate the reference image. The reference static features may comprise a ground truth features. In an example embodiment, the static feature information/data is organized into tiles and/or stored in association with and/or as part of a digital map. For example, a map tile of a digital map may comprise static feature information/data for one or more static features located within the map tile.

For example, a map may be tiled such that map information/data may be stored, received, provided, transmitted, and/or the like in a modular format (e.g., tile by tile). In various embodiments, the tiles may be defined by a set of parallel and perpendicular tile boundaries. For example, the tiles may be rectangular or square (e.g., 2 km by 2 km squares). In other embodiments, the tiles may be defined by boundaries which are curved, not parallel and/or perpendicular to one or more other boundaries, and/or the like. In various embodiments, the tiles may be a uniform tiling of the map. In other embodiments, the tiles may vary in size and/or shape based on the geography of the map region, the topology of the map region, population or feature density within the map region, and/or the like. In an example embodiment, reference image information/data is organized such that reference image information/data corresponding to reference static features located with a particular tile is stored in association with the map information/data for the particular tile. For example, the map information/data for a tile may comprise reference image information/data for reference static features located within the tile. In an example embodiment, accessing and generating the reference image may comprise selecting one or more reference static features that are associated with a tile based on the observed position and/or pose. The reference static feature information/data corresponding to the selected reference static features may be accessed and the reference image may be generated therefrom by applying the observed position and/or pose. For example, the reference image may be a two dimensional projection of the reference static features expected to be viewable at the observed position and from the observed pose.

FIG. 5Bshows an example reference image60. In an example embodiment, the reference image60is a binary map. In some example embodiments, the reference image is a multi-channel representation of one or more reference static features. For example, the reference image60may comprise one or more reference static features64,66. The reference static features64,66may be ground truth features. In an example embodiment, the reference static features64,66may be landmarks in the area, such as building facades, windows, doors, signs, lamp posts, lane lines, man holes, guard rails tagged with their locations, geological features, and/or the like. In some embodiments, the reference static features64,66may comprise features that are not noticeable by the human eye. In an example embodiment, the reference static features64,66may comprise features within the road surface, and/or any other computer detectable feature that is expected to not change significantly over the course of a few days, few weeks, a few months, and/or a few years.

Continuing withFIG. 4, at block106, an image is captured. For example, the vehicle apparatus20may capture an image. For example, the vehicle apparatus20may comprise means, such as image capturing device32, processor22, and/or the like, for capturing an image. The captured image may be an image, video, and/or the like of the surroundings of the vehicle5and/or vehicle apparatus20. The captured image may comprise one or more features.FIG. 5Aillustrates an example captured image50. The captured image50may comprise one or more static features54,56, and/or one or more non-static features52. In example embodiments, the captured image may comprise red, green, blue (RGB); cyan, yellow, magenta, and black (cymk), or other multi-channel color information/data.

Returning toFIG. 4, at block108, the reference image and the captured image may be compared. For example, the vehicle apparatus20may compare the reference image and the captured image. For example, the vehicle apparatus20may comprise means, such as processor22, pose error network34, and/or the like, for comparing the reference image and the captured image. For example, the reference image and the captured image may be compared by the pose error network34to determine how the static features in the reference image align with the captured image. In example embodiments, the captured image may be implicitly compared to the reference image by the pose error network34. For example, the pose error network34may be a trained relative pose regression neural network and the vehicle apparatus20may use the pose error network34to compare a captured RGB (or other multi-channel color) image and the projected binary map that is the reference image. For example, as shown inFIG. 5C, the reference image60may be compared to the captured image50. For example, the missregistration between the static features64and66in the reference image60and the corresponding static features54and56in the captured image50may be determined. As shown inFIG. 5C, the captured image50may comprise RGB, cymk, or other multi-channel color information and the reference image60may be a binary two dimensional projection of a three dimensional feature map. As the reference image is generated to provide a view of the surroundings of the vehicle5and/or the vehicle apparatus20from the observed position and pose, the location and orientation of the reference static features in the reference image will be similar to the location and orientation of the static features in the captured image. Thus, the static features in the captured image need not be explicitly identified. Rather the alignment between the captured image and the reference image may be used to determine the missregistration between the reference static features and the static features present in the captured image.

For example,FIG. 5Cshows the reference image60overlaid on the captured image50. As can be seen fromFIG. 5C, the correspondence between static feature54and reference static feature64is easily determined without actively requiring identification of the static feature54. Therefore, the missregistration between the captured image and the reference image may be quickly determined. In example embodiments, the reference image and the captured image may be compared (or implicitly compared) and, due to an error in the observed location, the reference image and the captured image may be misaligned. For example, a static feature in the reference image may be translated with respect to a corresponding static feature in the captured image. The misalignment between the reference image and the captured image may be used to determine a correction to the observed position. In an example embodiment, a corrected position may be determined based on the correction to the observed position and the observed position and an updated reference image may be generated based on the corrected position. The correction to the observed pose may then be determined by comparing (or implicitly comparing) the captured image to the updated reference image.

Continuing withFIG. 4, at block110, a correction to the observed position and/or pose of the vehicle5and/or the vehicle apparatus20may be determined. For example, the vehicle apparatus20may determine a correction to the observed position and/or pose. For example, the vehicle apparatus20may comprise means, such as the processor22, pose error network34, and/or the like, for determining a correction to the observed position and/or pose of the vehicle5and/or vehicle apparatus20. For example, based on the comparison of the captured image and the reference image, the pose error network34may determine the missregistration and/or relative pose difference between the reference image and the captured image. In an example embodiment, the correction is determined based on the result of the comparison and/or alignment of the reference image and the captured image. For example, the correction may be determined based on the missregistration and/or relative pose difference between the reference image and the captured image. For example, the pose error may be equal in magnitude to the missregistration and/or relative pose difference between the reference image and the captured image.

Once the correction to the observed position and/or pose is determined, the vehicle apparatus20may apply the correction to the observed position and/or pose to determine the corrected location and/or pose of the vehicle5and/or vehicle apparatus20. The corrected location and/or pose may then be provided through an output device of the vehicle apparatus20(e.g., a display of the user interface28, and/or the like), communicated to another computing entity (e.g., the remote apparatus10), and/or used as input to one or more processes executed by the vehicle apparatus20that require an accurate determination of the vehicle and/or vehicle apparatus's location and/or pose. For example, the corrected location and/or pose may be used as input to one or more processes used to autonomously drive the vehicle5and/or assist in driving the vehicle5(e.g., route planning and/or the like). For example, the vehicle apparatus20may make one or more route planning decisions and operate the vehicle5in accordance therewith based on the corrected location and/or pose.

Thus, an accurate, real time (or near real time) determination of the location and/or pose of the vehicle5and/or vehicle apparatus20may be determined. In an example embodiment, the location of the vehicle5and/or vehicle apparatus20on board the vehicle5may be determined to within a predefined positional accuracy, such as an accuracy of10centimeters. In certain example embodiments, the pose of the of the vehicle5and/or vehicle apparatus20on board the vehicle may be determined to within a predefined accuracy, such as an accuracy of two degrees. Thus, in an example embodiment, the location and/or pose of the vehicle5and/or vehicle apparatus20may be determined with an accuracy that enables and/or facilitates autonomous operation of the vehicle5. Various predefined positional and pose accuracies may be used in various embodiments, as appropriate for the application.

In some example embodiments, two or more captured images may be used to determine a correction of the observed position and/or pose of the vehicle5and/or vehicle apparatus20. For example, two or more captured images may be used to generate a captured three dimensional image and/or to determine the location and/or orientation of one or more static features in the surroundings of the vehicle5and/or vehicle apparatus20in three dimensions. For example, two or more images may be captured by the image capturing device32that substantially overlap. Two or more images may substantially overlap if at least one static feature is present and/or identifiable in each of the two or more images. Three dimensional information/data for one or more static references identified in the two or more captured images may be determined based on the two or more images. In an example embodiment, determining the correction to the pose may be more efficient and/or more accurate when three dimensional information/data is used to describe one or more static features identified in the captured images.

Alternative Method for Determining a Correction to the Observed Position and/or Pose

FIG. 6provides a flowchart illustrating processes and procedures that may be completed by a vehicle apparatus20to perform localization of a vehicle5and/or vehicle apparatus20. Starting at block202, an observed position and/or pose of the vehicle5and/or vehicle apparatus20is determined. For example, the vehicle apparatus20may determine an observed position and/or pose of the vehicle5and/or vehicle apparatus20. For example, the vehicle apparatus20may comprise means, such as location sensor(s)30(e.g., GPS and/or IMU sensors) and/or the like, for determining an observed position and/or pose of the vehicle5and/or the vehicle apparatus20onboard the vehicle5.

At block204, a reference image may be generated and/or accessed. For example, the vehicle apparatus20may generate and/or access a reference image. For example, the vehicle apparatus20may comprise means, such as processor22and/or the like, for generating and/or accessing a reference image. For example, as described above, the vehicle apparatus20may access static feature information/data from a reference static feature library, database, or repository. In an example embodiment, the static feature information/data is stored as part of a map tile of a digital map. The static feature information/data may then be used to generate the reference image. In an example embodiment, the reference image is a three dimensional image. For example, the reference image may be a three dimensional feature map based on the static feature information/data. In an example embodiment, the reference image a binary three dimensional feature map. In an example embodiment, the reference image is generated based on the observed position and/or pose. For example, the reference image may be an expected view of one or more static features at the observed position and/or pose. In an example embodiment, the reference image may be a three dimensional feature map. The feature map may be a binary feature map or a multi-channel feature map.

At block206, a first image is captured. For example, the vehicle apparatus20may capture a first image. For example, the vehicle apparatus20may comprise means, such as image capturing device32, processor22, and/or the like, for capturing a first image. The first captured image may be an image, video, and/or the like of the surroundings of the vehicle5and/or vehicle apparatus20onboard the vehicle5. The first captured image may comprise one or more features. For example, the first captured image may comprise one or more non-static features and one or more static features.

At block208, one or more static features in the first captured image are identified. For example, the vehicle apparatus20may identify one or more static features in the first captured image. For example, that vehicle apparatus20may comprise means, such as processor22and/or the like, for identifying one or more static features in the first captured image. In an example embodiment, the reference image and the static features therein may be used to guide the identification of the static features in the captured image.

At block210, at least one second image is captured. For example, the vehicle apparatus20may capture one or more second images. For example, the vehicle apparatus20may comprise means, such as image capturing device32, processor22, and/or the like, for capturing one or more second images. The second captured image may be an image, video, and/or the like of the surroundings of the vehicle5and/or vehicle apparatus20onboard the vehicle5. The second captured image may comprise one or more features. For example, the second captured image may comprise one or more non-static features and one or more static features. In at least some example embodiments, the first captured image and the second captured image substantially overlap. For example, at least one static reference is present and/or identifiable in both the first captured image and the second captured image. In an example embodiment, the image capture device32may be configured to capture a series of images at a rate of approximately 30 Hz, for example. In an example embodiment, the first and second captured images may be consecutive images in the series of captured images. In example embodiments, a plurality of second images may be captured. For example, in some embodiments, may be small or short (e.g., tenth of a second long, half a second long, one second long, and/or the like) video feed captured by the image capturing device32.

At block212, one or more static features in the second captured image may be identified. For example, the vehicle apparatus20may identify one or more static features in the second captured image. For example, that vehicle apparatus20may comprise means, such as processor22and/or the like, for identifying one or more static features in the second captured image. In an example embodiment, the static features in the second image may be identified based on the reference static features in the reference image. In an example embodiment, at least one static feature is identified in the second image that corresponds to a static feature identified in the first image.

At block214, three dimensional information/data may be determined and/or generated for one or more static features identified in both the first and second captured images. For example, the three dimensional information/data may comprise three dimensional shape information/data for each of static features identified in both the first and second captured images. In some example embodiments, a three dimensional image is generated based on the first and second captured images. For example, a three dimensional image may be generated based on the three dimensional information/data determined and/or generated for the one or more static features identified in both the first and second captured images. For example, the vehicle apparatus20may determine and/or generate three dimensional information/data for one or more static features identified in both the first and second captured images. For example, the vehicle apparatus20may generate a three dimensional image based on the three dimensional information/data for one or more static features identified in both the first and second captured images. For example, the vehicle apparatus20may comprise means, such as processor22and/or the like, for determining and/or generating three dimensional information/data for one or more static features identified in both the first and second captured images. For example, the vehicle apparatus20may comprise means, such as the processor22and/or the like, for generating a three dimensional image based on the three dimensional information/data for one or more static features identified in both the first and second captured images. For example, tie points may be established between the first captured image and the second captured image. The tie points may then be used to determine three dimensional information for one or more static features identified in both the first captured image and the second captured image. As should be understood various techniques may be used to determine the three dimensional information/data for the one or more static features identified in both the first image and the second image based on the first captured image and the second captured image.

At block216, the generated three dimensional image may be compared to the reference image. For example, the vehicle apparatus20may compare the location and/or orientation of the static features in the generated three dimensional image to the location and/or orientation of the reference static features in the reference image. For example, the vehicle apparatus20may comprise means, such as the processor22, pose error network34, and/or the like, for comparing the location and/or orientation of the static features in the generated three dimensional image to the location and/or orientation of the reference static features in the reference image. In example embodiments, the error pose network34may implicitly compare the generated three dimensional image and the reference image. For example, the missregistration and/or relative pose difference between the reference image and the generated three dimensional image may be determined. In some embodiments a three dimensional image may not be generated and the location of the static features within the first and/or second captured images and the three dimensional information/data for the static features may be compared to corresponding information/data for the reference static features.

At block218, a correction to the observed position and/or pose of the vehicle5and/or the vehicle apparatus20may be determined. For example, the vehicle apparatus20may determine a correction to the observed position and/or pose. For example, the vehicle apparatus20may comprise means, such as the processor22, pose error network34, and/or the like, for determining a correction to the observed position and/or pose of the vehicle5and/or vehicle apparatus20. In an example embodiment, the correction is determined based on the result of the comparison of the three dimensional image and the reference image. For example, the correction may be determined based on the comparison of the location and/or orientation of the static features identified in the generated three dimensional image to the reference static features in the reference image. For example, the correction may be determined based on the average translation and/or rotation that must be performed to align each of the reference static features in the reference image with the corresponding static features in the generated three dimensional image.

Once the correction to the observed position and/or pose is determined, the vehicle apparatus20may apply the correction to the observed position and/or pose to determine the corrected location and/or pose of the vehicle5and/or vehicle apparatus20. The corrected location and/or pose may then be provided through an output device of the vehicle apparatus20(e.g., a display of the user interface28, and/or the like), communicated to another computing entity (e.g., the remote apparatus10), and/or used as input to one or more processes executed by the vehicle apparatus20that require an accurate determination of the vehicle5and/or vehicle apparatus's location and/or pose. For example, the corrected location and/or pose may be used as input to one or more processes used to autonomously drive the vehicle5and/or assist in driving the vehicle5.

Thus, an accurate, real time (or near real time) determination of the location and/or pose of the vehicle5and/or vehicle apparatus20may be determined. In an example embodiment, the location of the vehicle5and/or vehicle apparatus20on board the vehicle5may be determined to within a predefined positional accuracy, such as an accuracy of 10 centimeters. In an example embodiment, the pose of the of the vehicle5and/or vehicle apparatus20on board the vehicle5may be determined to within a predefined accuracy, such as an accuracy of two degrees. Thus, in an example embodiment, the location and/or pose of the vehicle5and/or vehicle apparatus20may be determined with an accuracy that enables and/or facilitates autonomous operation of the vehicle5. Various predefined positional and pose accuracies may be used in various embodiments, as appropriate for the application.

In some example embodiments, the process of determining the correction of the position and/or pose of the vehicle5and/or vehicle apparatus20on board the vehicle5is completed by the vehicle apparatus20. In some example embodiments, the vehicle apparatus20may determine the observed position and pose and capture the one or more images. The vehicle apparatus may then provide (e.g., transmit) the observed position and pose and one or more captured images to the remote apparatus10. The remote apparatus10may then identify static features within the captured images and perform the regression to determine the correction to the observed position and/or pose based on the differences in the location and/or orientation of the static features in the captured image(s) compared to the location and/or orientation of the reference static features in the reference image. The remote apparatus10may then provide (e.g. transmit) the correction to the position and/or pose or the corrected position and/or pose to the vehicle apparatus20.

III. Example Apparatus

The vehicle apparatus20and/or remote apparatus10of an example embodiment may be embodied by or associated with a variety of computing devices including, for example, a navigation system including an in-vehicle navigation system, a vehicle control system, a personal navigation device (PND) or a portable navigation device, an advanced driver assistance system (ADAS), a global positioning system (GPS), a cellular telephone, a mobile phone, a personal digital assistant (PDA), a watch, a camera, a computer, and/or other device that can perform navigation-related functions, such as digital routing and map display. Additionally or alternatively, the vehicle apparatus20and/or remote apparatus10may be embodied in other types of computing devices, such as a server, a personal computer, a computer workstation, a laptop computer, a plurality of networked computing devices or the like, that are configured to update one or more map tiles, analyze probe points for route planning or other purposes. In this regard,FIG. 2Adepicts a remote apparatus10andFIG. 2Bdepicts a vehicle apparatus20of an example embodiment that may be embodied by various computing devices including those identified above. As shown, the remote apparatus10of an example embodiment may include, may be associated with or may otherwise be in communication with a processor12and a memory device14and optionally a communication interface16and/or a user interface18. Similarly, a vehicle apparatus20of an example embodiment may include, may be associated with, or may otherwise be in communication with a processor22, and a memory device24, and optionally a communication interface26, a user interface28, one or more location sensors30(e.g., a location sensor such as a GPS sensor; IMU sensors, and/or the like), one or more image capturing devices (e.g., camera(s); 2D and/or 3D LiDAR(s); long, medium, and/or short range RADAR; ultrasonic sensors; electromagnetic sensors; (near-)IR cameras, 3D cameras, 360° cameras and/or the like) and/or other sensors that enable the vehicle apparatus to determine one or more features of the corresponding vehicle's surroundings, and/or other components configured to perform various operations, procedures, functions or the like described herein. In an example embodiment, the vehicle apparatus20may further comprise a pose error network34. For example, the pose error network may be a trained deep net and/or neural network. For example, network weights and/or parameters for the pose error network34may be provided (e.g., transmitted) to the vehicle apparatus20by the remote apparatus10.

In some embodiments, the remote apparatus10and/or vehicle apparatus20may include a user interface18,28that may, in turn, be in communication with the processor12,22to provide output to the user, such as a proposed route, and, in some embodiments, to receive an indication of a user input. As such, the user interface may include a display and, in some embodiments, may also include a keyboard, a mouse, a joystick, a touch screen, touch areas, soft keys, a microphone, a speaker, or other input/output mechanisms. Alternatively or additionally, the processor may comprise user interface circuitry configured to control at least some functions of one or more user interface elements such as a display and, in some embodiments, a speaker, ringer, microphone and/or the like. The processor and/or user interface circuitry comprising the processor may be configured to control one or more functions of one or more user interface elements through computer program instructions (e.g., software and/or firmware) stored on a memory accessible to the processor (e.g., memory device14,24, and/or the like).

In addition to embodying the remote apparatus10and/or vehicle apparatus20of an example embodiment, a navigation system may also include or have access to a geographic database that includes a variety of data (e.g., map information/data) utilized in constructing a route or navigation path and determining the time to traverse the route or navigation path. For example, a geographic database may include node data records (e.g., including anchor node data records comprising junction identifiers), road segment or link data records, point of interest (POI) data records and other data records. More, fewer or different data records can be provided. In one embodiment, the other data records include cartographic (“carto”) data records, routing data, and maneuver data. One or more portions, components, areas, layers, features, text, and/or symbols of the POI or event data can be stored in, linked to, and/or associated with one or more of these data records. For example, one or more portions of the POI, event data, or recorded route information can be matched with respective map or geographic records via position or GPS data associations (such as using known or future map matching or geo-coding techniques), for example. In an example embodiment, the data records (e.g., node data records, link data records, POI data records, and/or other data records) may comprise computer-executable instructions, a reference to a function repository that comprises computer-executable instructions, one or more coefficients and/or parameters to be used in accordance with an algorithm for performing the analysis, one or more response criteria for providing a response indicating a result of the analysis, and/or the like. In at least some example embodiments, the vehicle apparatus20may be configured to execute computer-executable instructions provided by and/or referred to by a data record. In an example embodiment, the remote apparatus10may be configured to modify, update, and/or the like one or more data records of the geographic database.

In an example embodiment, the road segment data records are links or segments, e.g., maneuvers of a maneuver graph, representing roads, streets, or paths, as can be used in the calculated route or recorded route information for determination of one or more personalized routes. The node data records are end points corresponding to the respective links or segments of the road segment data records. The road link data records and the node data records represent a road network, such as used by vehicles, cars, and/or other entities. Alternatively, the geographic database can contain path segment and node data records or other data that represent pedestrian paths or areas in addition to or instead of the vehicle road record data, for example.

The road/link segments and nodes can be associated with attributes, such as geographic coordinates, street names, address ranges, speed limits, turn restrictions at intersections, and other navigation related attributes, as well as POIs, such as gasoline stations, hotels, restaurants, museums, stadiums, offices, automobile dealerships, auto repair shops, buildings, stores, parks, etc. The geographic database can include data about the POIs and their respective locations in the POI data records. The geographic database can also include data about places, such as cities, towns, or other communities, and other geographic features, such as bodies of water, mountain ranges, etc. Such place or feature data can be part of the POI data or can be associated with POIs or POI data records (such as a data point used for displaying or representing a position of a city). In addition, the geographic database can include and/or be associated with event data (e.g., traffic incidents, constructions, scheduled events, unscheduled events, etc.) associated with the POI data records or other records of the geographic database.

In an example embodiment, reference image information/data is stored in association with the map information/data. For example, the geographic database may further comprise a database, library, and/or the like of reference static feature information/data that is stored in association with the map information/data. For example, in some embodiments, reference static feature information/data corresponding to static features located within a particular map tile is stored as part of (e.g., as a layer, associated map information/data, and/or the like) of the particular map tile.

The geographic database can be maintained by the content provider (e.g., a map developer) in association with the services platform. By way of example, the map developer can collect geographic data to generate and enhance the geographic database. There can be different ways used by the map developer to collect data. These ways can include obtaining data from other sources, such as municipalities or respective geographic authorities. In addition, the map developer can employ field personnel to travel by vehicle along roads throughout the geographic region to observe features and/or record information about them, for example. Also, remote sensing, such as aerial or satellite photography, can be used. In an example embodiment, the geographic database may be updated based on information/data provided by one or more vehicle apparatuses. For example, the remote apparatus10may update the geographic database based on a most preferred version map tile as determined from a plurality of responses received from a plurality of vehicle apparatuses20, as described elsewhere herein.

For example, geographic data is compiled (such as into a platform specification format (PSF) format) to organize and/or configure the data for performing navigation-related functions and/or services, such as route calculation, route guidance, map display, speed calculation, distance and travel time functions, and other functions. The navigation-related functions can correspond to vehicle navigation or other types of navigation. The compilation to produce the end user databases can be performed by a party or entity separate from the map developer. For example, a customer of the map developer, such as a navigation device developer or other end user device developer, can perform compilation on a received geographic database in a delivery format to produce one or more compiled navigation databases. Regardless of the manner in which the databases are compiled and maintained, a navigation system that embodies an apparatus10in accordance with an example embodiment may determine the time to traverse a route that includes one or more turns at respective intersections more accurately.

IV. Apparatus, Methods, and Computer Program Products