Patent Description:
Mapping and localization are increasingly important tasks in many technological fields, such as robotics, advanced driver-assistance systems (ADAS) and self-driving systems.

Typically, a defined area/route is first mapped by capturing sensor information about the defined area to create a map. The map is a collection of geometric information and other data depicting the properties of an area of interest (e.g. a route).

For localization purposes, the map is used together with current sensor information to deduce the current location of the vehicle or system within the mapped area. One of the most important types of sensor inputs comes from optical sensors, such as digital cameras.

In current solutions, features are extracted from the images based on the captured image sequence. The extracted features are processed to deduce the scene geometry and scene structure in order to create a map of the defined area, which typically contains image features with their respective three-dimensional (3D) locations, as well as pose nodes which contain information about the location and orientation of the vehicle along its trajectory during the mapping.

Localization may be conducted using the image features (e.g. point or edge features), location data and orientation data contained in the map by matching current sensor readings with the map data. These approaches are typically gradient-based optimization methods such as simultaneous localization and mapping (SLAM).

SLAM technologies exploit point features (e.g. ORB2 SLAM) or line/edge features (e.g. LSD-SLAM) for mapping and localization. This may be an adequate solution scenes with high texture and limited moving objects. However, in scenes with limited texture and/or many or large moving objects, these approaches may lead to incorrect localization results which are difficult to compensate for.

Another problem encountered by these approaches is that the scene must appear similar in both mapping time and localization time. This requirement is not satisfied, for example, when the mapping is done in winter (e.g. snowy scene) but the localization is done in summer. The scene structure may be too different for feature points or edge features to be identified and matched. Furthermore, some applications have very specific conditions for the scenes (e.g. parking places) with repetitive patterns in the scene. Using only local information within the image (such as point and edge features) may make it difficult to have a reliable mapping and localization system.

Other methods which are used for localization are based on unsupervised deep learning for odometry using a Convolutional Neural Network (CNN). These methods do not take into account a pre-mapped sequence and are therefore not directly applicable to high accuracy localization. In one approach, ego-motion, odometry poses and corresponding depth maps are simultaneously estimated from multiple views in an unsupervised manner. Relative poses with respect to a reference pose depend solely on the corresponding images and the corresponding estimated depth maps. In a similar approach, optical flow is also estimated and used to compute relative poses. Only images and corresponding depth maps and optical flow fields are input to the CNN. In yet another approach, the focus is on estimating the scale parameter of a monocular camera trajectory.

PoseNet is a convolutional network for real-time six degrees of freedom (<NUM>-DOF) camera relocalization. PoseNet implements localization by training a CNN to estimate a 6DOF pose using a single image as input. PoseNet has large errors due to the limited amount of information about the scene being provided to the estimator/CNN. The larger the area covered for localization, the larger the localization estimation error will be.

<NPL>, discloses combining KITTI depth and Cityscapes semantic segmentation datasets, on Monocular Depth Estimation.

<NPL>, discloses using depth data to further refine the poses.

It is an object of the present invention to provide an apparatus, a system, a computer program product and a method for robust and accurate localization using deep learning to exploit information provided by images which were captured during the mapping of an area of interest. The present invention is as defined in the appended set of claims.

In order to exploit more image information during localization, the images obtained during the mapping process (or a selected subset of key frames) are stored. Optionally, a necessary amount of content and/or scene overlap is maintained between each consecutive key-frame.

During training, a neural network (such as a deep neural network) is trained to predict pose differences, based on a data set which includes multiple data samples. Each data sample assigns respective poses to a subset of the images that were captured when the area of interest was mapped. Optionally the poses are expressed relative to the pose of a reference image as described below.

During the localization phase, the trained neural network estimates the pose of the localization image, based on the localization image itself and stored images captured during mapping and their respective poses.

The output of the neural network is an estimate of the pose of the localization image. When the pose estimate is relative to a reference image, a transformation may be applied to convert the pose estimate to a map-relative pose.

The types of neural networks which may be trained and used for localization include but are not limited to:.

According to a first aspect of the invention, an apparatus for generating a model for pose estimation of a system includes processor circuitry. The processor circuitry obtains training data for a multiple locations. The training data includes one or more images captured by an image capturing device and the respective poses of the captured images, where a respective pose of an image is the respective location and orientation of the image capturing device during capture of the image. At least one data sample is generated by the processing circuitry from the training data for each of the captured images, where a data sample for an image is an assignment of the image and at least one other image selected from the training data to respective poses. The processor circuitry trains a neural network with a data set made up of the data samples to estimate a respective pose of a localization image from:.

According to a second aspect of the invention, a method for generating a neural network for pose estimation of a system includes obtaining training data for a multiple locations. The training data includes one or more images captured by an image capturing device and the respective poses of the captured images, where a respective pose of an image is the respective location and orientation of the image capturing device during capture of the image. The method further includes generating at least one data sample for each of the captured images from the training data and training the neural network with a data set made up of the data samples, where a data sample for an image is an assignment of the image and at least one other image selected from the training data to respective poses. The neural network is trained to estimate a respective pose of a localization image from:.

The data set may be built automatically and may include images obtained under many conditions, making the trained neural network robust to changes in lighting conditions, to the presence of dynamic objects in the images and to other changes. Additionally, it is possible to use supervised learning to train the neural network, yielding increased performance relative to methods which use unsupervised learning.

With reference to the first and/or second aspects, in a possible implementation generating at least one data sample includes:.

Basing the data sample on a single reference image simplifies the construction of the data sample as well as the processing of the data sample during neural network training.

With reference to the first and/or second aspects, in a possible implementation, for at least one of the data samples, assigning an image and at least one other image selected from the training data to respective poses includes:.

Using multiple reference images increases the amount of information in the data sample and thus the amount of information provided to the neural network.

With reference to the first and/or second aspects, in a possible implementation respective poses of the at least one proximate image are within a specified distance from a pose of at least one of the reference images. Thus the proximate image(s) may be easily selected, as distant images are automatically eliminated from consideration.

With reference to the first and/or second aspects, in a possible implementation an error is introduced into the training data by changing at least one pose in the data samples prior to the training. Deliberately introducing errors into the training data increases the robustness of the trained neural network changes in the area of interest after the mapping is performed.

With reference to the first and/or second aspects, in a possible implementation errors in estimation of the poses are tracked during the training, and to respective confidence values are calculated for poses estimated by the neural network based on the tracked errors. The confidence values increase the amount of information available for training the neural network, thereby improving the training process.

With reference to the first and/or second aspects, in a possible implementation at least one of the data samples further includes a semantic segmentation computed for at least one image mapped in the data set.

With reference to the first and/or second aspects, in a possible implementation at least one of the data samples further includes a depth map computed for at least one image mapped in the data set.

Semantic segmentation and/or depth map information which is included in the data sample serves as additional input to the neural network, thereby relaxing the training data requirements and improving neural network generalization.

With reference to the first and/or second aspects, in a possible implementation the neural network is a deep neural network (DNN). DNNs are able to model complex and non-linear relationships as required for image analysis and recognition, which is beneficial for deriving pose information using actual images as input to the neural network.

According to a third aspect of the invention, an apparatus for image-based localization of a system in an area of interest includes a memory and processor circuitry. The memory is configured to store images captured by an image capturing device of the system at a multiple locations in an area of interest, and respective poses of the image capturing device associated with the stored images at each of the locations, wherein a pose of an image comprises a location and orientation of the capturing device associated with a respective image. The processor circuitry is configured to:.

According to a fourth aspect of the invention, a method for image-based localization of a system in an area of interest includes:.

The neural network is able to provide a highly accurate pose estimate based on the large amount of information available by using the map images themselves as part of the data input to the neural network during the localization process.

With reference to the third and fourth aspects, in a possible implementation an approximate location of the image capturing device during the capturing of the localization image is determined and the subset of stored images is selected based on the approximate location. Using close images may assist with detecting errors in the pose estimates. Since the pose difference between an image in the subset and the localization image should be small, large differences between the predicted pose and the known poses of the images may indicate an error.

With reference to the third and fourth aspects, in a possible implementation the neural network outputs a single estimated pose relative to a pose of a single reference image selected from the retrieved images. Outputting a single pose estimate per localization image minimizes the processing required after the neural network.

With reference to the third and fourth aspects, in a possible implementation the neural network outputs multiple estimated poses, each of the estimated poses being relative to a pose of a different reference image selected from the retrieved images. A final estimated pose is calculated from the multiple estimated poses. Calculating the final estimated pose from multiple intermediate estimates increases the amount of information utilized to obtain the final pose estimate, thereby improving its accuracy.

With reference to the third and fourth aspects, in a possible implementation an estimation accuracy measure is calculated by comparing a pose change between multiple estimated poses for the localization image to at least one pose change between reference images used for calculating the multiple estimated poses. The estimation accuracy measure ensures that inaccurate pose estimates are identified and are not relied on by a user or external system.

With reference to the third and fourth aspects, in a possible implementation the trained neural network is a deep neural network (DNN). DNNs are able to model complex and non-linear relationships as required for image analysis and recognition, which is beneficial for deriving pose information using actual images as input to the neural network.

With reference to the third and fourth aspects, in a possible implementation the trained neural network is further provided with a semantic segmentation computed for at least one of the images provided to the trained neural network.

With reference to the third and fourth aspects, in a possible implementation wherein the trained neural network is further provided with a depth map computed for at least one of the images provided to the trained neural network.

The semantic segmentation and/or depth map information serves as additional input to the neural network, thereby improving the accuracy of the pose estimate.

In the drawings:.

The present invention, in some embodiments thereof, relates to localization of an image in an area of interest, and, more specifically, but not exclusively, to localization of an image in an area of interest using a neural network.

Embodiments of the invention presented herein take a hybrid approach to localization using a trained neural network. Localization is performed by the neural network not only based on data extracted from the images captured during the mapping but also using the images themselves. The images captured during the mapping process are used to train the neural network and also as input to the trained neural network during localization.

Using stored map images as input to the neural network during localization increases the amount of image information available to the neural network, resulting in highly accurate and reliable localization estimates.

As used herein, the term "pose of an image" means the location and orientation of the image capturing device when it captured the image (i.e. at the time of image capture). The pose may be expressed as three location coordinates and three angular coordinates, which indicate the location and orientation associated with an image. It will be appreciated that the pose is relative to a coordinate system and that a pose may be transformed from one coordinate system to another by a linear and/or angular transformation.

As used herein, the term "localization image" means an image whose pose is being estimated by the neural network.

As used herein the term "map-relative pose" means a pose expressed in the map coordinate system (also denoted a ground truth pose).

As used herein the term "pose relative to a reference image" means a pose expressed in the coordinate system of a reference image.

Reference is now made to <FIG>, which illustrates mapping and localization using a neural network, according to an exemplary embodiment of the invention.

During mapping, a sequence of images is captured at multiple locations in an area of interest (in this case the route of a vehicle). The map includes the captured images and their map-relative poses (i.e. ground truth poses).

Some exemplary embodiments are described herein in a non-limiting manner for a map represented as a pose graph (denoted the mapping sequence). The pose graph includes interconnected pose nodes which define one or multiple chains of poses (e.g. follow one or multiple trajectories of a vehicle). It is to be understood that corresponding embodiments may be implemented for other representations of the map used to train the neural network and/or of the image and pose information used as input to the neural network during localization.

Optionally, the neural network is trained to estimate the pose of the localization image relative to the coordinate system of a selected reference node <NUM>. The ground truth pose estimate for the localization image is calculated from the ground-truth pose of reference node <NUM> (which is known from the mapping) and the estimated pose output by the neural network.

As used herein the terms "pose node" and "node" mean an element having an associated image and pose, and information about connections to other pose nodes.

A pose node may include additional information, including but not limited to:.

Optionally, the neural network is further trained to detect when there is a high probability that the localization image is from a location outside the mapped area of interest, in which case it may not be possible to perform accurate localization based on the localization image.

During the mapping process, images are captured by an image capturing device over an area of interest (e.g. a route). A mapping sequence is created from the captured images and their respective poses. In one example, a vehicle carrying an image capturing device and location and orientation sensors travels along a route. The image capturing device captures images at a fixed rate and the pose at the time each image is captured.

As used herein, the term "image capturing device" means any type of device capable of capturing an image, without limitation as to how the image is captured or to the hardware or type of the image capturing device. Examples of an image capturing device include, but are not limited to, a camera and an image sensor.

One or more images may be captured at each location, possibly at different times and/or at different orientations (for example stereo images). The types of images which may be captured, include but are not limited to RGB images, chrominance-luminance images and infra-red images.

Optionally, the map is updated during the localization stage in order to maintain accuracy.

Reference is now made to <FIG>, which is a simplified block diagram of an apparatus for generating a model for pose estimation of a system, according to embodiments of the invention. Apparatus <NUM> includes processor circuitry <NUM>. Embodiments of apparatus <NUM> may further include one or more of:.

Processor circuitry <NUM> performs processing operations for training the neural network.

Apparatus <NUM> obtains training data which was gathered at multiple locations during the mapping process. The training data includes least one image captured at each location and respective poses of the captured images.

Optionally, the apparatus includes at least one memory <NUM>, and some or all of the captured images and their respective poses are stored in memory <NUM> for use as training data. Alternately or additionally, the apparatus includes communication interface <NUM>, and some or all of the training data is obtained from an external element (e.g. an external memory, server, over a network, etc.).

Apparatus <NUM> generates data samples from the training data. Each data sample assigns images selected from the training data to their respective poses. Examples of generating data samples are illustrated in <FIG> and are described in more detail below.

The data samples are combined into a data set which is used to train the neural network. The neural network is trained to identify the pose of a localization image from:.

Reference is now made to <FIG>, which is a simplified flowchart of a method for generating a neural network for pose estimation of a system, according to embodiments of the invention.

In <NUM>, the training data is obtained.

In <NUM>, at least one data sample is generated for each of the images. A data sample for an image is an assignment of the image and at least one other image selected from said training data to respective poses.

In <NUM>, a data set for training a neural network is formed from multiple data samples.

In <NUM>, the neural network is trained with the data set to estimate a pose of a localization image from:.

In some embodiments of the invention, a data sample is generated by selecting at least one reference image from the training data. At least one proximate image is also selected from the training data.

The proximate image(s) have respective pose(s) which are proximate to the pose of at least one reference image. In some embodiments, the proximate images are images that were captured at a close distance from the location at which a reference image was captured. The proximate nodes are not necessarily consecutive within the mapping sequence nor are they necessarily the closest nodes to the reference node(s). Optionally, the proximate nodes are located within a specified distance from one or more reference nodes.

The data sample is created by assigning respective poses to each proximate image relative to each reference image.

Optionally, the data sample further includes semantic segmentation (SS) information for at least one of its images. The semantic segmentation information classifies pixels in the image as to the type of object it is depicting (e.g., pixel of a car, pixel of a vegetation, building, sky, road.

Optionally, the data sample further includes depth map information for at least one of its images. The depth map (DM) provides respective depth information for a scene point in the image.

The data samples which form the data set used for training are not necessarily of the same structure or type. Possible differences between data samples include but are not limited to:.

One benefit of creating the training data set according to embodiments of the invention is that there is no need for manual labelling. This allows for fully-automatic training data set generation.

In the non-limiting embodiments described below, the pose node of a reference image is denoted a reference node and the pose node of a proximate image is denoted a proximate node.

Optionally, at least one data sample has a single reference node and multiple proximate nodes. In the non-limiting examples illustrated in <FIG> and <FIG>, the reference nodes are shown as part of the first mapping loop and the proximate nodes are shown as part of the second mapping loop. It is noted that the reference and proximate nodes are not necessarily selected from different mapping loops but rather may be in any order or combination. Furthermore, the mapping sequence is not necessarily defined as two mapping loops.

Reference is now made to <FIG> which illustrates the creation of a data sample having a single reference node M and proximate nodes (G1, G2. ), according to embodiments of the invention. Reference node M is selected as the origin coordinate system.

In a first example the data sample is the assignment: <MAT> where I(X) means the image at the pose node X and C1, C2,. are the image-relative poses expressed with respect to node M.

In a second example semantic segmentation data and/or depth mapping data is available for one or more nodes and the data sample is: <MAT> where SS(X) means the semantic segmentation information for the image at pose node X and DM(X) means the depth map information for the image at pose node X.

Reference is now made to <FIG>, which is a simplified flowchart of a method for creating a data sample having a single reference node and multiple proximate nodes, according to embodiments of the invention.

In <NUM>, proximate nodes G1, G2. are selected.

In <NUM>, a data sample is built for M, G1, G2. by assigning the respective poses. In the example of <FIG>, the data sample does not include SS and DM information. Other types of data samples may be built for the same images with SS and/or DM information and/or additional information as described below.

Optionally, at least one data sample has multiple reference nodes (denoted Mk+n) and multiple proximate nodes (denoted Gj). Respective poses are designated for each proximate image relative to each of the reference images. By including the additional information provided by the multiple reference nodes, the available information is better exploited when training the neural network.

Optionally, the pose information of all the reference nodes Mk+n is included as information in the data sample. This provides the benefit of wider coverage of the scene information provided by the proximate nodes.

Reference is now made to <FIG> which illustrates the creation of a data sample having multiple reference nodes, Mk+n, and multiple proximate nodes, Gj, according to embodiments of the invention.

In a first example, the data sample is the assignment: <MAT> where I(Mk+n) means the images at reference nodes Xk. Xk+n, I(Gj) means the images at proximate nodes Gj. M'k+n are the poses of the reference nodes relative to a reference node which is selected as the origin coordinate system (in this example Mk), where: <MAT> and Ck,j are the poses of each proximate node relative to each reference node as expressed in the origin coordinate system.

In a second example semantic segmentation data and/or depth mapping data is available for one or more nodes and the data sample is: <MAT>.

Reference is now made to <FIG>, which is a simplified flowchart of a method for creating a data sample having multiple reference nodes and multiple proximate nodes, according to embodiments of the invention.

In <NUM>, multiple reference nodes, Mk+n, are selected and one of the reference nodes, Mk, is selected as the origin of the coordinate system.

In <NUM>, proximate nodes Gj are selected.

In <NUM>, the poses of the reference nodes are transformed to the Mk coordinate system.

In <NUM>, a data sample is built for Mk+n and Gj by assigning the respective poses Ck,j. In the example of <FIG>, the data sample includes SS and DM information, and the data sample is (I(Mk+n), l(Gj), M'k+n,. ,[SS(Mk+n), SS(Gj),. ,DM(Mk+n)]) → (Ck,j). Other data samples may be built for the same images without SS and/or DM information.

Optionally, pose estimation errors are tracked while the neural network is being trained. The errors are used to calculate confidence values, w, for pose estimates output by the neural network. Further optionally, the confidence values are used as additional input for training the neural network.

In a first example, the data sample is generated for a single reference node and multiple proximate nodes with no SS or DM information (as in <FIG>). The data sample may be expressed as: <MAT> where wx is the confidence value for pose CX.

Reference is now made to <FIG>, which is a simplified flowchart of a method for creating a data sample with confidence values, according to an exemplary embodiment of the invention. In the embodiment of <FIG>, the data sample is for multiple reference nodes and multiple proximate nodes with SS and DM information, according to embodiments of the invention.

In <NUM>, a data sample is built for Mk+n and Gj by assigning the respective poses Ck,j. In the example of <FIG>, the data sample includes SS and DM information, and the data sample is (I(Mk+n), I(Gj), M'k+n,. ,[SS(Mk+n), SS(Gj),. ,DM(Mk+n)]) → (Ckj, wk,j). Other data samples may be built for the same images without SS and/or DM information.

Optionally, one or more errors are introduced into the training data by changing at least one pose in at least one data sample. For example, minor angular and/or location may be added to the correct pose parameters. Adding error(s) to a pose simulates mapping inaccuracies, so that the neural network may be trained to maintain high accuracy even when the mapping used during localization is not accurate. Note that the accurate poses Ck,j are available during training, so that the neural network may be trained to estimate the poses correctly.

Reference is now made to <FIG>, which is a simplified flowchart of a method for introducing error into a data sample, according to an exemplary embodiment of the invention. In the embodiment of <FIG>, the data sample is for multiple reference nodes and multiple proximate nodes with SS and DM information, according to embodiments of the invention.

In <NUM>, the poses of the reference nodes are transformed with error ε to the Mk coordinate system, <MAT>.

In <NUM>, a data sample is built for M"k+n and Gj by assigning the respective poses Ck,j. In the example of <FIG>, the data sample includes SS and DM information, and the data sample is (I(Mk+n), I(Gj), M"k+n,. ,[SS(Mk+n), SS(Gj),. ,DM(Mk+n)]) → (Ck,j). Other data samples may be built for the same images without SS and/or DM information and/or confidence values.

In an additional example, the SS and DM information is not available, confidence values are available and the data sample is: <MAT>.

Reference is now made to <FIG>, which is a simplified block diagram of an apparatus for image-based localization of a system in an area of interest, according to embodiments of the invention. Localization is performed using a neural network which has been trained to perform pose estimation for a localization image using stored images that were captured during the mapping (denoted map images) and their respective poses.

Optionally, the trained neural network is one of:.

Apparatus <NUM> includes processor circuitry <NUM> and memory <NUM>. Optionally, apparatus <NUM> further includes one or more of:.

Processor circuitry <NUM> performs processing operations for performing the localization.

Memory <NUM> stores images that were captured at multiple locations in an area of interest during mapping and the respective poses of the stored images. As described above, the pose of an image is the location and orientation of the image capturing device which captured the stored image at the time of image capture.

Apparatus <NUM> obtains a localization image that was captured by an image capturing device. Optionally, apparatus <NUM> includes image capturing device <NUM> which captures the localization images. Alternately, the localization image is obtained from an external image capturing device.

To perform localization, a subset of the stored map images is retrieved from memory <NUM>. Information is provided to the trained neural network in order to obtain a pose estimate for the localization image. The information includes:.

Based on this input, the neural network outputs at least one pose estimate for the localization image (denoted a localization pose estimate).

Optionally, further input to the trained neural network includes: semantic segmentation information computed for at least one of the images and/or depth map(s) computed for at least one of the images.

Optionally, the subset of stored images retrieved from memory <NUM> is selected based on the approximate location of the system being localized. Further optionally, the subset includes some or all of the stored images that were captured within a certain distance from the approximate location of the system.

The approximate location of the system being localized may be determined, for example, using a simple GPS device which provides a low accuracy estimate of the location and/or by place recognition and/or based on previous localization pose estimates.

Optionally, one or more of the retrieved images is selected as a reference image.

The output of the neural network is at least one localization pose estimate, which is expressed relative to a reference node coordinate system and/or relative to the map coordinate system.

Optionally, the neural network outputs a single localization pose estimate which is expressed relative to the pose of a single reference image.

Alternately or additionally, the neural network outputs multiple localization pose estimates. Each localization pose estimate is expressed relative to the pose of a different reference image. A final localization pose estimate is calculated from the multiple pose estimates.

Optionally, the neural network is trained to jointly estimate poses for multiple localization images, for example consecutive localization images. Joint estimation of multiple localization poses reduces error, as the neural network may be trained to ensure that vehicle dynamics constraints are satisfied (e.g. maximum vehicle speed is not exceeded in consecutive pose estimates).

Reference is now made to <FIG>, which is a simplified flowchart of a method for image-based localization of a system in an area of interest, according to embodiments of the invention.

In <NUM>, the localization image is obtained from an image capturing device.

In <NUM>, input data is provided to a trained neural network. The input data includes:.

In <NUM>, at least one pose estimate is obtained from the trained neural network.

Reference is now made to <FIG>, which illustrates pose estimation of a localization pose, Lj, relative to a single reference node Mk, according to an exemplary embodiment of the invention.

In order to estimate localization pose Lj, respective images and poses of multiple mapped pose nodes, Mk+n, are retrieved from memory. One of the pose nodes, in this example Mk, is selected as a reference node. All poses are expressed relative to the Mk coordinate system, thereby bounding the pose estimate values within a limited range.

Optionally, reference node Mk is the pose node closest to the approximate location of the system. Further optionally, the pose node closest to the localization pose is found by computing the relative poses of Mk+n with respect to the localization image, and choosing the n having the smallest distance from the localization pose.

Optionally, the other pose nodes, Mk-<NUM>, Mk+<NUM> and Mk+<NUM>, are proximate to Mk, for example within a specified distance from Mk.

Optionally, multiple localization poses (. , Lj-<NUM>, Lj-<NUM>, Lj-1i, Lj. ) are estimated jointly, where Ck,j is the pose estimate for localization image I(Lj) relative to reference node Mk.

Reference is now made to <FIG>, which shows the inputs and outputs of the trained neural network in accordance with the exemplary embodiment of <FIG>. Inputs to neural network DNN include:.

The output of neural network DNN is the localization pose estimate Ck,j in the Mk coordinate system, which is transformed into the map-relative pose Lj.

Reference is now made to <FIG>, which illustrates pose estimation relative to multiple reference nodes, according to an exemplary embodiment of the invention. Using multiple reference nodes reduces estimation error by providing more information to the neural network.

In order to estimate localization pose Lj, respective images and poses of multiple reference nodes, Mk+m, are selected. Images and pose information for reference nodes Mk+m are retrieved from memory. The neural network calculates multiple pose estimates for each localization image, where each pose estimate is relative to a different reference node. The multiple pose estimates are combined to form the final pose estimate Lj. For example Lj may be computed as the weighted average of the multiple pose estimates, where the weights are the confidence values wk,j.

Optionally, multiple localization poses,. , Lj-<NUM>, Lj-<NUM>, Lj-<NUM>, Lj, are estimated jointly, where Ck,j is the pose estimate for localization image I(Lj) relative to reference node Mk.

The outputs of neural network DNN are multiple pose estimates (Ck+i,j) for each localization image, I(Lj), in the Mk+m coordinate system. A weighted average is calculated for the multiple pose estimates to obtain a single map-relative localization pose estimate: Lj = Σm ωk+m,jMk+mCk+m,j, where Σm ωk+m,j = <NUM>.

Optionally, an estimation accuracy measure is provided for a pose estimate. The estimation accuracy measure is based on a comparison of the difference between ground truth pose estimates obtained for the same localization image for different reference nodes to the actual ground truth pose difference for those reference nodes (which is known from the map).

Reference is now made to <FIG>, which illustrates how different estimates are obtained for a single localization pose, Lj. Ck,j is the pose estimate for localization image I(Lj) when Mk is used as the reference node, and Ck+<NUM>,j is the pose estimate for localization image I(Lj) when Mk+<NUM> is used as the reference node.

Reference is now made to <FIG>, which is a simplified flowchart of a method for determining an estimation accuracy measure for a pose estimate, according to embodiments of the invention.

In <NUM>, the change in the ground truth pose of nodes Mk and Mk+<NUM> is calculated as: <MAT>.

In <NUM>, the change in the ground truth pose estimates for Lj is calculated as: <MAT>.

In <NUM>, the difference is calculated as: E = |Sk,k+<NUM> - S'k,k+<NUM>|.

In <NUM>, a confidence value V is calculated from the difference E, where V is a measure of the accuracy of the pose estimate.

Claim 1:
An apparatus (<NUM>) for generating a model for pose estimation of a system, the apparatus comprising a processor circuitry (<NUM>), the processor circuitry being configured to:
obtain training data for a plurality of locations, said training data comprising at least one image captured by an image capturing device (<NUM>), and respective poses of said at least one image captured by said image capturing device, wherein a respective pose of an image comprises a respective location and orientation of the image capturing device during capture of the image;
generate from said training data at least one data sample for each of said captured images, wherein a data sample for an image comprises an assignment of said image and at least one other image selected from said training data to respective poses;
train a neural network with a data set comprising said data samples to estimate a respective pose of a localization image from:
said localization image;
at least one additional image from said training data; and
a respective pose of each of said at least one additional image; wherein said generating at least one data sample comprises:
selecting at least one reference image from said training data;
selecting at least one proximate image having a pose proximate to a pose of said at least one reference image from said training data; and
assigning said at least one reference image and said at least one proximate image to respective poses of each proximate image relative to each reference image;
wherein a pose node of said at least one reference image is denoted a reference node and a pose node of said at least one proximate image is denoted a proximate node;
wherein the proximate nodes are located within a specified distance from one or more reference nodes.