MAP PROCESSING DEVICE AND METHOD THEREOF

A map processing device obtains images from a camera transported with a mobile device, and detects 2D features in the images. At least some of the 2D features detected in the images are densified to generate a densified set of 2D features. Locations of the densified set of 2D features are determined in 3D space using an image-based localization algorithm. A structure-based map comprising depth information from a depth sensor transported by another mobile device is accessed, wherein the depth information includes a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor. The camera transported with the mobile device is localized to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information accessed in the structure-based map.

TECHNICAL FIELD

The present disclosure relates to a map processing device, a method by a map processing device, and a corresponding computer program product.

BACKGROUND

An important research topic in localization and mapping, also known as Simultaneous Localization and Mapping (SLAM), is performing SLAM with heterogeneous sensor information. It is anticipated that electronic devices will increasingly use heterogeneous sets of sensors to localize device locations relative to maps of the real-world. For example, a basic smartphone performing an augmented reality application may use only a monocular camera for localization. A more advanced smartphone (e.g., Apple iPhone 12 Pro and Ipad 12 Pro) may use a combination of a monocular camera and a Lidar for localization. Still other advanced devices, such as factory robots or mixed reality headsets, may use multiple monocular cameras or stereo cameras and a Lidar for localization.

SLAM algorithms can typically be split into image-based algorithms and structure-based algorithms. The image-based algorithms are configured for localization of devices using images from cameras, which are also referred to as image sensors. In contrast, structure-based algorithms are configured for localization of devices using depth information from a depth sensor which actively senses distance to a real-world feature, such as by bouncing a laser (e.g., Lidar), RF signal (e.g., radar), sound (e.g., ultrasonic sensor), etc. on the feature.

Because of the diverse types of sensors which are being used by devices for localization, situations arise where a device that needs to be localized using an image-based SLAM algorithm does not have access to an existing image-based map for the present device location, although an existing structure-based map does exist for use by structure-based SLAM algorithms. A problem with current SLAM algorithms is that they only work with a single type of sensor, e.g. image-based or structure-based. Recently there have been proposed new solutions to localize a device with an image-based sensor (e.g., monocular or stereo camera) in a map created by a structure-based sensor (e.g., Lidar).

One existing approach for addressing this problem is described in a paper by T. Caselitz, B. Steder, M. Ruhnke, and W. Burgard, “Monocular camera localization in 3D LiDAR maps,” 2016IEEE/RSI International Conference on Intelligent Robots and Systems(IROS), Daejeon, Korea (South), pp. 1926-1931, doi: 10.1109/IROS.2016.7759304, IEEE, 2016. This paper proposes to extract the two-dimensional (2D) features from the current image, compute a set of data points indicating locations in 3D space corresponding to the 2D features in the real-world using SLAM, and compute a transformation between the set of 3D data points and a set of data points in a Lidar map, assuming that the location of the device is previously known within the Lidar map up to a small error. This simplifies the problem since the search for similarities between the two sets is constrained to a small range where the authors assume that a correspondence has to be found.

Another approach is described in a paper by D. Cattaneo, M. Vaghi, A. L. Ballardini, S. Fontana, D. G. Sorrenti, and W. Burgard, “CMRNet: Camera to LiDAR-Map Registration,” 2019IEEE Intelligent Transportation Systems Conference(ITSC), Auckland, New Zealand, pp. 1283-1289, doi: 10.1109/ITSC.2019.8917470, IEEE 2019. This paper proposes a full black box, machine learning (convolutional neural network (CNN)) based approach to convert an image directly into a depth image, and then match such depth image with a Lidar map. Again, the authors assume that the position of the monocular camera device is known up to a value (e.g. four meters in their experiments) with respect to the lidar map which makes the matching of the depth image generated from the monocular image and the Lidar map information simpler to obtain.

Another approach is described in a paper by Y. Lu, J. Lee, S.-H. Yeh, H.-M. Cheng, B. Chen, and D. Song, “Sharing Heterogeneous Spatial Knowledge: Map Fusion Between Asynchronous Monocular Vision and Lidar or Other Prior Inputs”, inRobotics Research. Springer Proceedings in Advanced Robotics, vol 10, pp. 727-741, Springer, 2020. This paper computes planes in 3D space from a set of data points indicating locations in 3D space corresponding to 2D features identified from the monocular images, and also from the Lidar map information. Then a matching process is performed between the planes, which allows finding a correspondence between two heterogeneous sensor types. However, planes may not always be available in all types of environments limiting the applicability of this solution.

Hence, there is a need to develop more adaptable SLAM algorithms and systems.

SUMMARY

Some embodiments disclosed herein are directed to a map processing device that includes at least one processor. The at least one processor is configured to obtain images from a camera transported with a mobile device, and detect 2D features in the images. The at least one processor performs densification of at least some of the 2D features detected in the images to generate a densified set of 2D features, and determines locations of the densified set of 2D features in 3D space using an image-based localization algorithm. The at least one processor accesses a structure-based map comprising depth information from a depth sensor transported by another mobile device, wherein the depth information comprises a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor. The at least one processor localizes the camera transported with the mobile device to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information accessed in the structure-based map.

Some other related embodiments are directed to a method by a map processing device. The method includes obtaining images from a camera transported with a mobile device, and detecting 2D features in the images. The method performs densification of at least some of the 2D features detected in the images to generate a densified set of 2D features, and determines locations of the densified set of 2D features in 3D space using an image-based localization algorithm. The method accesses a structure-based map comprising depth information from a depth sensor transported by another mobile device, wherein the depth information comprises a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor. The method localizes the camera transported with the mobile device to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information accessed in the structure-based map.

Some other related embodiments are directed to a computer program product including a non-transitory computer readable medium storing program code executable by at least one processor of a map processing device to perform operations. The operations include obtaining images from a camera transported with a mobile device, and detecting 2D features in the images. The operations perform densification of at least some of the 2D features detected in the images to generate a densified set of 2D features, and determine locations of the densified set of 2D features in 3D space using an image-based localization algorithm. The operations access a structure-based map comprising depth information from a depth sensor transported by another mobile device, wherein the depth information comprises a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor. The operations localize the camera transported with the mobile device to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information accessed in the structure-based map.

As will be explained in further detail below, a potential advantage provided by these and other embodiments is that an image-based localization algorithm can be used to process images from a camera transported with a mobile device to localize the camera and, thereby, the mobile device relative to a structure-based map. Therefore, a conventional image-based localization algorithm (e.g., image-based SLAM algorithm) can be used to process camera images for localization relative to a structure-based map. Moreover, by densifying at least some of the 2D features which are detected in the images before determining locations of the 2D features in 3D space using the image-based localization algorithm, the localization can be performed more accurately and computationally efficiently then if a non-densified set of 2D features were attempted to be used for localization relative to the typically higher density depth information contained in the structure-based map.

Other devices, methods, and computer program products according to embodiments will be or become apparent to one with skill in the art upon review of the following drawings and detailed description. It is intended that all such devices, methods, and computer program products be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims. Moreover, it is intended that all embodiments disclosed herein can be implemented separately or combined in any way and/or combination.

DETAILED DESCRIPTION

Embodiments of the present disclosure are directed to performing localization of a mobile device with a camera using an image-based localization algorithm to a structure-based map which can be created using a structure-based sensor. The term “camera” used herein refers to any type of image-based sensor, including without limitation a monocular camera, stereo cameras, etc. As will be explained in more detail below, the localization can be performed with increased accuracy using densification operations according to some embodiments, and can be performed in a computationally efficient manner which may enable online localization of mobile devices.

Some embodiments are now explained in the context ofFIG.1and an example operational scenario.FIG.1illustrates a map processing device100that performs mapping (map creation) and may further perform localization for a first mobile device110using, e.g., a structure-based localization algorithm such as structure-based SLAM. The map processing device100can perform localization and may further perform mapping for a second mobile device120using, e.g., an image-based localization algorithm such as image-based SLAM. Communications between the mobile devices110and120and the map processing device100can be performed through one or more radio access network(s)130and one or more public, e.g., Internet, and/or private networks132.

The first mobile device110includes a depth sensor112, at least one processor114(“processor”), at least one memory116(“memory”) storing program code executable by the processor114, and at least one wireless transceiver118(“wireless transceiver”) to communicate with the radio access network130. The depth sensor112actively senses distance to a real-world feature, such as by bouncing a laser (e.g., Lidar), RF signal (e.g., radar), sound (e.g., ultrasonic sensor), etc. on the real-world feature. The depth sensor112generates depth information which includes a set of data points indicating locations in 3D space corresponding to features in the real-world which are sensed by the depth sensor112. The depth information is communicated by the wireless transceiver118to the map processing device100. The first mobile device110may optionally include an inertial measurement unit (IMU) or other sensor (e.g., wheel odometer) that senses movement for generating pose data indicating poses of the depth sensor112when the depth information as captured and/or indicating transformations between the poses of the depth sensor112when the depth information was captured.

In the example ofFIG.1, the map processing device100includes at least one processor109(“processor”) and at least one memory108(“memory”) storing program code executable by the processor109. The program code can include a localization and mapping module101which updates depth information104in a structure-based map102based on the depth information received from the first mobile device110.

Alternatively or additionally, operations of the localization and mapping module101can be performed by the first mobile device110with the map processing device100sending a partial map to the first mobile device110for processing to perform localization and mapping operations for the first mobile device110. Still alternatively or additionally, the first mobile device110may locally store at least a part of a structure-based map for locally performing localization and mapping operations. Accordingly, the localization and mapping operations can be centrally performed by, e.g., a computing server, performed by distributed mobile devices, or performed by a combination of centrally and distributed.

The localization and mapping module101may be further configured to update an image-based map105based on the images received from the second mobile device120. For example, the localization and mapping module101may be configured to update features106that are detected in images received from a camera122of the second mobile device122and may further update pose data107that is determined from inertial measurement unit (IMU) data received from an IMU123of the second mobile device122. The image-based map105is optional because the localization and mapping module101may be configured to perform image-based localization operations using images from the second mobile device120without keeping (maintaining a programmed association to) the features determined in the images for later use.

The first mobile device110may be, for example, a robotic vacuum cleaner or a factory automated guided vehicle (AGV) that travels through a building while providing depth information to the map processing device100. When the first mobile device110is navigating through previously traversed portions of a building the map processing device100can process the received depth information through a structure-based SLAM algorithm, e.g., in a structure-based part of the localization and mapping module101, to localize the first mobile device110relative to the depth information104in the structure-based map102. The localization operations may enable the first mobile device110to autonomously navigate through the building. Moreover, when the first mobile device110is navigating through a newly traversed portion of the building, the map processing device100can use the received depth information to augment the depth information104stored in the structure-based map102for later use to localize the first mobile device110while navigating through that portion of the building.

In contrast to the first mobile device110, the second mobile device120does not have a depth sensor. Instead, the second mobile device120senses features of the real-world using a camera122that captures images as the second mobile device120moves. The second mobile device120includes the camera122, at least one processor124(“processor”), at least one memory126(“memory”) storing program code executable by the processor124, and at least one wireless transceiver128(“wireless transceiver”) to communicate through the radio access network130with the map processing device100. The second mobile device120provides the images to the map processing device100. In the example ofFIG.1, the map processing device100uses an image-based localization algorithm, e.g., image-based SLAM algorithm, to perform localization of the camera122and thereby localization of the second mobile device120to the real-world using the structure-based map102as explained in further detail below. The images may correspond to what are called keyframes, or keyframe images, in SLAM literature. The map processing device100may optionally store detected features from the images as feature data106in the optional image-based map105.

The second mobile device120may optionally include an IMU123or other sensor (e.g., wheel odometer) that senses movement for generating pose data indicating poses of the camera122when the images were captured and/or indicating transformations between the poses of the camera122when the images were captured. The pose data may be: 1) absolute pose data from which pose transformation data can be derived; and/or 2) relative pose data. The pose of an image may be defined by the six degree of freedom position and orientation of the camera122relative to a defined reference frame when the image was captured. The pose data can be provided to the map processing device100localization and may optionally be stored as pose data107associated with the features106in the image-based map105.

As used herein, the term “pose” refers to the position and/or the rotational angle of a mobile device relative to a defined coordinate system or another mobile device. A pose may therefore be defined based on only the multidimensional position of one mobile device relative to a defined coordinate system or another mobile device, based only on the multidimensional rotational angles of the mobile device relative to a defined coordinate system or another mobile device, or on a combination of the multidimensional position and the multidimensional rotational angles. The term “pose” therefore is used to refer to position, rotational angle, or combination thereof.

The second mobile device120may optionally include in local memory a localization and mapping module101′ that can perform at least some of the operations described herein for the localization and mapping module101in the map processing device100. The second mobile device120may include in local memory at least a portion of a structure-based map102′ that can comprise depth information104such as that described for the structure-based map102in the map processing device100. For example, the map processing device100may provide proximately located portions of the depth information from the structure-based map102in the map processing device100to the structure-based map102′ in the second mobile device120for local processing to localize the second mobile device120relative to the structure-based map102′.

The structure-based map102and/or the image-based map105may be organized as data structures that can be represented as graph structures as is commonly performed in structure-based and image-based SLAM approaches. A graph structure is defined by vertices and edges, where vertices contain the main map information, example of which may include, without limitation: for structure-based SLAM—depth information, point cloud or segment descriptors and their respective poses; or for image-based SLAM—keyframes, 2D features and their respective poses. Edges contain the geometric transformation required to be applied to go from edges connecting vertices which contain overlapping information relating the same location in an environment. For example, a set of features in vertex1are the same features observed in vertex100, so there should be an edge between them. Edges contain the geometric transformation required to be applied to traverse from one vertex to adjacent vertices.

Continuing with the above example operational scenario, the second mobile device120may be a mixed reality headset worn by a person who is traveling through the same building as the first mobile device110. As the second mobile device120travels through the building, the localization and mapping module101may augment the features106and possibly also the pose data107in the image-based map105. The image-based map105can then be processed through an image-based localization algorithm of the localization and mapping module101to localize the second mobile device120within the building.

Device localization can typically be performed more accurately using depth sensors and associated structure-based SLAM algorithms than using cameras and associated image-based SLAM algorithms. However, depth sensors can have a much higher cost and operationally consume more power than cameras. Thus, although the second mobile device120may have a lower cost and lower power consumption than the first mobile device110, the second mobile device120may be capable of less accurate localization using an image-based SLAM algorithm processing features106of the image-based map105. It would be advantageous for the second mobile device120to be localized using the depth information104of the structure-based map102.

However, the image-based localization algorithm is not configured to be able to directly process the structure-based map102to localize the second mobile device120, or may not be able to computationally efficiently and accurately process the structure-based map102to localize the second mobile device120. Thus, when the first mobile device110has assisted the map processing device100with mapping a portion of the building updated to the structure-based map102, the image-based localization algorithm may not be able to directly use the structure-based map102to localize the second mobile device120within that portion of the building.

Some embodiments of the present disclosure are directed to enabling an image-based localization algorithm to localize the second mobile device120based on content of the structure-based map102.

FIG.2illustrates a flowchart of operations performed by the map processing device100in accordance with some embodiments of the present disclosure. Referring toFIGS.1and2, the map processing device100is configured to obtain 200 images from the camera122transported with the second mobile device120. The map processing device100detects2022D features and images, and performs 204 densification of image pixels neighboring at least some of the 2D features detected in the images to generate a densified set of 2D features. The map processing device100determines206locations of the densified set of 2D features in 3D space using an image-based localization algorithm. The map processing device100accesses208the structure-based map102comprising depth information104from the depth sensor112transported by the first mobile device110, wherein the depth information comprises a set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor112. The map processing device100then localizes210the camera122transported with the second mobile device120to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information104accessed in the structure-based map102. Localization of the camera122enables localization of the second mobile device120to the real-world by use of a geometric transformation of a pose of the camera122relative to the second mobile device120, such as when the camera122is spaced apart with a fixed geometric relationship to the second mobile device120.

The operation to match the locations of the densified set of 2D features in 3D space to the depth information104accessed in the structure-based map102, may include any operation configured to match two sets of information. Example operations can include point set registration or scan matching operations. In computer vision, pattern recognition, and robotics, point set registration, also known as point cloud registration or scan matching, can include the process of finding a spatial transformation (e.g., scaling, rotation and translation) that aligns two sets of information, such as two sets of point clouds. For 2D point set registration used in image processing and feature-based image registration, a point set may be 2D pixel coordinates obtained by feature extraction from an image, for example corner detection.

Operations for detecting202the 2D features in the images, determining206, locations of the densified set of 2D features in 3D space, and then localizing210the camera based thereon may be performed using any type of image-based localization algorithm, such as an image-based SLAM algorithm described in a paper by C. Campos, R. Elvira, J. Rodriguez, J. Montiel, and J. Tardos, “Orb-slam3: An accurate open-source library for visual, visual-inertial and multi-map slam”, https://arxiv.org/pdf/2007.11898.pdf, 2020.FIG.8illustrates an example camera image from a KITTI dataset and the detected 2D features which are illustrated by boxes overlaid on the image.

The operations for detecting202the 2D features in the images can create a relatively very sparse depth image of the real-world. For example,FIG.9Aillustrates the example camera image ofFIG.8, andFIG.9Billustrates the 2D features points having associated depth values which have been detected in the camera image ofFIG.9A, in accordance with some embodiments of the present disclosure.

In some embodiments, the operations for determining206the locations of the 2D features in 3D space are limited to being performed on keyframe images instead of, for example, on each image captured by the camera122. A keyframe image, herein also referred to as keyframe, may correspond to what is defined in various SLAM literature as being an image out of a set of consecutive images which contains an amount of 2D features above a desired threshold and also where the detected 2D features are robust and accurate (i.e. the detectability of the features across multiple consecutive images is consistent and the position of the 2D feature in 3D space has a low variance). Limiting the operations for determining206the locations of the 2D features in 3D space can result in the operations being more accurate, robust, and computationally efficient, since they are applied on accurate, robust features from “keyframes” instead of every image.

FIG.3illustrates a flowchart of corresponding operations performed by the map processing device100for limiting the operations for determining206the locations of the 2D features in 3D space in accordance with some embodiments of the present disclosure.

Referring toFIG.3, the operations identify300keyframe images among the images from the camera122, and limit302the determination of the locations of the densified set of 2D features in 3D space to being performed on the keyframe images. In some further embodiments, the identification of a keyframe image is based on at least one of: 1) determining an image has a number of 2D features that is greater than a threshold number, 2) determining an image contains less than a threshold fraction of a number of 2D features which were detected in a preceding keyframe image, 3) determining an image was captured a threshold number of images after a preceding keyframe image was captured, and 4) a camera pose when capturing an image has changed by at least a threshold translation or rotation relative to a previous camera pose when capturing a preceding keyframe image. The “preceding keyframe image” can correspond to an image that was last identified as being a keyframe image from among the images from the camera122, and which may also be currently used as a keyframe image. The determination of the locations of the densified set of 2D features in 3D space can then be limited to being performed on the keyframe images.

The operations for determining206the locations of the 2D features in 3D space utilize the image-based map105which is available and adapted for use by the image-based SLAM algorithm. In such case, if a 2D feature detected in the current image is also detected in the image-based map105of the real-world, then such information is used to more accurately compute the pose of the camera122and also to more accurately compute the position of new 2D features detected in the current image. In the image-based SLAM algorithm described in the paper by Campos et al., an already known 2D feature will be removed from the list of 2D features for the current image. This has however a negative effect on the performance of the algorithm since having fewer features results in a sparser 3D representation of the real-world. Accordingly, in some embodiments when a match between a 2D feature (or a set of 2D features) in the current image and a 2D feature in an image-based map105of the real-world occurs, then the operations do not filter away the 2D feature but identify it as a “known 2D feature”, as will be described in further detail below regarding some ways of performing densification.

The operation for performing204densification of image pixels neighboring at least some of the 2D features detected in the images to generate a densified set of 2D features, can be configured to create a denser 3D representation of the real-world as seen from the camera122. The densification may operate to create more densified image pixel depth regions around the 2D features. In some embodiments, the densification204is performed using a depth completion algorithm or guided depth completion algorithm such as a non-local spatial propagation network (NLSPN) algorithm. An example NLSPN that can be used is described in a paper by J. Park, K. Joo, Z. Hu, C.-K. Liu, and I. So Kweon, “Non-local Spatial Propagation Network for Depth Completion”, inComputer Vision—ECCV2020, Lecture Notes in Computer Science, vol. 12358, pp. 120-136, Springer, 2020. Additionally, or alternatively, the densification204may be performed using interpolation or extrapolation among image pixels adjacent to a detected 2D feature.

Some further embodiments are directed to improving accuracy and/or computational efficiency of the densification204operation by constraining the densification204to being performed locally to a detected 2D feature, i.e., perform the densification204among image pixels that locally neighbor the detected 2D feature. In one illustrative embodiment shown inFIG.4, the operation to perform 204 densification of least some of the 2D features includes constraining400the densification of image pixels to only being performed within a threshold distance from respective ones of the 2D features detected in one of the images on which the densification is to be performed. The constraining400operation may include starting from an image pixel location of one of the 2D features detected in one of the images, and repetitively performing the densification on neighboring image pixels which extend in a sequence away from the image pixel location of the one of the 2D features and which are within a threshold number of image pixel locations from the image pixel location of the one of the 2D features detected in the one of the images.

The constraining400operation may include constraining the densification of image pixels to only being performed within defined areas which each have defined dimensions and enclose respective ones of the 2D features detected in the one of the images. For example, the densification204can include starting from the image pixel location of a detected 2D feature towards the neighboring image pixels and densifying up to a distance of Z image pixels from the image pixel location of the detected 2D feature. One or more of the areas can be defined to be circular having a defined radius in accordance with one embodiment. Alternatively or additionally, one or more of the areas can be defined to be rectangular with defined edge dimensions in accordance with another embodiment. Dimensions of the defined areas can be adapted based on trial and error, for example by evaluating how accurately the localization210is perform for the camera122based on different values of the area dimensions.

A reason to perform only local densification is that depth completion operations will introduce increasing errors the further away from the 2D feature the completion is performed. Accordingly, by constraining the densification204to only densify “just enough pixels” can reduce or minimize the localization error. Moreover, performing a local densification reduces the computational time and computational resources that are consumed by the densification204operation.

FIG.10Aillustrates a full 3D reconstruction of a scene by a depth completion algorithm andFIG.10Billustrates a partial 3D reconstruction of defined regions of the scene by the depth completion algorithm, in accordance with some embodiments of the present disclosure.

Some embodiments are directed to constraining the densification204to be performed only on image pixels neighboring the 2D features which are located within a desired region (min max distance, min max height, min max width, etc.) of an image from the camera122location. Estimation of the location of the 2D features by the image-based localization algorithm, e.g., image-based SLAM algorithm, may not be sufficiently accurate for distances which are too close or too far away, too high or too low, or too far on the edges of the image. Also, the estimation of the location of the 2D features may be affected by the brightness of the scene, the presence of dynamic objects and the speed of the first mobile device120. These and other embodiments may operate to trim the features which are used for localization210based on this additional information to improve the localization accuracy and robustness and/or to improve the computational efficiency of the computing resources used in the map processing device100. These operations can be tuned by trial and error and optimized for the specific characteristics of the camera122and the image-based localization algorithm, e.g., image-based SLAM algorithm.

Some operational embodiments are directed to improving accuracy and/or computational efficiency of the densification204operation by constraining the densification204to provide a desired final density of image pixels neighboring the 2D features. With continuing reference toFIG.4, the operation to constrain400the densification of image pixels to only being performed within a threshold distance from respective ones of the 2D features detected in the one of the images, can include starting from an image pixel location of one of the 2D features detected in one of the images, and repetitively performing the densification on every instance of a defined number of neighboring image pixels along a sequence extending away from the image pixel location of the one of the 2D features and which are within a threshold number of image pixel locations from the image pixel location of the one of the 2D features detected in the one of the images. In a further embodiment, the defined number of neighboring image pixels is determined so that a density of the locations of image pixels in the densified set of 2D features in 3D space satisfies a defined rule for matching a density of the locations in 3D space corresponding to the features indicated by the set of data points in the structure-based map102.

Thus, for example, the densification can be performed every X pixels (instead of every pixel) starting from a 2D feature location in order to create a region with a desired density W of image pixels, where the desired density W can correspond to the density of the 3D representation obtained from the depth sensor112to which the localization210is attempting to rely upon. It can be advantageous that the density of the two 3D representations, i.e., the density resulting from the densification204and the density of the depth information104provided by the depth sensor112, to be as close as possible.

With continued reference toFIG.4, some other operational embodiments are directed to improving accuracy and/or computational efficiency of the densification204operation by constraining402the densification204to only being performed within a defined region of the images. The operations may control the defined region to exclude image pixels that are less a threshold number of image pixels away from any edge of the images. The operations may control the defined region to exclude image pixels that are more than a threshold number of image pixels from a center of the images. The location of the defined region within the images may be determined based on feedback of accuracy obtained during the matching of the locations of the densified set of 2D features in 3D space to the depth information104accessed in the structure-based map102.

In some further embodiments, the operations may constrain the densification204to only being performed on image pixels neighboring the 2D features which are located within a defined region relative to location of the camera122. Alternatively or additionally, the operations may constrain the densification204to only being performed on the 2D features which satisfy at least one of: being stationary; having movement with speed less than a defined threshold; and having a brightness level that satisfies a brightness threshold.

Some further embodiments are directed to improving accuracy and/or computational efficiency of the densification204operation for 2D features which are contained in the image-based map105(also referred to as “known 2D features”). In some illustrative embodiments shown inFIG.5, the operation to perform 204 densification of least some of the 2D features detected in the images to generate the densified set of 2D features, includes determining500that one of the 2D features detected in the images corresponds to a 2D feature contained in an image-based map105, and retrieving 502 from the image-based map105a set of 2D features that are within a threshold distance of the 2D feature contained in the image-based map105for inclusion in the densified set of 2D features. The determination500that one of the 2D features detected in the images corresponds to a 2D feature contained in the image-based map105may include determining that a subset of 2D features detected in the images satisfies a defined rule for geometrically matching locations of a subset of 2D features contained in the image-based map105.

Thus, for example, when a detected 2D feature is a “known 2D feature” because it corresponds to a 2D feature contained in an image-based map105, the densification204can load the densified patch for the known 2D feature from the image-based map105instead of performing the densification only on image pixels neighboring the 2D feature. In operation to determine that a 2D feature is a “known 2D feature” isn't necessarily performed based on the individual 2D feature matching one-to-one a 2D feature contained in the image-based map105, but instead may use a voting operation that seeks to match a set of 2D features detected in an image to a corresponding set of 2D features contained in the image-based map105, where geometric consistency (e.g., where the relative geometric relations between the matched 2D features is the same) has to be verified (e.g. by a minimum of a defined number N of the 2D features currently detected in the image are geometrically consistent with respect to the defined number N of the 2D features contained in the image-based map105).

Some further embodiments are directed to selectively discarding a densified set of 2D features based on one or more defined conditions being satisfied.

In one embodiment, following the densification204, the operations determine whether a variance in depth values between pixels of neighboring 2D features in the densified set of 2D features exceeds a defined threshold. When the variance in depth values exceeds the defined threshold, the operations responsively discard the densified set of 2D features without performing the determination206of locations of the densified set of 2D features.

Thus, for example, after the densification204has been performed for a 2D feature, the created densified 2D feature can be analyzed to determine the variance between the depth values of image pixels of the densified 2D feature compared to depth values of the image pixels of a neighboring 2D feature. When the variance exceeds the defined threshold, this indicates that the depth completion algorithm determined there is a relatively big jump between the detected 2D feature depth value(s) and the neighbor 2D feature depth value(s), which also indicates that the detected 2D feature may have been incorrectly estimated by the image-based localization algorithm, e.g., image-based SLAM algorithm. Consequently, a region where the depth variance around the 2D feature exceeds the defined threshold should be discarded without further processing to determine locations of the 2D features or to localize the camera122.

The densified regions (densified set of 2D features) around each 2D feature detected in an image may be stored in the image-based map105, such as with corresponding 2D feature descriptors and 3D structure, so that the information can be re-used in the future. Advantageously, the storage of the densified regions can be performed on the map vertex of a graph-based map structure, where the vertex contains all the 2D features and densified patches observed from the vertex pose. Thus, in some embodiments, the operations by the map processing device100may add to the image-based map105, 2D feature descriptors106for the densified204set of 2D features and the locations determined206for the densified set of 2D features in 3D space. In this manner, the operations can add to the image-based map105based on processing of the images camera122being transported with the second mobile device120.

Some further operational embodiments are directed to improving the computational efficiency by performing incremental segmentation of the depth information104, stored in the structure-based map102, sparsifying the incremental segments of the depth information104, and then performing the localization of the camera122to the real-world based on matching the locations of the densified set of 2D features in 3D space to the sparsified incremental segments of the depth information. These and further embodiments are explained in the example context ofFIG.6which illustrates operations that can be performed by the map processing device100.

Referring toFIGS.1and6, the operations to localize210the camera122to the real-world based on matching the locations of the densified set of 2D features in 3D space to the depth information104accessed in the structure-based map102, can include performing600incremental segmentation of the depth information104, stored in the structure-based map102, to generate incremental segments of the depth information104. The depth information104stored in the structure-based map102comprises the set of data points indicating locations in 3D space corresponding to features in the real-world sensed by the depth sensor112transported by the first mobile device110. The operations then sparsify602the incremental segments of the depth information104to have a feature density that more closely matches a feature density of the densified set of 2D features. The operations then perform 604 the localization of the camera122transported with the second mobile device120to the real-world based on matching the locations of the densified set of 2D features in 3D space to the sparsified incremental segments of the depth information104. The operation to sparsify602the incremental segments of the depth information104to have a feature density that more closely matches a feature density of the densified set of 2D features, can improve the accuracy and/or the computational efficiency (e.g., utilization of processing and memory resources of the map processing device100) to perform 604 the localization of the camera122.

Some further embodiments are directed to updating the structure-based map102based on processing the images from the camera122transported with the second mobile device120, to provide improved localization capability for the first mobile device110using a structure-based localization algorithm, e.g., a structure-based SLAM algorithm. For example, the operations can add to the structure-based map102depth information generated from the determined locations of the densified set of 2D features in 3D space. The operations may localize the depth sensor112to the real-world based on matching depth information from the depth sensor112to the depth information104accessed in the structure-based map102, using a structure-based localization algorithm, e.g., a structure-based SLAM algorithm.

The sparsification operation602can be particularly beneficial when the 3D representation of the real-world provided by the depth sensor112has a higher density than the 3D representation of the real-world provided by the camera122.FIG.11Aillustrates depth information before sparsifying andFIG.11Billustrates the depth information after sparsifying, in accordance with some embodiments of the present disclosure. It is noted from comparison ofFIGS.11A and11Bthat the sparsification operation retains sufficient depth information to allow the image-based localization algorithm to perform localization with an acceptable accuracy.

FIG.12Aillustrates incremental segments generated using images from a camera andFIG.12Billustrates incremental segments generated using a Lidar sensor, in accordance with some embodiments of the present disclosure. It is noted from comparison ofFIGS.12A and12Bthat the incremental segments generated using images from the camera122can accurately correspond to the incremental segments generated using a Lidar type of depth sensor112.

FIG.7is a combined block diagram and flowchart of various combinations of operations explained above that can be performed in accordance with some embodiments.

Referring toFIG.7, images from the camera122are processed to detect2022D features using an image-based localization algorithm. The 2D feature detection202may rely on matching to features106in an image-based map105, when available such as described above for a “known 2D feature.” The detected2022D features are densified204. The densified set of 2D features are processed to determine206locations of the densified set of 2D features in 3D space using an image-based localization algorithm. The camera122is localized to the real-world based on matching the 3D locations of the densified set of 2D features to the depth information104accessed in the structure-based map102. The operations may also store an indication of the determined locations in 3D space of the features detected in the images as updated depth information104in the structure-based map102for subsequent use for, e.g., localizing the depth sensor112transported with the first mobile device110.

Further definitions and embodiments are now explained below.

Many variations and modifications can be made to the embodiments without substantially departing from the principles of the present inventive concepts. All such variations and modifications are intended to be included herein within the scope of present inventive concepts. Accordingly, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended examples of embodiments are intended to cover all such modifications, enhancements, and other embodiments, which fall within the spirit and scope of present inventive concepts. Thus, to the maximum extent allowed by law, the scope of present inventive concepts is to be determined by the broadest permissible interpretation of the present disclosure including the following examples of embodiments and their equivalents and shall not be restricted or limited by the foregoing detailed description.