Patent Description:
Global localization technology refers to a technology of estimating a six-degrees-of-freedom (6DoF) pose of a device based on map information associated with a target space when the map information is known in advance. Through the global localization technology, an absolute position of the device may be estimated based on coordinates of a determined map. The global localization technology may be used to initially estimate a pose of the device in the target space or used in a situation where tracking of a position of the device after initially estimating the pose is lost. The global localization technology using images captured or sensed by the device may include, for example, an image retrieval scheme of searching for at least one image matching a query image and estimating pose information corresponding to the found image, a direct pose regression scheme of directly regressing a pose from a query image using a pose regression deep network, a sparse feature matching scheme of storing feature information in a point cloud constituting a three-dimensional (3D) map, of matching the feature information to two-dimensional (2D) features of a query image, and of searching for a 2D-3D matching relationship, and a scene coordinate regression scheme of obtaining a 2D-3D matching relationship as a regression issue.

A global localization method of estimating 2D-3D correspondence with a deep network trained using a 3D map, an image, and a camera <NUM>-DOF pose enables fairly accurate pose estimation in a static environment when training is performed based on accurate data. In general, a Structure from Motion (SfM) is used to acquire a map and a ground truth pose, but it is difficult to acquire accurate data, and localization performance is greatly degraded if an inaccurate pose is used. If a realistic 3D map of a space requiring pose estimation is secured in advance, and if a synthetic image for multiple poses is created therefrom, inconsistency between an image and a pose can be overcome. However, if training is performed with these data, the trained network exhibits high performance for the synthetic image due to a domain gap between the real image and the synthetic image, but has a problem of exhibiting extremely low performance for the real image.

It is hence difficult to acquire labeled real data and to increase pose accuracy in a non-static environment through training database expansion.

In the paper "<NPL>ET AL disclose disclose relative camea pose estimation approach using synthetic data with domain adaptation via cycle-consistent adversarial networks to solve the continuous localization problem for autonomous navigation of unmanned systems.

This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description.

In one general aspect, a method for global localization includes: extracting a feature by applying an input image to a first network; estimating a coordinate map corresponding to the input image by applying the extracted feature to a second network; and estimating a pose corresponding to the input image based on the estimated coordinate map, wherein the first network is trained based on: a first generative adversarial network (GAN) loss determined based on a first feature extracted by the first network based on a synthetic image determined by three-dimensional (3D) map data and a second feature extracted by the first network based on a real image; and wherein the second network is trained based on: a second GAN loss determined based on a first coordinate map estimated by the second network based on the first feature and a second coordinate map estimated by the second network based on the second feature.

Either one or both of the first network and the second network may be trained further based on either one or both of: a first loss determined based on the first coordinate map and ground truth data corresponding to the synthetic image; and a second loss determined based on a first pose estimated based on the first coordinate map and the ground truth data corresponding to the synthetic image.

The ground truth data may include a pose of a virtual camera that captures the synthetic image and 3D coordinate data corresponding to each pixel of the synthetic image.

The pose may include a six-degrees-of-freedom (6DoF) pose of a device that captures the input image.

In another general aspect, one or more embodiments include a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform any one, any combination, or all operations and methods described herein.

In another general aspect, a method for global localization includes: determining a synthetic data set based on three-dimensional (3D) map data, the synthetic data set comprising a synthetic image captured by a virtual camera corresponding to an arbitrary pose and 3D coordinate data corresponding to each pixel of the synthetic image; determining a first generative adversarial network (GAN) loss based on a first feature determined by applying the synthetic image to the first network and a second feature determined by applying a real image captured by a real camera to the first network; determining a second GAN loss based on a first coordinate map determined by applying the first feature to the second network and a second coordinate map determined by applying the second feature to the second network; determining a first loss based on the first coordinate map and the 3D coordinate data corresponding to the synthetic image; determining a second loss based on a first pose estimated based on the first coordinate map and a pose of the virtual camera; and training the first network based on the first loss, the second loss and the first GAN loss; and training the second network based on the first loss, the second loss and the second GAN loss.

The determining of the synthetic data set further may include: extracting the first feature by applying the synthetic image to the first network; estimating the first coordinate map corresponding to each pixel of the synthetic image by applying the extracted first feature to the second network; estimating a first pose corresponding to the synthetic image based on the estimated first coordinate map; extracting the second feature by applying the real image to the first network; and estimating the second coordinate map corresponding to each pixel of the synthetic image by applying the extracted second feature to the second network.

The training of the either one or both of the first network and the second network may include training the first network and a first discriminator based on the first GAN loss, the first discriminator being configured to discriminate between the first feature extracted from the synthetic image and the second feature extracted from the real image.

The training of the either one or both of the first network and the second network may include training the second network and a second discriminator based on the second GAN loss, the second discriminator being configured to discriminate between the first coordinate map estimated from the synthetic image and the second coordinate map estimated from the real image.

The training of the either one or both of the first network and the second network may include iteratively back-propagating a gradient determined based on the first loss to the first network and the second network.

The training of the either one or both of the first network and the second network may include iteratively back-propagating a gradient determined based on the second loss to the first network and the second network.

The method may include, in response to the training of the either one or both of the first network and the second network: extracting a feature by applying an input image to the first network; estimating a coordinate map corresponding to the input image by applying the extracted feature to the second network; and estimating a pose corresponding to the input image based on the estimated coordinate map.

In another general aspect, an apparatus for global localization comprises one or more processors configured to perform the methods mentioned above.

The apparatus may include a memory storing instructions that, when executed by the one or more processors, configure the one or more processors to perform the methods mentioned above.

Also, descriptions of features that are known, after an understanding of the disclosure of this application, may be omitted for increased clarity and conciseness.

The terminology used herein is for the purpose of describing particular examples only, and is not to be used to limit the disclosure. As used herein, the terms "include," "comprise," and "have" specify the presence of stated features, numbers, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, elements, components, and/or combinations thereof. The use of the term "may" herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Although terms of "first" or "second" are used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms.

Likewise, expressions, for example, "between" and "immediately between" and "adjacent to" and "immediately adjacent to" may also be construed as described in the foregoing.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, consistent with and after an understanding of the present disclosure. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the examples with reference to the accompanying drawings, like reference numerals refer to like elements and a repeated description related thereto will be omitted.

<FIG> illustrates an example of a framework of a global localization model <NUM>.

Referring to <FIG>, the global localization model <NUM> (or a global localization device) may correspond to a model that performs a global localization method of receiving an image <NUM> (e.g., one or more images) and outputting a pose <NUM> corresponding to a global localization result. The global localization model <NUM> may include a first network <NUM>, a second network <NUM>, and a pose estimator <NUM>. The first network <NUM> and the second network <NUM> may be trained neural networks. Hereinafter, the global localization model <NUM> may be briefly referred to as a "model".

The image <NUM> may be an image obtained (e.g., determined) by projecting an object located in a three-dimensional (3D) space onto a two-dimensional (2D) plane, and a shape of the projected object in the image may vary according to a position and an angle at which the object is captured.

For example, referring to <FIG>, an image <NUM> projected onto a 2D plane may be obtained by capturing an object <NUM> located in a 3D space at a predetermined position and a predetermined angle. When a position and/or an angle at which the object <NUM> is captured are changed, a shape of the object <NUM> included in the image <NUM> may change. The shape of the object <NUM> in the image <NUM> may correspond to a shape obtained through projection transformation of an actual shape of the object <NUM> according to the position and angle at which the object <NUM> is captured, that is, according to a pose of a device <NUM> that captures or senses the object <NUM>.

Referring back to <FIG>, the model <NUM> may know 3D coordinates corresponding to the 3D space of the projected object on the input image <NUM>. In this example, a position and an angle at which the object is captured (for example, a pose of a device that captures or senses the object) may be estimated. The model <NUM> may estimate 3D coordinates (hereinafter, a coordinate map) corresponding to each pixel of an image including projection of an object based on the trained first and second networks <NUM> and <NUM>. The model <NUM> may estimate the pose <NUM> of the device that obtains the input image <NUM>, based on a relationship between 2D coordinates of pixels in the image <NUM> and 3D coordinates corresponding to the pixels, based on the pose estimator <NUM>.

The first network <NUM> and the second network <NUM> in the global localization model <NUM> may be, for example, neural networks each including at least one layer with parameters that are determined through training.

The global localization method based on the global localization model <NUM> may include an operation of extracting a feature by applying the input image <NUM> to the first network <NUM>, an operation of estimating a coordinate map corresponding to the input image <NUM> by applying the extracted feature to the second network <NUM>, and an operation of estimating the pose <NUM> corresponding to the input image <NUM> by applying the estimated coordinate map to the pose estimator <NUM>.

According to an example, the first network <NUM> may correspond to a neural network trained to extract a feature from an input image.

According to an example, the second network <NUM> may correspond to a neural network trained to estimate the coordinate map corresponding to the input image <NUM> based on the feature extracted by the first network <NUM>. The coordinate map may correspond to 3D coordinates of a region in a 3D space corresponding to each pixel of an image.

For example, referring to <FIG>, a pixel <NUM> included in an image <NUM> may correspond to a point <NUM> in a 3D space, and the point <NUM> in the 3D space corresponding to the pixel <NUM> may be located at coordinates (xp, yp, zp). A coordinate map <NUM> may include a matrix of elements c = {ci, | i = <NUM>, <NUM>,. , n, j = <NUM>, <NUM>,. , m} corresponding to pixels p = {pi,j | i = <NUM>, <NUM>,. , n, j = <NUM>, <NUM>,. , m} included in the image <NUM>, and a value of an element ci,j corresponding to each pixel pi,j may correspond to 3D coordinates ci,j = [xi,j, yi,j, zi,j].

Referring back to <FIG>, the pose estimator <NUM> may estimate the pose <NUM> corresponding to the input image <NUM>, based on the coordinate map estimated at the second network <NUM>. For example, the pose estimator <NUM> may include a solvePnP (or perspective-n-point) function that outputs the pose <NUM> by receiving the coordinate map. The pose <NUM> output from the pose estimator <NUM> may include position information and direction information of the device that captures or senses the input image <NUM>. The device may include, for example, a camera, a device including a camera, and/or a device including an image sensor.

The pose <NUM> may include a six-degrees-of-freedom (6DoF) pose. The 6DoF pose may include 3D position information about three axes orthogonal to each other in vertical, horizontal, and depth directions, and direction information about a degree of inclination with respect to the three axes.

For example, referring to <FIG>, a global localization model may estimate a 6DoF pose including position information and direction information of a device <NUM> (e.g., a camera) that captures an image including projection of an object <NUM> in a space in which the object <NUM> is located, based on a relationship between 2D coordinates of a pixel in the image and 3D coordinates corresponding to the pixel.

According to an example, the global localization model <NUM> may include the first network <NUM> and the second network <NUM> that are trained neural networks, and the first network <NUM> and the second network <NUM> may include parameters determined by a training process according to a backpropagation algorithm.

In other words, the global localization model <NUM> may be generated by a training process of a global localization model. The training process may include an operation of obtaining a synthetic image captured by a virtual camera corresponding to an arbitrary pose and 3D coordinate data corresponding to each pixel of the synthetic image, based on 3D map data, an operation of iteratively back-propagating a gradient obtained based on at least one loss associated with a global localization model, to update parameters of the first network <NUM> and the second network <NUM> that are neural networks included in the global localization model, and an operation of storing the parameters of the first network <NUM> and the second network <NUM>.

In an example, the first network <NUM> and/or the second network <NUM> may be trained to output ground truth data corresponding to an image included in training data. The image included in the training data may include an accurate coordinate map corresponding to the image and a synthetic image labeled with an accurate pose corresponding to the image. The synthetic image may be an image generated by assuming an image captured by a virtual camera based on 3D map data, and may be distinguished from a real image captured by a real camera in a real space. A non-limiting example of the training data including the synthetic image will be described below.

According to an example, at least one of the first network <NUM> and the second network <NUM> may be trained based on at least one of a first loss and a second loss. The first loss may be obtained based on ground truth data corresponding to a synthetic image included in training data and a coordinate map estimated by the second network <NUM> based on the synthetic image included in the training data, and the second loss may be obtained based on a pose estimated by the pose estimator <NUM> based on the synthetic image and the ground truth data corresponding to the synthetic image included in the training data. The ground truth data corresponding to the synthetic image included in the training data may include a ground truth (GT) of a coordinate map corresponding to the synthetic image, and a GT of a pose corresponding to the synthetic image. A non-limiting example of a method of training the global localization model based on the first loss and/or the second loss will be described in detail below.

In another example, the first network <NUM> and/or the second network <NUM> may be trained adversarially to a discriminator, which is configured to discriminate between a real image and a synthetic image, based on a generative adversarial network (GAN) loss according to the discriminator. For example, at least one of the first network <NUM> and the second network <NUM> may be trained based on at least one of a first GAN loss and a second GAN loss. The first GAN loss may correspond to a loss obtained based on a first feature extracted by the first network <NUM> based on a synthetic image obtained by 3D map data, and a second feature extracted by the first network <NUM> based on a real image. The second GAN loss may correspond to a loss obtained based on a first coordinate map estimated by the second neural network <NUM> based on the first feature, and a second coordinate map estimated by the second neural network <NUM> based on the second feature. An example of a method of training the global localization model based on the first GAN loss and/or the second GAN loss will be described in detail below.

<FIG> illustrates an example of a framework of a method of training a global localization model.

Referring to <FIG>, training data of the global localization model may include a synthetic data set <NUM> and a real image <NUM>. The synthetic data set <NUM> may include a synthetic image <NUM> captured by a virtual camera corresponding to an arbitrary pose, and 3D coordinate data corresponding to each pixel of the synthetic image <NUM>. The 3D coordinate data corresponding to each pixel of the synthetic image <NUM> may correspond to ground truth data <NUM> (hereinafter, referred to as a "GT coordinate map <NUM>") of a coordinate map corresponding to the synthetic image <NUM>. According to an example, the synthetic image <NUM> included in the synthetic data set <NUM> may include pose information of the virtual camera that captures the synthetic image <NUM>. In other words, the synthetic data set <NUM> may further include ground truth data <NUM> (hereinafter, referred to as a "GT pose <NUM>") of a pose corresponding to the synthetic image <NUM>. In other words, the synthetic image <NUM> included in the synthetic data set <NUM> may be labeled with the GT pose <NUM> and the GT coordinate map <NUM>.

According to an example, the synthetic data set <NUM> may be obtained based on 3D map data. For example, referring to <FIG>, 3D map data <NUM> may include a point cloud <NUM> corresponding to an arbitrary space. In other words, the 3D map data <NUM> may correspond to data obtained by modeling at least one object included in the arbitrary space as the point cloud <NUM> that is a set of points in a 3D virtual space. For example, referring to <FIG>, points included in a point cloud may be represented in a 3D coordinate system <NUM> having a predetermined position <NUM> as an origin in a 3D virtual space, and 3D coordinates corresponding to each point may be construed to indicate a position of a corresponding point in a 3D space. The points included in the point cloud may include 3D coordinates indicating a position and RGB values indicating a color.

Referring back to <FIG>, the synthetic image <NUM> captured by the virtual camera corresponding to the arbitrary pose may be obtained based on 3D map data. The synthetic image <NUM> may correspond to a composite projection image of an object obtained assuming that an object included in the 3D map data is captured by a virtual camera disposed at a predetermined angle and a predetermined position in a space corresponding to the 3D map data. In other words, the synthetic image <NUM> may be an image generated through projection transformation of some points included in the 3D map data, and may be distinguished from the real image <NUM> obtained by capturing a real space with a real camera. The projection transformation may be determined based on a position and an angle of the virtual camera. According to an example, the synthetic image <NUM> may further include color information based on RGB values of points projected on the synthetic image <NUM>.

3D coordinates in the 3D map data of a region corresponding to each pixel in the synthetic image <NUM> may be accurately identified, and accordingly the GT coordinate map <NUM> corresponding to the synthetic image <NUM> may be obtained. In addition, since the synthetic image <NUM> corresponds to an image generated based on 3D map data by assuming an arrangement of a virtual camera, a pose of a virtual device that captures the synthetic image <NUM> may be accurately known. Thus, the GT pose <NUM> corresponding to the synthetic image <NUM> may also be obtained.

According to an example, when the synthetic data set <NUM> includes the synthetic image <NUM>, and the GT pose <NUM> and the GT coordinate map <NUM> that correspond to the synthetic image <NUM>, a first loss <NUM> associated with a coordinate map estimated by the second network <NUM> and a second loss <NUM> associated with a pose estimated by a pose estimator <NUM> may be calculated in a training process.

According to an example, the first loss <NUM> may be obtained based on a first coordinate map output from the second network <NUM> based on the synthetic image <NUM> and the GT coordinate map <NUM> corresponding to the synthetic image <NUM>. The first coordinate map may be output as a result obtained by applying a first feature output from the first network <NUM> based on the synthetic image <NUM> to the second network <NUM>. In an example, a gradient obtained based on the first loss <NUM> may be iteratively back-propagated to the first network <NUM> and the second network <NUM>, so that weights of the first network <NUM> and the second network <NUM> may be updated. The gradient obtained based on the first loss <NUM> may correspond to a gradient obtained by a difference between the first coordinate map output from the second network based on the synthetic image <NUM> and the GT coordinate map <NUM> corresponding to the synthetic image <NUM>.

According to an example, the second loss <NUM> may be obtained based on a first pose output from the pose estimator <NUM> based on the synthetic image <NUM> and the GT pose <NUM> corresponding to the synthetic image <NUM>. The first pose may be output as a result obtained by inputting the first coordinate map output from the second network <NUM> based on the synthetic image <NUM> to the pose estimator <NUM>. For example, a gradient obtained based on the second loss <NUM> may be iteratively back-propagated to the first network <NUM> and the second network <NUM>, so that the weights of the first network <NUM> and the second network <NUM> may be updated. The gradient obtained based on the second loss <NUM> may correspond to a gradient obtained by a difference between the first pose output from the pose estimator <NUM> based on the synthetic image <NUM> and the GT pose <NUM> corresponding to the synthetic image <NUM>.

For example, when a weight of the second network <NUM> is denoted by ws, ws may be trained through two processes, e.g., a process of minimizing a difference from a GT coordinate map and a process of minimizing a difference from a GT pose. Equations for the two processes may be defined as shown in Equations <NUM> and <NUM> below, respectively, for example. <MAT> <MAT>.

In Equation <NUM>, f denotes an output of the first network <NUM>, S denotes the second network <NUM>, s = S(f;w) may correspond to a coordinate map output from f by a weight w of the second network <NUM>, s* denotes a GT coordinate map, and Ls denotes the first loss <NUM>. The first loss <NUM> Ls may be an error between the GT coordinate map s* and the coordinate map s estimated by the second network <NUM> and may be defined as a sum of si estimated for a pixel i included in the synthetic image <NUM> and a norm of si* that is ground truth data.

In Equation <NUM>, P denotes a pose estimator, p = P(S(f;w)) may correspond to a pose output from the pose estimator <NUM> based on the coordinate map s = S(f;w) output from the second network <NUM>, and Lp denotes the second loss <NUM>. The second loss <NUM> Lp may be an error between the pose p estimated by the pose estimator <NUM> and a GT pose p*, and may be determined as a maximum value of a rotation error ∠(θ, θ*) and a translation error ∥t - t*∥.

According to an example, when training of the global localization model is performed based on the synthetic data set <NUM> only, performance of global localization may be reduced with respect to the real image <NUM> corresponding to a domain different from that of the synthetic image <NUM>. To reduce a domain gap between the synthetic image <NUM> and the real image <NUM> in the global localization model, the training process of one or more embodiments may additionally be performed using a first discriminator <NUM> and a second discriminator <NUM>.

According to an example, the first discriminator <NUM> may perform domain classification of features output from the first network <NUM>, and the second discriminator <NUM> may perform domain classification of coordinate maps output from the second network <NUM>. The domain classification may correspond to an operation of determining whether input data corresponds to the synthetic image <NUM> or the real image <NUM>.

For example, the first discriminator <NUM> may include a neural network configured to output a value indicating the synthetic image <NUM> based on a feature extracted from the synthetic image <NUM>, and to output a value indicating the real image <NUM> based on a feature extracted from the real image <NUM>. The first discriminator <NUM> may be trained to output a value (e.g., a value of "<NUM>") indicating the synthetic image <NUM> when an input feature is determined to be a feature extracted from the synthetic image <NUM>, and to output a value (e.g., a value of "<NUM>") indicating the real image <NUM> when the input feature is determined to be a feature extracted from the real image <NUM>.

In addition, the second discriminator <NUM> may include a neural network configured to output a value indicating the synthetic image <NUM> based on a coordinate map estimated from the synthetic image <NUM>, and to output a value indicating the real image <NUM> based on a coordinate map estimated from the real image <NUM>. The second discriminator <NUM> may be trained to output a value (e.g., a value of "<NUM>") indicating the synthetic image <NUM> when an input coordinate map is determined to be a coordinate map estimated based on the synthetic image <NUM>, and to output a value (e.g., a value of "<NUM>") indicating the real image <NUM> when the input coordinate map is determined to be a coordinate map estimated based on the real image <NUM>.

According to an example, the first network <NUM> may be trained adversarially to the first discriminator <NUM>, based on a first GAN loss <NUM>. The first GAN loss <NUM> may be obtained based on a first feature extracted by the first network <NUM> based on the synthetic image <NUM> and a second feature extracted by the first network <NUM> based on the real image <NUM>. For example, when the synthetic image <NUM> is input to the global localization model, the first GAN loss <NUM> may be calculated based on a difference between a result output from the first discriminator <NUM> based on the first feature and a value indicating the synthetic image <NUM> which is ground truth data of domain classification corresponding to the first feature. When the real image <NUM> is input to the global localization model, the first GAN loss <NUM> may be calculated based on a difference between a result output from the first discriminator <NUM> based on the second feature and a value indicating the real image <NUM> which is ground truth data of domain classification corresponding to the second feature.

For example, a gradient obtained based on the first GAN loss <NUM> may be iteratively back-propagated to the first discriminator <NUM> and the first network <NUM>, so that weights of the first discriminator <NUM> and the first network <NUM> may be updated. The gradient obtained based on the first GAN loss <NUM> may correspond to a gradient obtained by a difference between a domain classification result output based on an input feature from the first discriminator <NUM> and ground truth data of domain classification corresponding to the input feature. For example, based on the first GAN loss <NUM>, the first discriminator <NUM> may be trained to output the ground truth data of the domain classification based on the input feature, and the first network <NUM> may be trained to extract features on which it is difficult for the first discriminator <NUM> to perform domain classification (e.g., the first network <NUM> may be trained to extract features such that an accuracy of domain classification performed by the first discriminator <NUM> based on the extracted features is below a predetermined threshold).

According to an example, in the training process, the synthetic image <NUM> and the real image <NUM> may be alternately input to the global localization model. In an example, a process of back-propagating a gradient obtained by inputting the first feature to the first discriminator <NUM> based on the first GAN loss <NUM> in response to an input of the synthetic image <NUM>, and a process of back-propagating a gradient obtained by inputting the second feature to the first discriminator <NUM> based on the first GAN loss <NUM> in response to an input of the real image <NUM> may be alternately performed. In another example, a process of back-propagating a gradient obtained by inputting the first coordinate map to the second discriminator <NUM> based on the second GAN loss <NUM> in response to an input of the synthetic image <NUM>, and a process of back-propagating a gradient obtained by inputting the second coordinate map to the second discriminator <NUM> based on the second GAN loss <NUM> in response to an input of the real image <NUM> may be alternately performed.

According to an example, the second network <NUM> may be trained adversarially to the second discriminator <NUM>, based on the second GAN loss <NUM>. The second GAN loss <NUM> may be obtained based on the first coordinate map estimated by the second network <NUM> based on the first feature, and the second coordinate map estimated by the second network <NUM> based on the second feature. As described above, the first feature may correspond to a feature output from the first network <NUM> based on the synthetic image <NUM>, and the second feature may correspond to a feature output from the first network <NUM> based on the real image <NUM>. For example, when the synthetic image <NUM> is input to the global localization model, the second GAN loss <NUM> may be calculated based on a difference between a result output from the second discriminator <NUM> based on the first coordinate map and a value indicating the synthetic image <NUM> which is ground truth data of domain classification corresponding to the first coordinate map. When the real image <NUM> is input to the global localization model, the second GAN loss <NUM> may be calculated based on a difference between a result output from the second discriminator <NUM> based on the second coordinate map and a value indicating the real image <NUM> which is ground truth data of domain classification corresponding to the second coordinate map.

In an example, a gradient obtained based on the second GAN loss <NUM> may be iteratively back-propagated to the second discriminator <NUM> and the second network <NUM>, so that weights of the second discriminator <NUM> and the second network <NUM> may be updated. The gradient obtained based on the second GAN loss <NUM> may correspond to a gradient obtained by a difference between a domain classification result output based on an input coordinate map from the second discriminator <NUM> and ground truth data of domain classification corresponding to the input coordinate map. For example, based on the second GAN loss <NUM>, the second discriminator <NUM> may be trained to output the ground truth data of the domain classification based on the input coordinate map, and the second network <NUM> may be trained to extract coordinate maps on which it is difficult for the second discriminator <NUM> to perform domain classification (e.g., the second network <NUM> may be trained to extract coordinate maps such that an accuracy of domain classification performed by the second discriminator <NUM> based on the extracted coordinate maps is below a predetermined threshold).

According to an example, a gradient of the second GAN loss <NUM> may also be back-propagated to the first network <NUM> in addition to the second discriminator <NUM> and the second network <NUM>, and accordingly the weight of the first network <NUM> may be updated.

According to an example, a process of back-propagating the gradient obtained based on the first GAN loss <NUM> may include a process of back-propagating a gradient of an error corresponding to an output obtained by inputting the first feature to the first discriminator <NUM> to the first discriminator <NUM> and the first network <NUM> based on the first GAN loss <NUM>, and a process of iteratively back-propagating a gradient of an error corresponding to an output obtained by inputting the second feature to the first discriminator <NUM> to the first discriminator <NUM> and the first network <NUM> based on the first GAN loss <NUM>. For example, the gradient of the error corresponding to the output obtained by inputting the first feature to the first discriminator <NUM>, and the gradient of the error corresponding to the output obtained by inputting the second feature to the first discriminator <NUM> may be alternately back-propagated.

According to an example, an operation of training at least one of the first network <NUM> and the second network <NUM> may include training the second network <NUM> and the second discriminator <NUM> based on the second GAN loss <NUM>. The second discriminator <NUM> may be configured to discriminate between a coordinate map estimated from the synthetic image <NUM> and a coordinate map estimated from the real image <NUM>. The second GAN loss <NUM> may include a loss that is obtained based on the first coordinate map estimated by the second network <NUM> based on the first feature and the second coordinate map estimated by the second network <NUM> based on the second feature. For example, the synthetic image <NUM> and the real image <NUM> may be alternately input to the global localization model, and a gradient of an error obtained based on the second GAN loss <NUM> may be iteratively back-propagated to the second discriminator <NUM> and the second network <NUM>.

According to an example, a process of back-propagating the gradient of the error obtained based on the second GAN loss <NUM> may include a process of back-propagating a gradient of an error corresponding to an output obtained by inputting the first feature to the second discriminator <NUM> to the second discriminator <NUM> and the second network <NUM> based on the second GAN loss <NUM>, and a process of iteratively back-propagating a gradient of an error corresponding to an output obtained by inputting the second feature to the second discriminator <NUM> to the second discriminator <NUM> and the second network <NUM> based on the second GAN loss <NUM>. For example, the synthetic image <NUM> and the real image <NUM> may be alternately input to the global localization model, and the gradient of the error corresponding to the output obtained by inputting the first feature to the second discriminator <NUM>, and the gradient of the error corresponding to the output obtained by inputting the second feature to the second discriminator <NUM> may be alternately back-propagated.

For example, when a weight of the first discriminator <NUM> and a weight of the second discriminator <NUM> are denoted by wD1 and wD2, respectively, wD1 and wD2 may be trained through processes of minimizing a difference from ground truth data of domain classification. Equations for the above processes may be defined as shown in Equations <NUM> and <NUM> below, for example. <MAT> <MAT>.

In Equations <NUM> and <NUM>, d*∈{<NUM>,<NUM>} denotes ground truth data of domain classification and may use a binary variable indicating the synthetic image <NUM> or the real image <NUM>, D<NUM> denotes the first discriminator <NUM>, D<NUM>(f; w) denotes a domain classification result output from the first discriminator <NUM> based on an input feature f, LD<NUM> denotes the first GAN loss <NUM>, D<NUM> denotes the second discriminator <NUM>, D<NUM>(s; w) denotes a domain classification result output from the second discriminator <NUM> based on an input coordinate map s, and LD<NUM> denotes the second GAN loss <NUM>. The first GAN loss <NUM> LD<NUM> and the second GAN loss <NUM> LD<NUM> may use binary cross entropy.

As described above, the first network <NUM> may be trained adversarially to the first discriminator <NUM> based on the first GAN loss <NUM>, and may be trained based on the first loss <NUM> and the second loss <NUM>. In an example, a process of training the first network <NUM> may be defined as two operations, that is, a training operation to update the weight of the first network <NUM> based on the first GAN loss <NUM> and the first loss <NUM>, and a training operation to update the weight of the first network <NUM> based on the first GAN loss <NUM> and the second loss <NUM>, as shown in Equations <NUM> and <NUM> below, respectively, for example. <MAT> <MAT>.

As described above, the second network <NUM> may be trained adversarially to the second discriminator <NUM> based on the second GAN loss <NUM>, and may be trained based on the first loss <NUM> and the second loss <NUM>. In an example, a process of training the second network <NUM> may be defined as two processes, that is, a training process to update the weight of the second network <NUM> based on the second GAN loss <NUM> and the first loss <NUM>, and a training process to update the weight of the second network <NUM> based on the second GAN loss <NUM> and the second loss <NUM>, as shown in Equations <NUM> and <NUM> below, respectively, for example. <MAT> <MAT>.

As described above, the gradient of the second GAN loss <NUM> may also be back-propagated to the first network <NUM> in addition to the second discriminator <NUM> and the second network <NUM>, and accordingly the first network <NUM> may be trained to update the weight of the first network <NUM> based on the second GAN loss <NUM>.

<FIG> is a flowchart illustrating an example of a method of training a global localization model.

Referring to <FIG>, the method of training the global localization model may include operation <NUM> of obtaining a synthetic data set based on 3D map data, operation <NUM> of obtaining a first GAN loss, operation <NUM> of obtaining a second GAN loss, operation <NUM> of obtaining a first loss, operation <NUM> of obtaining a second loss, and operation <NUM> of training at least one of a first network and a second network. Operations <NUM> to <NUM> of the method of training the global localization model may be performed by at least one processor.

Operation <NUM> may include obtaining a synthetic data set including a synthetic image captured by a virtual camera corresponding to an arbitrary pose and 3D coordinate data corresponding to each pixel of the synthetic image. As described above, the 3D coordinate data corresponding to each pixel of the synthetic image may correspond to a GT coordinate map, and the synthetic data set may further include a GT pose corresponding to the synthetic image.

The method of training the global localization model may further include, after operation <NUM>, an operation of extracting a first feature by applying the synthetic image to the first network, an operation of estimating a first coordinate map corresponding to each pixel of the synthetic image by applying the extracted first feature to the second network, and an operation of estimating a first pose corresponding to the synthetic image based on the estimated first coordinate map. In addition, the method may further include an operation of extracting a second feature by applying a real image captured by a real camera to the first network, and an operation of estimating a second coordinate map corresponding to each pixel of the real image by applying the extracted second feature to the second network.

Operation <NUM> may include an operation of obtaining the first GAN loss based on the first feature obtained by applying the synthetic image to the first network and the second feature obtained by applying the real image to the first network.

Operation <NUM> may include obtaining the second GAN loss based on the first coordinate map obtained by applying the first feature to the second network and the second coordinate map obtained by applying the second feature to the second network.

Operation <NUM> may include an operation of obtaining the first loss based on the first coordinate map and the 3D coordinate data corresponding to the synthetic image.

Operation <NUM> may include obtaining the second loss based on the first pose estimated based on the first coordinate map and a pose of the virtual camera.

Operation <NUM> may include training at least one of (e.g., either one or both of) the first network and the second network based on any one or any combination of the first loss, the second loss, the first GAN loss, and the second GAN loss. For example, operation <NUM> may include an operation of iteratively back-propagating a gradient obtained based on at least one loss (e.g., at least one of the first loss, the second loss, the first GAN loss, and the second GAN loss) associated with the global localization model, to update parameters of a neural network (e.g., at least one of the first network and the second network) included in the global localization model, and an operation of storing the parameters of the neural network. The parameters of the neural network may be stored in a recording medium or a memory in which the global localization model is stored.

According to an example, operation <NUM> may include an operation of training the first network and a first discriminator, which is configured to discriminate between a feature extracted from the synthetic image and a feature extracted from the real image, based on the first GAN loss. For example, operation <NUM> may include an operation of iteratively back-propagating a gradient obtained by inputting the first feature to the first discriminator to the first discriminator and the first network, based on the first GAN loss, and an operation of iteratively back-propagating a gradient obtained by inputting the second feature to the first discriminator to the first discriminator and the first network, based on the first GAN loss. As described above, the first discriminator may include a neural network configured to output a value indicating the synthetic image based on the feature extracted from the synthetic image, and to output a value indicating the real image based on the feature extracted from the real image.

According to an example, operation <NUM> may include an operation of training the second network and a second discriminator, which is configured to discriminate between a coordinate map estimated from the synthetic image and a coordinate map estimated from the real image, based on the second GAN loss. For example, operation <NUM> may include an operation of iteratively back-propagating a gradient obtained by inputting the first coordinate map to the second discriminator to the second discriminator and the second network, based on the second GAN loss, and an operation of iteratively back-propagating a gradient obtained by inputting the second coordinate map to the second discriminator to the second discriminator and the second network, based on the second GAN loss. As described above, the second discriminator may include a neural network configured to output a value indicating the synthetic image based on the coordinate map estimated from the synthetic image, and to output a value indicating the real image based on the coordinate map estimated from the real image.

In an example, operation <NUM> may include iteratively back-propagating a gradient obtained based on the first loss to the first network and the second network. In another example, operation <NUM> may include iteratively back-propagating a gradient obtained based on the second loss to the first network and the second network.

Referring to <FIG>, the global localization model <NUM> may correspond to a model configured to perform a global localization method of receiving an image <NUM> and outputting a pose <NUM> corresponding to a global localization result. The global localization model <NUM> may include a trained neural network, and the neural network may include a first network <NUM> and a third network <NUM>.

According to an example, the image <NUM> and the pose <NUM> that are input data and output data of the global localization model <NUM>, respectively, may respectively correspond to the image <NUM> and the pose <NUM> that are input data and output data of the global localization model <NUM> of <FIG>, respectively.

The first network <NUM> may be, for example, a network trained to extract a feature from an input image and may correspond to the neural network <NUM> of <FIG>.

The third network <NUM> may correspond to, for example, a neural network trained to estimate a pose corresponding to the input image <NUM> based on a feature extracted by the first network <NUM>. The pose estimator <NUM> of <FIG> may be a module to estimate the pose <NUM> corresponding to the input image <NUM> using, for example, a PnP algorithm, by receiving, as an input, a coordinate map that is an output of the second network <NUM>, whereas the third network <NUM> may be a neural network to estimate the pose <NUM> corresponding to the input image <NUM> by receiving, as an input, a feature that is an output of the first network <NUM>. Thus, the global localization model <NUM> may differ from the global localization model <NUM> of <FIG>.

According to an example, the first network <NUM> and the third network <NUM> included in the global localization model <NUM> may include parameters determined by a training process based on a backpropagation algorithm. In other words, the global localization model <NUM> may be generated by a training process of a global localization model. The training process may include an operation of obtaining a synthetic image captured by a virtual camera corresponding to an arbitrary pose, based on 3D map data, an operation of iteratively back-propagating a gradient obtained based on at least one loss associated with the global localization model to update parameters of the first network <NUM> and the third network <NUM> included in the global localization model <NUM>, and an operation of storing the parameters of the first network <NUM> and the third network <NUM>.

In an example, the first network <NUM> and/or the third network <NUM> may be trained to output ground truth data corresponding to an image included in training data. For example, the first network <NUM> and the third network <NUM> may be trained based on a second loss obtained based on a pose estimated by the third network <NUM> based on a synthetic image included in the training data and ground truth data corresponding to the synthetic image included in the training data. The ground truth data corresponding to the synthetic image included in the training data may include a ground truth of a pose corresponding to the synthetic image. The second loss may correspond to the second loss described above with reference to <FIG>. An example of a method of training the global localization model based on the second loss will be described in detail below.

In another example, the first network <NUM> and the third network <NUM> may be trained adversarially to a discriminator, which is configured to discriminate between a real image and a synthetic image, based on a GAN loss according to the discriminator. For example, the first network <NUM> may be trained based on a first GAN loss. The first GAN loss may correspond to the first GAN loss described above with reference to <FIG>. In other words, the first GAN loss may correspond to a loss obtained based on a first feature extracted by the first network <NUM> based on a synthetic image obtained based on 3D map data, and a second feature extracted by the first network <NUM> based on a real image. An example of a method of training the global localization model based on the first GAN loss will be described in detail below.

<FIG> illustrates another example of a framework of a method of training a global localization model.

Referring to <FIG>, training data of the global localization model may include a synthetic data set <NUM> and a real image <NUM>. The synthetic data set <NUM> may include a synthetic image <NUM> captured by a virtual camera corresponding to an arbitrary pose, and ground truth data <NUM> (hereinafter, referred to as a "GT pose <NUM>") of a pose corresponding to the synthetic image <NUM>. As described above, the synthetic image <NUM> included in the synthetic data set <NUM> may include pose information of the virtual camera that captures the synthetic image <NUM>. Accordingly, the synthetic image <NUM> may be labeled with the GT pose <NUM>.

According to an example, a second loss <NUM> may correspond to the second loss <NUM> described above with reference to <FIG>. In other words, the second loss <NUM> may be obtained based on a pose estimated based on the synthetic image <NUM> in a third network <NUM> and the GT pose <NUM> corresponding to the synthetic image <NUM>. For example, a gradient obtained based on the second loss <NUM> may be iteratively back-propagated to a first network <NUM> and the third network <NUM>, so that weights of the first network <NUM> and the third network <NUM> may be updated.

According to an example, the first network <NUM> may be trained adversarially to a first discriminator <NUM>, based on a first GAN loss <NUM>. The first discriminator <NUM> may correspond to the first discriminator <NUM> described above with reference to <FIG>. In other words, the first discriminator <NUM> may be trained to output a value (e.g., a value of "<NUM>") indicating the synthetic image <NUM> if an input feature is determined to be a feature extracted from the synthetic image <NUM>, and to output a value (e.g., a value of "<NUM>") indicating the real image <NUM> if the input feature is determined to be a feature extracted from the real image <NUM>.

According to an example, the first GAN loss <NUM> may correspond to the first GAN loss <NUM> described above with reference to <FIG>. A gradient obtained based on the first GAN loss <NUM> may be iteratively back-propagated to the first discriminator <NUM> and the first network <NUM>, so that weights of the first discriminator <NUM> and the first network <NUM> may be updated. For example, based on the first GAN loss <NUM>, the first discriminator <NUM> may be trained to output ground truth data of domain classification based on an input feature, and the first network <NUM> may be trained to extract features on which it is difficult for the first discriminator <NUM> to perform domain classification (e.g., the first network <NUM> may be trained to extract features such that an accuracy of domain classification performed by the first discriminator <NUM> based on the extracted features is below a predetermined threshold).

<FIG> illustrates an example of a configuration of a global localization apparatus.

Referring to <FIG>, a global localization apparatus <NUM> may include a processor <NUM> (e.g., one or more processors), a memory <NUM> (e.g., one or more memories), and an input/output (I/O) device <NUM>. The global localization apparatus <NUM> may include, for example, a user device (e.g., a smartphone, a personal computer (PC), a tablet PC, etc.), AR glasses, and a server.

The memory <NUM> in the global localization apparatus <NUM> may record a global localization model. The global localization model may include, for example, the global localization model described above with reference to <FIG>, or the global localization model described above with reference to <FIG> and <FIG>. The memory <NUM> may be, for example, a volatile memory or a non-volatile memory.

Hereinafter, the global localization model recorded in the memory <NUM> will be described as an example of the global localization model described above with reference to <FIG>. In other words, the global localization model may include a first network configured to extract a feature of an input image, a second network configured to estimate a coordinate map of the input image based on an output of the first network, and a pose estimator configured to estimate a pose corresponding to a global localization result based on an output of the second network.

As described above, the global localization model may be generated by a method of training the global localization model. For example, the global localization model may be generated by an operation of obtaining a synthetic data set based on 3D map data, an operation of iteratively back-propagating a gradient obtained based on at least one loss associated with the global localization model to update parameters of a neural network included in the global localization model, and an operation of storing the parameters of the neural network in the memory <NUM>. In an example, the neural network included in the global localization model may include a first network and a second network, as described above with reference to <FIG>. In another example, the neural network included in the global localization model may include a first network and a third network, as described above with reference to <FIG>.

According to an example, the memory <NUM> may store a program in which the above-described global localization model is implemented, and the processor <NUM> may execute the program stored in the memory <NUM> and control the global localization apparatus <NUM>. For example, the processor <NUM> may obtain the feature of the input image which is output from the first network by applying the input image to the first network, may obtain a coordinate map that corresponds to the input image and that is output from the second network by applying the feature to the second network, and may obtain a pose that corresponds to the input image and that is output from the pose estimator based on the coordinate map. The processor <NUM> may perform any or all operations described above with reference to <FIG>.

The global localization apparatus <NUM> may be connected to an external device (e.g., a PC, a server, or a network) through the I/O device <NUM> to exchange data with the external device. For example, the global localization apparatus <NUM> may receive an image through the I/O device <NUM>, and may output a pose estimated based on an image that is a result of the global localization model.

The global localization apparatuses, processors, memories, I/O devices, global localization apparatus <NUM>, processor <NUM>, memory <NUM>, I/O device <NUM>, and other devices, apparatuses, devices, units, modules, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD- Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and provide the instructions or software and any associated data, data files, and data structures to one or more processors or computers so that the one or more processors or computers can execute the instructions.

Claim 1:
A computer-implemented training method for global localization, the method comprising:
determining a synthetic data set based on three-dimensional, 3D, map data, the synthetic data set comprising a synthetic image captured by a virtual camera corresponding to an arbitrary pose and 3D coordinate data corresponding to each pixel of the synthetic image;
determining a first generative adversarial network, GAN, loss based on a first feature determined by applying the synthetic image to the first network and a second feature determined by applying a real image captured by a real camera to the first network;
determining a second GAN loss based on a first coordinate map determined by applying the first feature to the second network and a second coordinate map determined by applying the second feature to the second network;
determining a first loss based on the first coordinate map and the 3D coordinate data corresponding to the synthetic image;
determining a second loss based on a first pose estimated based on the first coordinate map and a pose of the virtual camera; and
training the first network based on the first loss, the second loss and the first GAN loss; and
training the second network based on the first loss, the second loss and the second GAN loss.