Patent Description:
When a vehicle or other object is moving, a navigation system for the vehicle receives radio waves from a satellite belonging to a plurality of global positioning systems (GPSs), and verifies a current location of the vehicle and a speed of the vehicle. The navigation system may calculate a three-dimensional (3D) current location of the vehicle including latitude, longitude and altitude information based on information received from a GPS receiver.

<CIT>, here acknowledged as prior art, describes methods and systems for determining a vehicle location and pose wherein a location and pose for the vehicle are first estimated. Based on this estimate, three-dimensional map data is retrieved and projected onto a camera image from a vehicle mounted camera. The projected three-dimensional map data is then compared to the camera image to determine the vehicle location and pose. This method includes edge detection in the camera image.

The invention provides an apparatus as defined in the appended independent apparatus claim, a processor-implemented method according to the appended independent method claim and a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform said method.

Throughout the drawings and the detailed description, the same drawing reference numerals refer to the same elements, features, and structures.

Herein, it is noted that use of the term "may" with respect to an example or embodiment, e.g., as to what an example or embodiment may include or implement, means that at least one example or embodiment exists in which such a feature is included or implemented while all examples and embodiments are not limited thereto.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains after an understanding of the disclosure of this application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of this application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

For example, global positioning system (GPS) signals for estimating the location of a target, for example, a moving object, such as a vehicle, may include a GPS location error of about <NUM> meters (m) to <NUM>. According to example embodiments disclosed herein, such a location error may be corrected based on other information.

<FIG> is a block diagram illustrating an example of a location estimation apparatus.

Referring to <FIG>, a location estimation apparatus <NUM> may include, for example, a sensor <NUM>, a processor <NUM>, and a memory <NUM>.

The sensor <NUM> generates sensing data. For example, the sensor <NUM> generates sensing data by sensing information used for estimating a location of a target. The target is, for example, an object of which a pose and a location are estimated by the location estimation apparatus <NUM>. In an example, the target may be the location estimation apparatus <NUM>. In another example, the location estimation apparatus <NUM> may be mounted on a vehicle and the target may be the vehicle. The information used for estimating a location includes various signals, for example, a global navigation satellite system (GNSS) signal (for example, a global positioning system (GPS) signal), an acceleration signal, a velocity signal, and an image signal. The processor <NUM> primarily measures initial localization information based on the sensing data. The initial localization information includes information indicating a location and a pose of the target measured coarsely. For example, the sensor <NUM> may include an inertial measurement unit (IMU) and a GNSS module. In such an example, the sensor <NUM> acquires an IMU signal and GNSS signals indicating an acceleration and an angular velocity of the target as the sensing data.

The IMU is also referred to as "inertial measurer". The IMU measures a change in pose, a change speed for location movement, and a displacement. The IMU includes a three-axis accelerometer that senses a translational motion, for example, an acceleration, and a three-axis gyroscope that senses a rotation motion, for example, an angular velocity. Since the IMU does not depend on external information, an acceleration signal and an angular velocity signal may often be stably collected. The GNSS module may receive signals transmitted from at least three artificial satellites to calculate locations of the satellites and the location estimation apparatus <NUM>, and may also be referred to as a GNSS. The GNSS module may measure an absolute location at <NUM> hertz (Hz) and may stably operate due to relatively low noise of the GNSS module. The IMU may measure a relative location at <NUM> and perform a measurement at a high speed.

Also, the sensor <NUM> senses image data. For example, the sensor <NUM> may include a camera sensor. The camera sensor receives rays in a visible light range and senses an intensity of the light. The camera sensor generates a color image by sensing an intensity of light corresponding to each of a red channel, a blue channel and a green channel. However, a wavelength band that may be sensed by the sensor <NUM> is not limited to a visible light band. The sensor <NUM> may further or alternatively include, for example, an infrared sensor that senses rays in an infrared band, a Lidar sensor, a RADAR sensor that senses a signal in an electromagnetic band, a thermal image sensor, or a depth sensor. The camera sensor generates image data by capturing an external view, for example, a front view from the location estimation apparatus <NUM>.

The processor <NUM> may generate final localization information of a target by correcting initial localization information of the target. The processor <NUM> may estimate the initial localization information and a rotation reference point to correct a localization error (for example, a rotation error and a location error) between an actual location and a pose of the target. Herein, the rotation reference point is a point used as a reference to correct a rotation error in localization information estimated for the target. The localization information may include information on a location, a pose and a movement, for example, a speed and an acceleration, of the target. The rotation reference point may be used in a localization, and may be used to, for example, correct a localization based on map data.

For example, a first rotation reference point may be a three-dimensional (3D) point corresponding to a vanishing point on a two-dimensional (2D) image plane and may indicate a point sufficiently distanced from an image sensor among points on a lane boundary line. The processor <NUM> may determine the first rotation reference point based on map data and localization information estimated for the target. A second rotation reference point may be a rotation reference point estimated based on image data sensed by the sensor <NUM>. The second rotation reference point may be, as a kind of a vanishing point, a point to which at least two parallel straight lines (for example, lane boundary lines) converge in a 3D space when the straight lines are projected onto a 2D image plane. The processor <NUM> may estimate the second rotation reference point from image data associated with a front view of the target.

In the following description of <FIG>, the first rotation reference point is referred to as an "initial reference point", and the second rotation reference point is referred to as a "rotation reference point".

Referring to <FIG>, the processor <NUM> corrects localization information (for example, initial localization information) based on a rotation parameter calculated based on the first rotation reference point and the second rotation reference point. The rotation parameter is a parameter to compensate for an error between initially estimated initial localization information and an actual location and a pose of the target, and may be, for example, a parameter to match the first rotation reference point and the second reference point. For example, the camera sensor included in the sensor <NUM> may be attached to the target, for example, a vehicle, such that the camera sensor has an optical axis parallel to a longitudinal axis of the target. In this example, since the camera sensor makes a same motion as a motion of the target, the image data sensed by the camera sensor includes visual information matching the location and pose of the target. Thus, the processor <NUM> may apply the rotation parameter so that the first rotation reference point corresponding to the initial localization information matches the second rotation reference point detected from the image data.

The memory <NUM> temporarily or permanently stores data used to perform a location estimation method. For example, the memory <NUM> stores a convolutional neural network (CNN) trained for a local filtering from image data and an attention neural network trained for a nonlocal filtering. However, the disclosure is not limited to the memory <NUM> storing a CNN and an attention neural network. For example, other types of neural networks, such as a recurrent neural network (RNN), a deep belief network, a fully connected network, a bi-directional neural network, a restricted Boltzmann machine, or a neural network including different or overlapping neural network portions respectively with full, convolutional, recurrent, and/or bi-directional connections may be stored on the memory <NUM> to perform the local filtering and the nonlocal filtering.

As described above, the location estimation apparatus <NUM> estimates initial localization information including an initial location and an initial pose of the target based on sensing data sensed by the sensor <NUM>. The initial localization information may include a pose error and a location error. An error in the initial localization information will be described in greater detail with reference to <FIG>.

<FIG> illustrate examples of a location estimation error.

A location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, measures initial localization information based on a GNSS signal and an acceleration signal, for example. <FIG> illustrates a situation in which an initial pose of the initial localization information has a rotation error Δθ with respect to an actual pose. The initial localization information indicates that a target including the location estimation apparatus has an arbitrary initial location and an arbitrary initial pose. Based on map data, for example, a high definition (HD) map, a map lane boundary line <NUM> corresponding to the initial localization information is as shown in <FIG>. An image lane boundary line <NUM> appearing in image data that is actually captured by an image sensor may be at an angle different from that of the map lane boundary line <NUM>. Thus, the location estimation apparatus may compensate for the above-described rotation error based on a rotation parameter.

The map data includes information associated with a map and includes, for example, information on a landmark. The landmark may be, for example, an object fixed at a predetermined geographical location to provide a driver with information used to drive a vehicle on a road. For example, road signs and traffic lights may belong to the landmark. For example, according to the Korean Road Traffic act, landmarks are classified into six classes, for example, a caution sign, a regulatory sign, an indication sign, an auxiliary sign, a road sign, and a signal sign. However, classification of the landmarks is not limited to the above description. Classes of the landmark may differ for each country, for example.

Also, herein, a lane boundary line (e.g., a line corresponding to the map lane boundary line <NUM> or the image lane boundary line <NUM>) is a line for defining a lane. The land boundary line may be, for example, a solid line or a broken line marked on a road surface, or a curb arranged along an outer edge of a road.

<FIG> illustrates an example situation in which an initial location in initial localization information has a horizontal location error Δx with respect to an actual location. Based on map data, a map lane boundary line <NUM> corresponding to the initial localization information is as shown in <FIG>. An image lane boundary line <NUM> appearing in image data that is actually captured by an image sensor may be at a different location from the map lane boundary line <NUM>.

<FIG> illustrates an example situation in which an initial pose in initial localization information has a rotation error <NUM>Δθy with respect to an actual pose about a lateral axis. Based on map data, a first rotation reference point <NUM> corresponding to a map lane boundary line corresponding to the initial localization information is as shown in <FIG>. A second rotation reference point <NUM> corresponding to an image lane boundary line appearing in image data that is actually captured by an image sensor corresponds to a different pose from that of the first rotation reference point <NUM>. Thus, the location estimation apparatus (e.g., the location estimation apparatus <NUM>) may compensate for the rotation error <NUM>Δθy based on a rotation parameter.

<FIG> is a flowchart illustrating an example of a location estimation method.

Referring to <FIG>, in operation <NUM>, a location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, calculates edge direction information of a plurality of edge component pixels extracted from an input image. An edge component pixel is a pixel indicating an edge component in the input image. The edge component pixels will be further described with reference to <FIG> below. The edge direction information is information indicating a direction (for example, a 2D direction in an image) in which an edge indicated by an edge component pixel faces. Example edge direction information will be further described with reference to <FIG> below. Although the input image as described herein is a grey scale image obtained by converting a color image, the disclosure is not limited to such an example. The input image may be, for example, a color image. When the location estimation apparatus is included in a vehicle, the input image may be a travelling image representing a front view from the vehicle while the vehicle is travelling. For example, when the input image is an RGB image and has <NUM> horizontal pixels and <NUM> vertical pixels, a total value of <NUM> × <NUM> × <NUM> pixels may be used as input data.

In operation <NUM>, the location estimation apparatus determines an inlier pixel among the plurality of edge component pixels based on the edge direction information and virtual horizontal line information. The virtual horizontal line information is straight line information corresponding to a horizontal line within an angle of view (or field of view) of a camera sensor as prior information. The virtual horizontal line may be a virtual horizon, a virtual horizontal line including or dependent on a determined estimated vanishing point, or an otherwise determined or pre-set (such as dependent on a pose of the capturing image sensor) virtual line. The virtual horizontal line information will be further described with reference to <FIG> below. The inlier pixel is a pixel used to estimate a rotation reference point among the plurality of edge component pixels, and an outlier pixel is one of pixels other than the inlier pixel among the plurality of edge component pixels. For example, the location estimation apparatus may calculate an intersection point between a straight line that is based on the edge direction information and a straight line that is based on the virtual horizontal line information at each of locations of the plurality of edge component pixels, and may determine, as an inlier pixel, pixels forming an intersection point at locations adjacent to each other. Examples of determining an inlier pixel will be further described with reference to <FIG> and <FIG> below.

In operation <NUM>, the location estimation apparatus estimates a rotation reference point based on the inlier pixel. In an example, the location estimation apparatus may determine a point in an image which has a minimum distance to straight lines corresponding to inlier pixels to be the above-described rotation reference point. In another example, the location estimation apparatus may determine an average point of intersection points between the virtual horizontal line and straight lines that are based on edge direction information of inlier pixels to be the rotation reference point. In another example, the location estimation apparatus may determine an average point of intersection points between straight lines that are based on edge direction information of inlier pixels to be the rotation reference point. However, the disclosure is not limited to the foregoing examples, and the rotation reference point may be determined by various schemes. The rotation reference point may be expressed by 2D coordinates (rx, ry), and rx and ry may have values in a range of [<NUM>, <NUM>] as relative coordinate values with respect to a size of an image.

In operation <NUM>, the location estimation apparatus corrects localization information of the location estimation apparatus based on the estimated rotation reference point. For example, the location estimation apparatus may estimate initial localization information based on a GNSS signal, an IMU signal and map data, and may also estimate a corresponding initial reference point. The location estimation apparatus may estimate a localization error (for example, a rotation error and a location error) between the initial reference point and the rotation reference point. An error between the initial reference point and the rotation reference point may correspond to an error between an initial pose and an actual pose. Thus, the location estimation apparatus may correct the initial localization information by rotating and/or moving the initial localization information by the localization error, and may generate final localization information. Thus, the location estimation apparatus may determine a more accurate pose from an initial pose estimated to be twisted unlike an actual pose.

<FIG> illustrates an example of a location estimation process.

Referring to <FIG>, in operation <NUM>, a location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, preprocesses an input image. The location estimation apparatus may preprocess the input image to extract edge component pixels. In operation <NUM>, the location estimation apparatus performs a local filtering on edge component pixels, to determine edge direction information for each of the edge component pixels. In operation <NUM>, the location estimation apparatus performs a nonlocal filtering on edge component pixels, to exclude outlier pixels and maintain inlier pixels. Though not limited thereto, examples of the preprocessing, the local filtering and the nonlocal filtering will be further described with reference to <FIG>, <FIG> and <FIG> below, respectively.

<FIG> illustrates an example of a preprocessing operation in a location estimation process.

Referring to <FIG>, a location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, acquires an input image <NUM>. For example, the location estimation apparatus may be mounted on a vehicle and may include a camera sensor installed to have an angle of view to show a front view of the vehicle. The camera sensor may capture a scene corresponding to the angle of view and acquire the input image <NUM>.

As illustrated in <FIG>, operation <NUM> in FIG. may include operations <NUM> and <NUM>. In operation <NUM> of <FIG>, the location estimation apparatus extracts a plurality of edge component pixels by preprocessing the input image <NUM>. For example, as shown in <FIG>, in operation <NUM>, the location estimation apparatus calculates gradient information for each pixel from the input image <NUM>. The gradient information is a value indicating an intensity change or a directional change of a color, for example. Gradient information at an arbitrary point in an image may be expressed as a 2D vector with differential values in a vertical direction and a horizontal direction. The location estimation apparatus may calculate a gradient value of a horizontal direction suitable for an estimation of a lane boundary line. The calculated gradient value may be a predetermined width value regardless of a width of a lane boundary line.

The location estimation apparatus extracts an edge component pixel having calculated gradient information exceeding a pre-defined threshold among pixels included in the input image <NUM>. For example, the location estimation apparatus calculates a magnitude of a gradient vector (for example, the 2D vector with the differential values) as gradient information of each of the pixels in the input image <NUM>, and determines a pixel having a gradient vector exceeding the threshold as an edge component pixel. Thus, the location estimation apparatus may extract, as an edge component pixel, a pixel of which an intensity value or a color value is rapidly changed in comparison to a neighboring pixel. An image including a plurality of edge component pixels is an edge component image <NUM>.

In operation <NUM>, the location estimation apparatus masks a portion of the plurality of edge component pixels. For example, the location estimation apparatus may exclude a pixel on a boundary from among the plurality of edge component pixels. The location estimation apparatus may exclude pixels corresponding to an area above a virtual horizontal line <NUM> in the input image <NUM>. Thus, the location estimation apparatus may acquire information focused on a road area. The virtual horizontal line <NUM> will be further described with reference to <FIG> below. Although the virtual horizontal line <NUM> is illustrated as a boundary in <FIG>, the disclosure is not limited to this example.

The plurality of edge component pixels extracted in operation <NUM> may indicate a lane boundary line, and a line along a lane, for example, a curb, or may indicate a line unrelated to the lane, for example, an outline of a building around a road. Since a rotation reference point <NUM>, as a vanishing point, is a point to which lines (for example, lane boundary lines) along a lane converge in an image, a line perpendicular to a road, such as an outline of a building, may be less relevant to the vanishing point. Thus, the location estimation apparatus may remove pixels having no determined or estimated relevance, or selectively chosen lessened relevance, to the vanishing point by excluding edge component pixels above the road area in operation <NUM>, as described above. An image from which pixels corresponding to such an area above the road area are removed is a masked image <NUM>.

The location estimation apparatus generates an edge direction image <NUM> through the local filtering in operation <NUM>, and generates an inlier image <NUM> through the nonlocal filtering in operation <NUM>. The location estimation apparatus estimates the rotation reference point <NUM> from the inlier image <NUM>, and acquires a result <NUM> to which the rotation reference point <NUM> is mapped.

The above-described operations are not necessarily performed in an order as shown in the drawings, and may be performed in parallel or in a different order, unless otherwise described. In an example, the location estimation apparatus may calculate a virtual horizontal line while calculating a gradient value. In another example, the location estimation apparatus may extract an edge component pixel after performing masking based on a virtual horizontal line.

<FIG> illustrates an example of virtual horizontal line information.

Referring to <FIG>, the above-described virtual horizontal line information of <FIG> indicates a straight line in which a horizontal line is virtually present in an image captured by a camera sensor <NUM>. The virtual horizontal line information may be prior information and may be determined in advance by a camera parameter (for example, camera calibration information). For example, the location estimation apparatus may determine a virtual horizontal line in an input image based on a roll parameter and a pitch parameter of the camera sensor <NUM> that captures the input image.

In <FIG>, an x-axis and a z-axis are parallel to the ground, and a y-axis is defined as an axis perpendicular to the ground. In an image <NUM> captured when a central axis of the camera sensor <NUM> is parallel to the z-axis, the virtual horizontal line appears as a straight line crossing a central portion of the image <NUM> as shown in <FIG>. The central axis of the camera sensor <NUM> may be referred to as an "optical axis. " When the central axis of the camera sensor <NUM> is assumed to be parallel to the z-axis, a roll axis <NUM> and a pitch axis <NUM> of the camera sensor <NUM> may be the z-axis and the x-axis, respectively. A roll parameter of the camera sensor <NUM> may be determined by a roll rotation <NUM> and a pitch parameter of the camera sensor <NUM> may be determined by a pitch rotation <NUM>. For example, the roll parameter may indicate an angle at which the camera sensor <NUM> is rotated about the roll axis <NUM> with respect to the ground, and the virtual horizontal line may appear as a straight line tilted by the roll parameter in an image <NUM>. The pitch parameter may indicate an angle at which the camera sensor <NUM> is rotated about the pitch axis <NUM> with respect to the ground and the virtual horizontal line may appear as a straight line at a location vertically moved by the pitch parameter in an image <NUM>.

<FIG> illustrates an example of a local filtering.

Referring to <FIG>, the location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, calculates edge direction information for each of a plurality of edge component pixels in the masked image <NUM> by performing the local filtering in operation <NUM> of <FIG>. The edge direction information indicates an edge direction indicated by an edge component pixel, and represents a direction of an edge in an image. For example, the edge direction information may be distinguished by a value of an angle formed with an arbitrary reference axis (for example, a horizontal axis) in an input image, however, the disclosure is not limited to this example.

The location estimation apparatus determines an edge direction for each of the plurality of edge component pixels based on an edge direction estimation model. The edge direction estimation model is a model to output an edge direction of each edge component pixel from the edge component image <NUM>, may have, for example, a machine learning structure, and may include, for example, a neural network. In <FIG>, the example edge direction estimation model is implemented as a CNN <NUM> including a plurality of kernels <NUM>. For example, the CNN <NUM> may include convolutional layers connected through the plurality of kernels <NUM>. A neural network may extract an edge direction from an image by mapping input data and output data that are in a nonlinear relationship based on a deep learning. This example deep learning may be a machine learning scheme configured to solve an edge direction extraction problem from a big data set, and may be used to map input data and output data through supervised learning or unsupervised learning. Thus, the CNN <NUM> may be a network that is trained based on training data in advance to output edge direction information corresponding to each edge component pixel from each edge component pixel.

For example, the location estimation apparatus calculates a convolution operation value of each of the plurality of edge component pixels based on kernels for each preset angle in the CNN <NUM>. Each of the plurality of kernels <NUM> may be a filter in which element values are disposed for extracting image information along or with respect to a predetermined edge direction through respective convolution operations and input information. For example, a kernel <NUM> among the plurality of kernels <NUM> may be a filter that includes element values configured to extract image information along or with respect to an edge direction at <NUM> degrees from a horizontal axis of an image. The location estimation apparatus may perform a convolution operation of a pixel at an arbitrary location and neighboring pixels in an image based on element values of a filter. For example, a convolution operation value of an arbitrary pixel calculated based on a kernel for extracting image information along or with respect to a predetermined edge direction (for example, a direction corresponding to <NUM> degrees) may be a value (for example, a probability value) indicating a possibility of the pixel being the predetermined edge direction.

The location estimation apparatus may calculate convolution operation values for each of pixels included in an image with respect to one kernel while sweeping each of the pixels. Also, the location estimation apparatus may calculate convolution operation values of the entire image for each of the plurality of kernels <NUM>. For example, the edge component image <NUM> may be an image including two-hundred and fifty-six (<NUM>) horizontal pixels and two-hundred and fifty-six.

(<NUM>) vertical pixels, and the CNN <NUM> may include one-hundred and eighty (<NUM>) kernels. Although the <NUM> kernels are filters to extract edge directions corresponding to angular samples (for example, a <NUM> degree, <NUM> degrees, <NUM> degrees or <NUM> degrees) by <NUM> degree from <NUM> degrees to <NUM> degrees, an angle that may be extracted by a kernel is not limited to these examples. The location estimation apparatus may calculate a convolution operation value of each of the <NUM> × <NUM> pixels for each kernel. Also, the location estimation apparatus may calculate a convolution operation value of each of the <NUM> × <NUM> pixels for each of the <NUM> kernels. Thus, output data of the CNN <NUM> may include tensor data with a size of <NUM> × <NUM> × <NUM>. In the described example, each element of the tensor data indicates edge probability information for <NUM> angular samples at a sample location of a 2D image with <NUM> × <NUM> pixels.

For example, a value indicated by tensor coordinates (x, y, θ) in the tensor data may be a value obtained by applying a kernel for extracting an edge direction of an angle θ to a pixel at coordinates (x, y) of the edge component image <NUM> through a convolution operation. Thus, when an edge direction of an angle indicated by the kernel at each 2D location of an image is detected, an element of the tensor data may be a non-zero value (for example, a probability of being a corresponding angle), and otherwise, may be a value of "<NUM>".

The location estimation apparatus determines, as edge direction information, an angle value indicated by a kernel corresponding to a highest convolution operation value among a plurality of convolution operation values calculated for each edge component pixel. As described above, each element value of tensor data may indicate a probability of a pixel at predetermined coordinates in an image being an edge direction of a predetermined angle. Thus, the highest convolution operation value among the plurality of convolution operation values calculated for each edge component pixel may correspond to a convolution operation value by a kernel with a highest probability of being an angle of a corresponding edge component pixel among a plurality of angles. Thus, the location estimation apparatus may estimate an edge direction with a highest possibility of being an angle of a corresponding edge component pixel for each edge component pixel.

<FIG> illustrates an example of a nonlocal filtering.

Referring to <FIG>, the location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an embodiment, determines an inlier pixel in the edge direction image <NUM> by performing the nonlocal filtering in the operation <NUM>. For example, the location estimation apparatus may maintain an inlier pixel by removing an outlier pixel from the plurality of edge component pixels based on intersection points between a virtual horizontal line and edge lines corresponding to edge direction information of the plurality of edge component pixels. The location estimation apparatus may remove an outlier pixel based on an adjacency level between the intersection points. The adjacency level between the intersection points is a degree by, or extent to, which the intersection points are adjacent to each other in the image, and may correspond to, for example, a determined and/or compared distance between the intersection points.

For example, in <FIG>, a first straight line <NUM> corresponding to an edge direction of a first edge component pixel <NUM> and a virtual horizontal line <NUM> may be determined to form a first intersection point <NUM>. A second straight line <NUM> corresponding to an edge direction of a second edge component pixel <NUM> and the virtual horizontal line <NUM> may be determined to form a second intersection point <NUM>. A third straight line <NUM> corresponding to an edge direction of a third edge component pixel <NUM> and the virtual horizontal line <NUM> form a third intersection point <NUM>. As shown in <FIG>, the first edge component pixel <NUM> and the second edge component pixel <NUM> correspond to points on lane boundary lines of a road. The third edge component pixel <NUM> corresponds to a point on a bridge rail. Locations at which the first intersection point <NUM> and the second intersection point <NUM> are detected are adjacent to each other, and the third intersection point <NUM> is detected at a relatively far away location. Thus, the location estimation apparatus may exclude the third edge component pixel <NUM>, forming the third intersection point <NUM>, as an outlier pixel. The location estimation apparatus may maintain, as inlier pixels, the first edge component pixel <NUM> forming the first intersection point <NUM> and the second edge component pixel <NUM> forming the second intersection point <NUM>.

An example of determining an inlier pixel and excluding an outlier pixel using an attention neural network is described in greater detail below with reference to <FIG>.

<FIG> illustrates an example of an attention neural network <NUM> that excludes an outlier pixel.

Referring to <FIG>, a location estimation apparatus (e.g., the location estimation apparatus <NUM>), according to an example embodiment, generates inlier data from which an outlier pixel is removed among a plurality of edge component pixels based on the attention neural network <NUM>. The attention neural network <NUM> may be a network that is trained based on training data in advance to output an inlier pixel remaining by removing an outlier pixel from the plurality of edge component pixels.

For example, the location estimation apparatus may determine an inlier pixel among the plurality of edge component pixels based on a similarity between an intersection point formed by an individual edge component pixel and an intersection point formed by another individual edge component pixel.

The location estimation apparatus may calculate a reference vector from an intersection point between an edge line and a virtual horizontal line <NUM> to a reference point <NUM> on an input image through a first reference vector transformation <NUM> and a second reference vector transformation <NUM> for each of a plurality of edge component pixels. The reference point <NUM> is designated as an arbitrary point on the input image. For example, a reference vector to express a similarity may be defined. As shown in <FIG>, an intersection point between the virtual horizontal line <NUM> and a straight line corresponding to an edge direction of each of the edge component pixels may be formed from each of the edge component pixels. A reference vector corresponding to an arbitrary edge component pixel may be a vector (for example, a unit vector) from the reference point <NUM> to an intersection point between the virtual horizontal line <NUM> and a straight line corresponding to an edge direction of each of the edge component pixels.

When a distance between intersection points corresponding to the plurality of edge component pixels decreases, a similarity between reference vectors corresponding to the plurality of edge component pixels increases. Thus, an inner product between the reference vectors may be implemented to determine the adjacency level of the intersection points. An intersection point between the virtual horizontal line <NUM> and a straight line corresponding to an edge direction of an i-th edge component pixel is represented by φi. A reference vector from the reference point <NUM> to the intersection point φi corresponding to the i-th edge component pixel is represented by <MAT>. In <FIG>, i, j, and k are integers that are greater than or equal to "<NUM>". The location estimation apparatus may transform the intersection point φi corresponding to the i-th edge component pixel to an i-th reference vector <MAT> through the first reference vector transformation <NUM>. Also, the location estimation apparatus may transform an intersection point φj corresponding to a j-th edge component pixel to a j-th reference vector <MAT> through the second reference vector transformation <NUM>.

The location estimation apparatus calculates a similarity between reference vectors of the plurality of edge component pixels as an adjacency level between the intersection points. For example, the location estimation apparatus may input, to the attention neural network <NUM>, the i-th reference vector <MAT> as a query value and input the j-th reference vector <MAT> as a key value. The location estimation apparatus may calculate the similarity between the reference vectors to be an attention value as shown in Equation <NUM> below, through a matrix multiplication <NUM>.

In Equation <NUM>, aj is an inner product between a transposed vector of the i-th reference vector <MAT> and the j-th reference vector <MAT>, and A() is an inner product operation. Also, q is a query value, and kj is a j-th key value. As described above, an inner product between reference vectors indicates the adjacency level between the intersection points, and accordingly aj is an adjacency level between the intersection point φi corresponding to the i-th edge component pixel and the intersection point φj corresponding to the j-th edge component pixel.

The location estimation apparatus may remove an outlier pixel by applying the calculated similarity to image coordinates of an edge component pixel and a vector value transformed from edge direction information. For example, the location estimation apparatus may apply an inner product value between reference vectors to a SoftMax operation <NUM>, to convert the similarity between the reference vectors to a weight based on Equations <NUM> and <NUM> shown below. <MAT><MAT>.

In Equation <NUM>, n is an integer that is greater than or equal to "<NUM>," and is a number of intersection points, and weights w<NUM> to wn are softmax function results for attention values based on Equation <NUM>. An individual weight is represented as shown in Equation <NUM>. In Equation <NUM>, a is indices indicating reference vectors and is an integer that is greater than or equal to "<NUM>". In Equation <NUM>, e is Euler's number, wj is a weight corresponding to the adjacency level between the intersection point φi corresponding to the i-th edge component pixel and the intersection point φj corresponding to the j-th edge component pixel. The location estimation apparatus may acquire weight vectors of [w<NUM>,. , wn] for all intersection points from Equation <NUM>.

Also, the location estimation apparatus may acquire a vector value by transforming image coordinates of an edge component pixel and edge direction information. For example, the location estimation apparatus may generate value data vj = gj(xj,yj,θj) through a g transformation <NUM> from the j-th edge component pixel. The location estimation apparatus may input a vector value vj generated through the g transformation <NUM> to the attention neural network <NUM>.

The location estimation apparatus may perform a matrix multiplication <NUM> between the vector value vj and the weight vector of Equation <NUM> and may generate an output value as shown in Equation <NUM> below.

In Equation <NUM>, oi is a value of an i-th edge component pixel. In an example, when an intersection point corresponding to the i-th edge component pixel is located far from intersection points corresponding to other edge component pixels, a value corresponding to the i-th edge component pixel may converge to "<NUM>", and accordingly an outlier pixel may be excluded. In another example, when the intersection point corresponding to the i-th edge component pixel is adjacent to the intersection points corresponding to the other edge component pixels, a value corresponding to the i-th edge component pixel may be a nonzero value due to its great weight, and thus an inlier pixel may be maintained. Thus, the location estimation apparatus may maintain an inlier pixel by excluding an outlier pixel from among the plurality of edge component pixels through the attention neural network <NUM> and Equations <NUM> through <NUM>.

As described above, the location estimation apparatus may determine edge component pixels that form intersection points with the virtual horizontal line <NUM> that are adjacent to each other to be inlier pixels. The location estimation apparatus may estimate the rotation reference point <NUM> based on an inlier image including inlier pixels. For example, the location estimation apparatus may determine a point with a minimum distance to an edge line corresponding to an inlier pixel in the input image to be the rotation reference point <NUM>. An edge line is a straight line passing through inlier pixels and may be, for example, a straight line having a slope corresponding to an edge direction indicated by inlier pixels. The location estimation apparatus may determine a point at which a distance between edge lines and a horizontal line is minimized to be the rotation reference point <NUM>. However, determining of the rotation reference point <NUM> is not limited to the above description, and various schemes may be used to determine the rotation reference point <NUM>. Although an example of directly estimating a rotation reference point from an inlier pixel has been described in <FIG>, the disclosure is not limited to this example.

An example of performing a line fitting and estimating a rotation reference point is described below with reference to <FIG>.

<FIG> illustrates an example of a line fitting. A location estimation apparatus, according to an example embodiment (e.g., the location estimation apparatus <NUM>), performs a line fitting on an inlier pixel among a plurality of edge component pixels. The location estimation apparatus clusters the plurality of edge component pixels into at least one straight line group and determines a representative edge line for each of the at least one straight line group, to perform the line fitting.

<FIG> illustrates a first edge component pixel <NUM> and a second edge component pixel <NUM> in a portion <NUM> of an inlier image <NUM>, according to an example. A first edge line <NUM> by the first edge component pixel <NUM> and a second edge line <NUM> by the second edge component pixel <NUM> form angles <NUM> and <NUM> that are different from each other with respect to an arbitrary reference axis. Since an edge direction of the first edge line <NUM> and an edge direction of the second edge line <NUM> are similar to each other, the location estimation apparatus clusters the first edge line <NUM> and the second edge line <NUM> into the same straight line group, and determines a representative edge line <NUM> of the straight line group to which the first edge line <NUM> and the second edge line <NUM> belong. The representative edge line <NUM> has an average edge direction of edge lines included in the straight line group, however, the disclosure is not limited to such an example.

For the above-described clustering, the location estimation apparatus converts an image including a plurality of inlier pixels to a bird's eye view image. Edge lines with similar edge directions among edge lines projected onto the bird's eye view image may be distinguished more clearly from each other than edge lines with different edge directions. The location estimation apparatus clusters the plurality of inlier pixels into at least one straight line group based on lines corresponding to the plurality of inlier pixels in the bird's eye view image. The location estimation apparatus determines an edge line representative of the at least one straight line group.

The location estimation apparatus estimates a rotation reference point based on an edge line obtained by the line fitting. For example, the location estimation apparatus may obtain representative edge lines for each individual straight line group by the line fitting, and determine a point having a minimized distance to the representative edge lines as a rotation reference point. An example of correcting localization information based on a rotation reference point determined by the method of <FIG> is described below with reference to <FIG>.

<FIG> is a diagram illustrating an example of a rotation reference point error between initial localization information estimated for a location estimation apparatus and an actual pose of the location estimation apparatus.

In the example of <FIG>, it is assumed that a target <NUM> of a pose based on initial localization information is located obliquely with respect to a lane boundary. The pose of the target <NUM> is virtually illustrated based on the initial localization information. A location estimation apparatus, which may be disposed on or in the target, for example, determines a point located beyond a threshold distance from the target <NUM> among points defining a boundary line of a lane in which the target <NUM> is travelling, to be an initial reference point <NUM>. When the initial reference point <NUM> is projected onto an image plane corresponding to an angle of view of a sensor <NUM> attached to the target <NUM> in the pose based on the initial localization information, a point <NUM> to which the initial reference point <NUM> is projected is indicated as shown in <FIG>.

In the example of <FIG>, unlike the initial localization information, an actual target <NUM> is oriented parallel to the lane boundary. A sensor <NUM> of the actual target <NUM> captures a front-view image from the actual target <NUM>. The location estimation apparatus detects a rotation reference point <NUM> from image data <NUM> as described with reference to <FIG>.

As described above, due to an error in the initial localization information, a point to which the initial reference point <NUM> is projected directly on the image plane and the rotation reference point <NUM> may appear at different points. Thus, a difference <NUM> between the point to which the initial reference point <NUM> is projected on the image plane and the rotation reference point <NUM> may correspond to the error, for example, a rotation error and a location error, in the initial localization information.

By using a rotation parameter for correcting a localization error, the location estimation apparatus may correct the initial localization information. The location estimation apparatus may generate final localization information of the location estimation apparatus by correcting at least one of the location error and the rotation error in the localization information based on a rotation reference point.

<FIG> illustrates an example of an operation of a location estimation apparatus <NUM>, according to an embodiment, mounted on a vehicle.

Referring to <FIG>, the location estimation apparatus <NUM> includes, for example, a sensor <NUM>, a processor <NUM>, a memory <NUM> and a display <NUM>.

The sensor <NUM> acquires an input image. The sensor <NUM> may be a camera sensor <NUM> configured to capture an external scene <NUM>. The camera sensor <NUM> generates a color image as an input image. The color image includes an image corresponding to each of a red channel, a blue channel and a green channel, however, a color space is not limited to an RGB color space. The camera sensor <NUM> is located to capture an outside environment of the location estimation apparatus <NUM>.

The processor <NUM> calculates edge direction information of a plurality of edge component pixels extracted from the input image, determines an inlier pixel among the plurality of edge component pixels based on edge direction information and virtual horizontal line information, estimates a rotation reference point based on the inlier pixel, and corrects localization information of the location estimation apparatus <NUM> based on the estimated rotation reference point.

The memory <NUM> temporarily or permanently stores data required to perform a location estimation method.

The display <NUM> performs a visualization by mapping a virtual content object <NUM> to coordinates of the display <NUM> determined based on final localization information of the location estimation apparatus <NUM>. The display <NUM> transfers the external scene <NUM> to a user. For example, the display <NUM> may provide the user with the external scene <NUM> captured by the camera sensor <NUM>. However, the disclosure is not limited to such an example. In an example, the display <NUM> may be implemented as a transparent display <NUM> to transfer the external scene <NUM> to the user by allowing the external scene <NUM> to be seen through the transparent display <NUM>. In another example, when the location estimation apparatus <NUM> is mounted on a vehicle, the transparent display <NUM> may be combined with a windshield glass of the vehicle.

The processor <NUM> corrects initial localization information of the location estimation apparatus <NUM> based on the rotation reference point and generates the final localization information of the location estimation apparatus <NUM>. Thus, the processor <NUM> may match an estimated location to an actual location <NUM> of the location estimation apparatus <NUM>. For example, the location estimation apparatus <NUM> may control an operation of an apparatus based on localization information corrected based on the rotation reference point.

For example, the processor <NUM> may determine a location of the virtual content object <NUM> on the display <NUM> based on the final localization information of the location estimation apparatus <NUM>. For example, the display <NUM> may perform a visualization by overlaying the virtual content object <NUM> on the external scene <NUM> as shown in <FIG>. When the display <NUM> visualizes the external scene <NUM> captured by the camera sensor <NUM>, the processor <NUM> may perform a visualization through the display <NUM> by overlaying the virtual content object <NUM> on the external scene <NUM> that is rendered. When the display <NUM> is implemented as a transparent display <NUM>, the processor <NUM> may map a location of the virtual content object <NUM> to actual physical location coordinates of the external scene <NUM>, and may visualize the virtual content object <NUM> at a point on the display <NUM> corresponding to the mapped location. The virtual content object <NUM> may be, for example, an object that guides information associated with travelling of a vehicle, and may indicate a variety of information associated with a travelling direction and a direction indication (for example, a left turn point and a right turn point), a speed limit and a current speed of the vehicle.

For example, when the location estimation apparatus <NUM> is included in a vehicle, the location estimation apparatus <NUM> may control an operation of the vehicle based on the final localization information. The location estimation apparatus <NUM> may determine, based on the final localization information, a lane on which the vehicle is located among a plurality of lanes defined by a lane boundary line <NUM> on a road in which the vehicle is travelling, and may detect a distance between the location estimation apparatus <NUM> and a neighboring obstacle (for example, a preceding object). The location estimation apparatus <NUM> may control any one or any combination of an acceleration, a speed, and a steering of the vehicle based on the distance detected based on the final localization information. For example, when a distance between the location estimation apparatus <NUM> and a preceding object decreases, the location estimation apparatus <NUM> may reduce the speed of the vehicle. When the distance between the location estimation apparatus <NUM> and the preceding object increases, the location estimation apparatus <NUM> may increase the speed of the vehicle. Also, the location estimation apparatus <NUM> may change the steering (for example, a right turn and/or a left turn) of the vehicle based on final localization information of the vehicle along a preset travel route.

An example of using a rotation reference point in a localization based on map data and a travelling image in an augmented reality (AR) environment has been described with reference to <FIG>. However, the disclosure is not limited to an application to the AR in a travelling environment. For example, a rotation reference point estimation scheme according to examples to enhance a localization performance even in various environments, for example, an AR navigation during walking may be used.

For example, the location estimation apparatus may also enable an estimation performance for input data with a different feature from that of data used for training of a neural network. Thus, the location estimation apparatus may exhibit a rotation reference point estimation performance that is robust against various environments. For example, the location estimation apparatus may accurately estimate the rotation reference point regardless of a unique characteristic of a camera sensor, a characteristic according to a travelling area (for example, a city area or a country area), image characteristics of a travelling city and/or country, and an illuminance characteristic. This is because edge direction information used to estimate the rotation reference point is less affected by a characteristic of a camera sensor, an area or an illumination. The location estimation apparatus may estimate the rotation reference point from a single image instead of performing a separate operation of a motion vector. The above-described location estimation method may be implemented by, for example, software, a web service, and a chip (e.g., a processor and a memory).

Parameters of the machine learning modules, e.g., of any, any combinations, or all of the neural networks, are stored in one or more of the memories and are implemented by the reading and implementing of the read parameters by the one or more processors.

The, processors, the processors <NUM> and <NUM>, the sensors, the GPS receivers, the GNSS modules, the IMUs, the memories, and the memories <NUM> and <NUM> in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

While this disclosure includes specific examples, it will be apparent after an understanding of the disclosure of this application that various changes in form and details may be made in these examples without departing from the scope of the claims and at least in or for some jurisdictions including equivalents of features defined in appended claims. The examples described herein are to be considered in a descriptive sense only, and not for purposes of limitation. Descriptions of features or aspects in each example are to be considered as being applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, and/or replaced or supplemented by other components. Therefore, the scope of the disclosure is defined not by the detailed description, but by the claims, and all variations within the scope of the claims are to be construed as being included in the disclosure.

Claim 1:
A location estimation apparatus (<NUM>; <NUM>), comprising:
a sensor (<NUM>; <NUM>; <NUM>, <NUM>; <NUM>) configured to acquire an input image (<NUM>); and
a processor (<NUM>; <NUM>) configured to:
calculate edge direction information of edge component pixels (<NUM>, <NUM>, <NUM>; <NUM>, <NUM>) extracted from the input image (<NUM>);
characterized in that the processor (<NUM>; <NUM>) is further configured to:
determine an inlier pixel (<NUM>, <NUM>) among the edge component pixels (<NUM>, <NUM>, <NUM>) by removing an outlier pixel (<NUM>) among the edge component pixels (<NUM>, <NUM>, <NUM>) based on an adjacency level between intersection points (<NUM>, <NUM>, <NUM>) of a virtual horizontal line (<NUM>; <NUM>; <NUM>), which corresponds to a virtual horizon in the input image (<NUM>), and edge lines (<NUM>, <NUM>, <NUM>) corresponding to the edge direction information of the edge component pixels (<NUM>, <NUM>, <NUM>, wherein the adjacency level is a degree by which the intersection points are adjacent to each other;
estimate a rotation reference point (<NUM>) based on the inlier pixel (<NUM>, <NUM>); and correct initial localization information of the apparatus (<NUM>; <NUM>) by applying a rotation parameter to match an initial reference point, which is based on map data and the initial localization information, and the estimated rotation reference point (<NUM>).