Patent Description:
An image-based positioning method may include various methods, such as, for example, structure from motion (SfM), simultaneous localization and mapping (SLAM), and visual odometry (VO). The positioning methods may perform positioning by calculate a depth value of each pixel in a photographed image and a position of a camera photographing the image using homography. Homography may represent a relationship between pixels of consecutive images acquired by the camera. However, in the positioning method, an accuracy may be reduced due to degradation in tracking performance and/or a tracking loss that occurs when tracking a movement of a moving object. A lot of resources may be consumed in repetitively selecting and/or removing an area to be an outlier when selecting feature points from images, i.e., an area corresponding to the moving object. The applicant disclosed in <CIT> an apparatus and method for estimating camera pose, comprising the steps of:.

In one general aspect, there is provided a visual inertial odometry method of obtaining pose information according to the appended independent method claim.

In another general aspect, there is provided an visual inertial odometry apparatus for outputting pose information according to the appended independent apparatus claim.

If the specification states that one component is "connected," "coupled," or "joined" to a second component, the first component may be directly "connected," "coupled," or "joined" to the second component, or a third component may be "connected," "coupled," or "joined" between the first component and the second component. However, if the specification states that a first component is "directly connected" or "directly joined" to a second component, a third component may not be "connected" or "joined" between the first component and the second component. Similar expressions, for example, "between" and "immediately between" and "adjacent to" and "immediately adjacent to," are also to be construed in this manner.

When a part is connected to another part, it includes not only a case where the part is directly connected but also a case where the part is connected with another part in between. Also, when a part includes a constituent element, other elements may also be included in the part, instead of the other elements being excluded, unless specifically stated otherwise. Although terms such as "first," "second," "third" "A," "B," (a), and (b) may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms.

Hereinafter, the examples will be described in detail with reference to the accompanying drawings, wherein like drawing reference numerals are used for like elements.

<FIG> is a diagram illustrating an example of a process of outputting pose information using an outputting apparatus. <FIG> illustrates a configuration and an operation of an apparatus <NUM> for outputting pose information based on a visual inertial odometry (VIO) (hereinafter, referred to as "outputting apparatus"). Pose information includes any one or any combination of rotation information associated with a rotation of the outputting apparatus, translation information associated with a translation of the outputting apparatus.

In general, a visual odometry estimates pose information using a visual function. However, when a dynamic object is present in an image, the pose information may not be accurately estimated because it is difficult to distinguish whether a camera is moving or an ambient object is moving using only information in an image that is captured by the camera. Also, it may be difficult to estimate a sudden movement using visual odometry. In contrast, an inertial measurement device such as an inertial measurement unit (IMU) sensor detects such sudden movement, but may not be suitable for continuous detection of a motion over long time intervals due to a characteristic of error accumulation.

Accordingly, the pose information may be more accurately calculated by distinguishing between a motion of the camera (or a motion of a static object in an image due to the motion of the camera) and a motion of the dynamic object using the information on the image captured by the camera together with motion information sensed by the inertial measurement device. The inertial measurement device provides the motion information irrespective of the image, and thus, may increase accuracy and an operation speed of the pose information by using the motion information and the information on the image.

In an example, the outputting apparatus <NUM> includes a first sensor <NUM>, a second sensor <NUM>, a filter <NUM>, a point selector <NUM>, a rotation information acquirer, for example, a VIO rotation acquirer <NUM>, and a translation information acquirer, for example, a VIO translation acquirer <NUM>. Here, operations of the filter <NUM>, the point selector <NUM>, the VIO rotation acquirer <NUM>, and the VIO translation acquirer <NUM> are performed by a processor of the outputting apparatus <NUM>. Further details on the processor, such as, for example, processor <NUM> of <FIG>, is provided below.

The outputting apparatus <NUM> outputs pose information by calculating an amount of change in <NUM>-degree-of-freedom (DOF) pose between image frames based on a VIO. In an example, the 6DoF pose includes three-dimensional (3D) translation information T and 3D rotation (orientation) information R. The translation information T may be understood as a positional change and the rotation information R may be understood as an orientation change.

The first sensor <NUM> captures a plurality of image frames. The first sensor <NUM> is, for example, a camera, an image sensor, or a vision sensor. In an example, the first sensor <NUM> is attached to various positions on a vehicle, such as, for example, a windshield, a dashboard, and a rear-view mirror. The first sensor captures a driving image of a front view of the vehicle. For example, the first sensor <NUM> captures a first frame at a first time point and captures another frame at an nth time point.

The vehicle described herein refers to any mode of transportation, delivery, or communication such as, for example, an automobile, a truck, a tractor, a scooter, a motorcycle, a cycle, an amphibious vehicle, a snowmobile, a boat, a public transit vehicle, a bus, a monorail, a train, a tram, an autonomous or automated driving vehicle, an intelligent vehicle, a self-driving vehicle, an unmanned aerial vehicle, an electric vehicle (EV), a hybrid vehicle, a smart mobility device, or a drone. In an example, the smart mobility device includes mobility devices such as, for example, electric wheels, electric kickboard, and electric bike. In an example, vehicles include motorized and non-motorized vehicles, for example, a vehicle with a power engine (for example, a cultivator or a motorcycle), a bicycle or a handcart.

In addition to the vehicle described herein, methods and apparatuses described herein may be included in various other devices, such as, for example, a smart phone, a walking assistance device, a wearable device, a security device, a robot, a mobile terminal, and various Internet of Things (IoT) devices.

The second sensor <NUM> senses motion information. The second sensor <NUM> may be a single sensor or a plurality of sensors. The second sensor <NUM> includes, for example, an IMU sensor, a global positioning system (GPS) sensor, and an on-board diagnostics (OBD). The second sensor <NUM> senses first motion information at a first time point, senses second motion information at a second time point, and senses third motion information at an nth time point.

The point selector <NUM> selects an area to be used for calculating the pose information from the frame captured by the first sensor <NUM>. For example, the point selector <NUM> selects points (for example, first points) corresponding to a calculation area of a VIO from the frames.

The point selector <NUM> selects the points based on various methods. In one example, the point selector <NUM> selects the points evenly in an area of the plurality of frames. In another example, the point selector <NUM> selects the point for each of various classes such as a blob, an edge, and a corner included in the plurality of frames. In another example, the point selector <NUM> selects a smaller number of points from a center of the image toward a boundary (an outer edge) than the center of the image because of error occurring when restoring an image that is distorted due to a lens of a camera capturing frames. An operation of the point selector <NUM> will be further described with reference to <FIG>.

The filter <NUM> estimates states of an ego-motion corresponding to the plurality of frames based on the motion information sensed by the second sensor <NUM>. The ego-motion may be a 3D motion of a device in an environment and includes, for example, an orientation motion and a translation motion of the outputting apparatus. The filter <NUM> estimates a state of the ego-motion based on the motion information, which in an example of an observed value or a measurement. The filter <NUM> outputs information on the ego-motion based on the estimated state. For example, the filter <NUM> estimates rotation information Rinitial between the plurality of frames based on the estimated states. In an example, the filter <NUM> transmits the rotation information Rinitial to the VIO rotation acquirer <NUM>.

In an example, a time point at which the first sensor <NUM> captures the plurality of image frames is not synchronized with a time point at which the second sensor <NUM> senses the motion information. For example, an interval between time points when the first sensor <NUM> captures the plurality of image frames may be greater than an interval between time points when the second sensor <NUM> senses the motion information.

The filter <NUM> selects time points corresponding to time points at which the first sensor <NUM> captures the plurality of frames from a plurality of time points at which the second sensor <NUM> senses the motion information, thereby estimating and transmitting rotation information between the selected time points.

The filter <NUM> estimates, for example, a speed, an IMU bias (for example, an angular velocity and an acceleration), and an absolute position based on the motion information sensed by the second sensor <NUM>. In an example, the motion information includes, for example, three-axis rotation information associated with pitch, roll, and yaw axes detected by a gyroscope of an IMU sensor, information on a movement in three axes, that is, forward/back, up/down, and left/right axes in a 3D space sensed by an accelerometer, a GPS position, an OBD wheel odometry, steering angle information, and pose information. An operation of the filter <NUM> will be described with reference to <FIG> and <FIG>.

The VIO rotation acquirer <NUM> receives the rotation information Rinitial from the filter <NUM>. In an example, the VIO rotation acquirer <NUM> excludes second points corresponding to an outlier from the first points selected by the point selector <NUM> based on the rotation information Rinitial. In an example, the VIO rotation acquirer <NUM> estimates rotation information Restimate based on third points obtained by excluding the second points from the first points. The VIO rotation acquirer <NUM> transmits the estimated rotation information Restimate to the filter <NUM>. In an example, the second points corresponding to the outlier may be points corresponding to a dynamic object.

Hereinafter, the "first points" are points selected from an area for calculating pose information in frames captured by the first sensor <NUM>. The "second points" are points corresponding to a dynamic object from among the first points, for example, points to be removed to recognize a motion of a static object included in the frames. The second points may also be referred to as an "outlier" because the second points are removed for calculation of the pose information. The "third points" are points that remain after the second points are removed from the first points, for example, points corresponding to the static object. The third points may also be referred to as an "inlier".

The VIO rotation acquirer <NUM> performs an outlier rejection <NUM> and a rotation evaluation <NUM>.

In outlier rejection <NUM>, the VIO rotation acquirer <NUM> rejects the outlier (the second points corresponding to the dynamic object) from the first points based on the rotation information Rinitial estimated based on the motion information received from the filter <NUM>.

In rotation evaluation <NUM>, the VIO rotation acquirer <NUM> estimates the rotation information Restimate between the plurality of frames based on the third points transferred through the outlier rejection <NUM>. In an example, to evaluate (or estimate) rotation information based on visual information, the VIO rotation acquirer <NUM> directly uses a pixel intensity of a point in an image. Although directly using the pixel intensity of the point in the image is described in the present examples, other methods to evaluate (or estimate) rotation information based on visual information, such as, for example using a feature extracted from an image, may be used without departing from the scope of the illustrative examples described.

The VIO rotation acquirer <NUM> transfers, to the filter <NUM>, the rotation information Restimate estimated based on the third points. An operation of the VIO rotation acquirer <NUM> will be further described with reference to <FIG>.

The filter <NUM> receives the rotation information Restimate from the VIO rotation acquirer <NUM>. The filter <NUM> corrects the rotation information Rinitial based on the rotation information Restimate and outputs rotation information R*. For example, when the rotation information Restimate estimated based on the third points is received from the VIO rotation acquirer <NUM>, the filter <NUM> updates the state of the ego-motion based on a difference between the rotation information Restimate and the rotation information Rinitial estimated based on the motion information. The filter <NUM> outputs the rotation information R* based on the updated state. The rotation information may be expressed in a form of, for example, a quaternion or a matrix.

In an example, an operation of the filter <NUM> to estimate the rotation information Rinitial based on the second sensor <NUM> is distinguished from an operation of the VIO rotation acquirer <NUM> to estimate the rotation information Restimate based on the first sensor <NUM>. Thus, the example illustrated in <FIG> provides a characteristic of a loosely-coupled system where the processing of inertial information and the processing of visual information are loosely-coupled to each other.

In an example, the rotation information Rinitial is used to select points for estimating the rotation information Restimate and the rotation information Restimate is used to correct the rotation information Rinitial, whereby the rotation information R* is output. Thus, the example illustrated in <FIG> provides a characteristic of a tightly-coupled system where the processing of inertial information and the processing of visual information are tightly-coupled to each other.

Accordingly, the examples provide advantages of both the loosely-coupled system and the tightly-coupled system. For example, an operation speed is improved by removing the second points, which is difficult to remove using only the image frames of the first sensor <NUM>, in a dynamic object area based on state information of the second sensor <NUM> integrated through the filter <NUM>. Also, a visual information-based estimation result is provided to an inertial information-based filter, so that the accuracy of the estimation is improved.

The VIO translation acquirer <NUM> calculates an amount of change in motion information. The VIO translation acquirer <NUM> calculates translation information T* between the plurality of frames based on the rotation information R* corrected in the filter and the third points. An operation of the VIO translation acquirer <NUM> will be further described with reference to <FIG>.

The outputting apparatus <NUM> outputs the rotation information R* and the translation information T*.

Examples set forth hereinafter may be used to provide pose information, generate visual information to assist steering of an autonomous vehicle, or provide various control information for driving in an augmented reality (AR) navigation system of a smart vehicle. The examples may be used to provide visual information and assist safe and pleasant driving in a device including an intelligent system such as a head-up display (HUD) installed for driving assistance or fully autonomous driving of a vehicle. The examples may be applied to, for example, an autonomous vehicle, an intelligent vehicle, a smart phone, and a mobile device.

<FIG> is a diagram illustrating an example of a method of outputting pose information. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, in operation <NUM>, an apparatus for outputting pose information (hereinafter, referred to as "outputting apparatus") selects first points from frames captured by a first sensor. A method of selecting the first points using the outputting apparatus will be further described with reference to <FIG>.

In operation <NUM>, the outputting apparatus estimates rotation information between the frames based on motion information sensed by a second sensor. The outputting apparatus estimates the rotation information between the frames using, for example, a Kalman filter, an extended Kalman filter, an iterated extended Kalman filter, an unscented Kalman filter, and a particle filter. A method of estimating the rotation information between the plurality of frames using the outputting apparatus will be further described with reference to <FIG>.

In operation <NUM>, the outputting apparatus corrects the estimated rotation information based on third points. The third points are the remaining point in the first points when second points corresponding to a dynamic object is excluded. A method of correcting the estimated rotation information using the outputting apparatus will be further described with reference to <FIG> and <FIG>.

In operation <NUM>, the outputting apparatus calculates translation information between the frames based on the third points and the corrected rotation information. The outputting apparatus determines the translation information based on the corrected rotation information such that an energy function associated with a difference in intensity between the frames to which the third points belongs is less than a target value. The energy function includes functions, such as, for example, a Gauss-Newton optimization function that approximates a nonlinear function locally with a linear function to obtain a solution and a Levenberg-Marquardt optimization function that finds repeatedly optimized solutions starting from the initial solution when the initial solution is provided. In an example, the Gauss-Newton optimization function may obtain a solution using both the gradient and the curvature of a function. A process of calculating the motion information using the outputting apparatus will be further described with reference to <FIG>.

In operation <NUM>, the outputting apparatus outputs the rotation information corrected in operation <NUM> and the translation information calculated in operation <NUM>.

<FIG> is a diagram illustrating an example of a method of estimating rotation information between a plurality of frames. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, an outputting apparatus estimates states of an ego-motion corresponding to a plurality of frames based on motion information in operation <NUM>. In an example, the outputting apparatus estimates a state of a second time point based on a state of a first time point and motion information of the second time point. In an example, the second time point corresponds to a current time point and the first time point corresponds to a previous time point. The states of the ego-motion include a first state for estimating rotation information and a second state for estimating translation information. In an example, the rotation information is estimated based on the first state and the translation information is estimated based on the second state.

In operation <NUM>, the outputting apparatus estimates rotation information between the plurality of frames based on the states estimated in operation <NUM>. A method of estimating the states of the ego-motion and estimating the rotation information between the plurality of frames using the filter <NUM> of the outputting apparatus will be further described with reference to <FIG> and <FIG>.

<FIG> is a diagram illustrating an example of a method of estimating rotation information between a plurality of frames. <FIG> illustrates states (Statei-<NUM>, Statei, and Statei') of an ego-motion estimated by the filter <NUM> based on a plurality of frames.

The filter <NUM> estimates states of an ego-motion corresponding to a plurality of frames based on sensor inputs ii. Here, the sensor inputs ii may correspond to motion information sensed by the second sensor as described above.

In an example, the filter <NUM> estimates a state at a time point i (Statei) based on a state at a time point i-<NUM> (Statei-<NUM>) and motion information (ii) at the time point i. In an example, the filter <NUM> "estimates" rotation information Rinitial between a plurality of frames based on the estimated states (the state at the time point i-<NUM> (Statei-<NUM>) and the state at the time point i (Statei)). The rotation information Rinitial is calculated as a rotation change amount ΔR between an output value of the state at the time point i-<NUM> (Statei-<NUM>) and an output value of the state at the time point i (Statei). In an example, the state of the time point i-<NUM> (Statei-<NUM>) corresponds to a time point at which a first frame is captured by the first sensor <NUM> and the state of the time point i (Statei) corresponds to a time point at which a second frame is captured by the first sensor <NUM>.

The filter <NUM> receives rotation information Restimate estimated based on third points from the VIO rotation acquirer <NUM>. In this example, the rotation information Restimate estimated based on the third points is used to correct the rotation information Rinitial. The rotation information Restimate corresponds to, for example, a rotation change amount ΔR between the first frame captured at the time point i-<NUM> and the second frame captured at the time point i.

The filter <NUM> "corrects" the rotation information Rinitial based on the rotation information Restimate and outputs rotation information R*. The filter <NUM> updates the state of the ego-motion at the time point i (Statei) to be a state (Statei') based on a difference between the rotation information Restimate estimated based on the third point and the rotation information Rinitial estimated based on motion information. The filter <NUM> outputs the rotation information R* based on the updated state (Statei'). For example, the rotation information R* is calculated as a rotation change amount ΔR between an output value Oi-<NUM> of the state (Statei-<NUM>) and an output value Oi' of the state (Statei').

<FIG> is a diagram illustrates an example of a method of estimating rotation information in a Kalman filter. <FIG> illustrates an estimation process <NUM> of "estimating" rotation information Rinitial between a plurality of frames in the Kalman filter and a correction process <NUM> of correcting rotation information Rinitial. Although an operation of the Kalman filter which is an example of the filter <NUM> is described for brevity of description, a type of filter is not limited thereto. In addition to the Kalman filter, various filters such as, for example, a particle filter may be used without departing from the scope of the illustrative examples described.

In the estimation process <NUM>, an operation of the Kalman filter (hereinafter, referred to as "filter") is as follows.

The filter determines a current state x̂k based on a previous state x̂k-<NUM> and a current input uk. The filter also calculates an error covariance Pk. In the estimation process <NUM>, an output value Hx̂k. is determined based on the current state x̂k. Since a state estimated in the filter is a hidden value, it is difficult to directly correct the state.

The filter corrects the state based on a difference between a measurement zk and the output value Hx̂k. of the estimation process <NUM> through the correction process <NUM>. In this example, a degree to which the state is corrected is determined using a Kalman gain Kk based on the error variance Pk. The error variance Pk is also updated to be Pk based on the Kalman gain Kk.

The aforementioned process may be applied to the outputting apparatus in the example of <FIG> as follows.

An IMU sensor senses an angular velocity w and an acceleration corresponding to motion information. In this example, a state of an ego-motion of the filter may separately include a state r for estimating the rotation information Rinitial and a state t for estimating translation information Tinitial.

When motion information (w<NUM>, a<NUM>) is sensed from the IMU sensor at a first time point, the filter updates the state r and the state t based on the motion information (w<NUM>, a<NUM>). The filter updates a state r<NUM> and a state t<NUM> to be a state r<NUM> and a state t<NUM> based on the motion information (w<NUM>, a<NUM>).

When motion information (w<NUM>, a<NUM>) is sensed from the IMU sensor at a second time point, the filter updates the state r<NUM> and the state t<NUM> to be a state r<NUM> and a state t<NUM> based on the motion information (w<NUM>, a<NUM>).

Likewise, when motion information (wn, an) is sensed from the IMU sensor at an nth time point, the filter updates a state rn-<NUM> and a state tn-<NUM> to be a state rn and a state tn based on the motion information (wn, an). At an initial time point (a zeroth time point) at which an operation of the IMU starts, rotation information R and translation information T are initialized (for example, set to be zero). Since an amount of change between rotation information estimated at each time point and an amount of change between translation information estimated at each time point are used, an initialization value may not affect a subsequent operation.

Also, it is assumed that a camera captures a first frame at the first time point and captures a second frame at the nth time point. In this example, the filter calculates a change amount O between an output value O<NUM> corresponding to the state r<NUM> of the first time point and an output value On corresponding to the state rn of the nth time point and outputs Rinitial.

The outputting apparatus rejects an outlier corresponding to a dynamic object in image information using Rinitial and calculates Restimate by applying a Perspective-n-Point (PNP) function to points (third points) obtained by rejecting the outlier. Restimate may be fed back as a measurement of the filter.

The outputting apparatus corrects the state r of the nth time point. The output estimated at the nth time point may be expressed as, for example, On = O<NUM> + Rinitial. Also, the measurement may be expressed as, for example, Zn = O<NUM> + Restimate.

To correct the state r of the nth time point, a difference between the measurement Zn and the output value On is used. Here, Zn - On = Restimate - Rinitial. Thus, the outputting apparatus corrects the state r using a value (Restimate - Rinitial). In this example, an output value On' is determined based on the corrected state r.

<FIG> is a diagram illustrating an example of a method of correcting estimated rotation information. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, in operation <NUM>, an outputting apparatus removes second points corresponding to a dynamic object from among first points based on estimated rotation information. The outputting apparatus estimates rotation information Rinitial obtained between a first time point and a second time point. The outputting apparatus removes the second points corresponding to the dynamic object from among the first points based on the rotation information Rinitial.

In operation <NUM>, the outputting apparatus estimates rotation information Restimate between a plurality of frames based on the third points. The third points being the remaining points when the second points are excluded from the first points. In an example, the outputting apparatus estimates the rotation information Restimate by applying a PNP function, a Gauss-Newton optimization function, and a Levenberg-Marquardt optimization function to the third points.

In operation <NUM>, the outputting apparatus corrects the rotation information Rinitial that is estimated based on motion information using the rotation information Restimate, which is estimated based on the third points. The outputting apparatus updates a state of an ego-motion corresponding to at least one of the plurality of frames based on a difference between the rotation information Restimate estimated based on the third points and the rotation information Rinitial estimated based on the motion information. The outputting apparatus estimates rotation information R* based on the updated state of the ego-motion.

<FIG> is a diagram illustrating an example of a method of correcting estimated translation information. <FIG> illustrates operations of the point selector <NUM>, the filter <NUM>, and the VIO rotation acquirer <NUM> for correcting translation information.

In an example, the VIO rotation acquirer <NUM> receives first points from the point selector <NUM> and receives rotation information Rinitial from the filter <NUM>. In an example, the VIO rotation acquirer <NUM> performs the outlier rejection <NUM> and the rotation evaluation <NUM> based on the first points and the rotation information Rinitial.

In outlier rejection <NUM>, the VIO rotation acquirer <NUM> rejects the outlier (the second points corresponding to the dynamic object) from the first points based on the rotation information Rinitial estimated based on the motion information received from the filter <NUM>. Through the outlier rejection <NUM>, third points are acquired by rejecting the outlier in the first points.

In an example, the VIO rotation acquirer <NUM> receives translation information Tinitial from the filter <NUM>. In an example, the VIO rotation acquirer <NUM> obtains a matching relationship between points included in each of the plurality of frames based on the rotation information Rinitial and the translation information Tinitial. The matching relationship is applied to the rotation evaluation <NUM>.

In rotation evaluation <NUM>, the VIO rotation acquirer <NUM> estimates the rotation information between the plurality of frames based on the third points transferred through the outlier rejection <NUM>.

The VIO rotation acquirer <NUM> transmits the rotation information Restimate for correcting the rotation information Rinitial based on the third points to the filter <NUM>. The VIO rotation acquirer <NUM> may evaluate the rotation information based on a difference between the rotation information Restimate estimated based on the third points and the rotation information Rinitial estimated based on the motion information, or an error between the two.

The VIO rotation acquirer <NUM> transfers the rotation information Restimate estimated based on the third points to the filter <NUM>. The filter <NUM> corrects the rotation information Rinitial based on the rotation information Restimate and outputs rotation information R*.

<FIG> is a diagram illustrating an example of a method of calculating translation information between a plurality of frames. <FIG> illustrates operations of the filter <NUM>, the VIO rotation acquirer <NUM>, and the VIO translation acquirer <NUM> for calculating translation information.

The VIO translation acquirer <NUM> calculates translation information between a plurality of frames, for example a change amount of translation information. The VIO translation acquirer <NUM> calculates translation information T* between the plurality of frames based on third points received from the VIO rotation acquirer <NUM> and rotation information R* that is corrected in the filter <NUM>.

In an example, the VIO translation acquirer <NUM> determines the translation information based on the rotation information R* such that an energy function associated with a difference in intensity between a plurality of frames to which the third points belongs is less than a target value. Here, the energy function may also be referred to as "photometric error function". In an example, the energy function is one of a Gauss-Newton optimization function and a Levenberg-Marquardt optimization function.

<FIG> is a diagram illustrating an example of selecting first points from a plurality of frames. The operations in <FIG> may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the scope of the illustrative examples described. Many of the operations shown in <FIG> may be performed in parallel or concurrently. One or more blocks of <FIG>, and combinations of the blocks, can be implemented by special purpose hardware-based computer that perform the specified functions, or combinations of special purpose hardware and computer instructions. In addition to the description of <FIG> below, the descriptions of <FIG> are also applicable to <FIG>, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring to <FIG>, in an example, the point selector <NUM> selects first points using a bucketing method <NUM>, a candidate selection method <NUM>, a careful selection method <NUM>, and a radial weighting method <NUM>. Although the <FIG> describes that the point selector <NUM> selects the first points by performing four selection methods in sequence, the order of the sequence may be changed or only some of the selections methods may be used without departing from the scope of the illustrative examples described. Also, the above-mentioned four selection methods may be combined or separated according to an implementation.

In an example, the point selector <NUM> equally divides an area of a plurality of frames into blocks having a size and evenly selects points from each of the blocks based on the bucketing method <NUM>. In an example, the point selector <NUM> selects points for each class such of a blob, an edge, and a corner based on the candidate selection method <NUM>. In an example, the point selector <NUM> may allow the points to be evenly selected for each of the blocks and classes based on the careful selection method <NUM>.

In an example, the point selector <NUM> selects points in consideration of an error occurring in restoring an image that is distorted due to a lens of a camera for capturing frames. Based on the radial weighting method <NUM>, the point selector <NUM> assigns a higher weight to points at a center of the image among points evenly selected for each of the blocks or classes and assigns a lower weight at an outer edge, which is away from the center. Thus, a large number of points at the center of the image are selected.

Operations of the bucketing method <NUM>, the candidate selection method <NUM>, the careful selection method <NUM>, and the radial weighting method <NUM> will be further described with reference to <FIG>.

<FIG> are diagrams illustrating examples of a method of selecting first points from a plurality of frames.

<FIG> illustrates a method of selecting first points based on a bucketing method. The bucketing method may be a method for improving a performance of a VIO by evenly selecting first points from each area of a plurality of frames <NUM>. In an example, an outputting apparatus equally divides each of the plurality of frames <NUM> into blocks (or buckets) of a size, such as frames <NUM>, for example. The outputting apparatus selects at least a number of first points from each of the equally divided blocks.

<FIG> illustrates a method of selecting first points using a candidate selection method. In an example, an outputting apparatus determines candidate points corresponding to a plurality of classes based on at least one of a magnitude and a direction of an intensity gradient of a pixel included in each of a plurality of frames and intensity relationships between pixels and pixels around the pixels. Here, the plurality of classes includes, for example, a corner, a blob, and an edge. In this example, the greater the magnitude of the intensity gradient of the pixel, the better the point. The outputting apparatus selects the first points from the candidate points included in each of the plurality of classes. The outputting apparatus selects first points from candidate points of various classes such as a plurality of frames <NUM> including a corner, a blob, and an edge based on, for example, a relationship between a direction of an intensity gradient of a corresponding pixel and an intensity of a pixel around the corresponding pixel.

<FIG> illustrates a method of selecting first points using a careful selection method. The careful selection method is a method of once more selecting the candidate points selected through the candidate selection in the example of 8B to optimize a performance of a VIO.

An outputting apparatus selects first points from candidate points for each class included in a plurality of frames and for each block into which a plurality of frames is divided when classifying the first points. In this example, the outputting apparatus assigns a priority to a block having a relatively fewer number of selected points such that the first points are selected evenly in the entire area of frames and the entire classes. Also, the outputting apparatus preferentially selects points corresponding to relatively less selected classes in a corresponding block.

<FIG> illustrates a method of selecting first points using a radial weighting method. In general, it is used in a VIO through an undistortion process for removing a barrel distortion caused by a lens. Here, since the undistortion process is performed based on an approximation algorithm, an error may increase in a direction toward an edge of an image. Thus, a relatively small number of points may be used in a direction toward an edge of an image so as to improve a performance of the VIO.

The radial weighting method is a method of adjusting a number of first points selected through the aforementioned process so that a smaller number of points are selected as a location of the first points is farther from a center of an image.

An outputting apparatus assigns a first weight to points at a center of each of a plurality of frames <NUM>. The outputting apparatus assigns a second weight that is less than the first weight to points gradually from the center toward an outer edge of each of the plurality of frames. For example, when the first weight is <NUM> or <NUM>, the second weight may have a value less than <NUM>. The outputting apparatus selects the first points from the points based on the first weight and the second weight. The outputting apparatus selects more points to which the higher first weight is assigned in comparison to points to which the second weight is assigned.

Referring to <FIG>, in operation <NUM>, an outputting apparatus retrieves images captured by a first sensor. For example, when a camera is a mono camera, the outputting apparatus acquires images corresponding to a single viewpoint. Also, when a camera is a stereo camera, the outputting apparatus acquires images corresponding to left and right viewpoints. In operation <NUM>, the outputting apparatus selects first points from the images retrieved in operation <NUM>.

In operation <NUM>, the outputting apparatus estimates a 6DoF pose change amount of a filter based on motion information sensed by a second sensor. In operation <NUM>, the outputting apparatus removes second points corresponding to a dynamic object from among the first points based on the change of the 6DoF pose amount estimated in operation <NUM>.

In operation <NUM>, the outputting apparatus evaluates rotation information of current frames based on third points. The third points are the points remaining after the second points are excluded from the first points , for example, an inlier, thereby calculating rotation information. In operation <NUM>, the outputting apparatus updates a state of the filter based on the rotation information calculated in operation <NUM>. Here, the state of the filter corresponds to a state of an ego-motion corresponding to the current frames.

In operation <NUM>, the outputting apparatus estimates rotation information of the filter using a value of the state updated in operation <NUM>. In operation <NUM>, the outputting apparatus evaluates (or estimates) translation information of a visual odometry based on the rotation information of the filter. In an example, the outputting apparatus estimates the translation information based on rotation information previously estimated for the inlier points.

<FIG> is a diagram illustrating an example of an apparatus for outputting pose information. Referring to <FIG>, an outputting apparatus <NUM> includes sensors <NUM> including a first sensor <NUM> and a second sensor <NUM>, and a processor <NUM>. The outputting apparatus <NUM> further includes a memory <NUM>, a communication interface <NUM>, and a display <NUM>. The sensors <NUM>, the processor <NUM>, the memory <NUM>, the communication interface <NUM>, and the display <NUM> communicate with one another through a communication bus <NUM>.

The first sensor <NUM> is, for example, an image sensor or a vision sensor. The first sensor <NUM> captures a plurality of frames corresponding to a driving image of a vehicle. The second sensor <NUM> senses motion information. The second sensor <NUM> includes sensors such as, for example, an acceleration sensor, a gyro sensor, a GPS sensor, an IMU sensor, a radar, and a lidar. The second sensor <NUM> senses sensing information of, for example, a speed, an acceleration, an angular velocity, a driving direction, a vehicle steering wheel angle, and a vehicle speed in addition to positioning information such as GPS coordinates, a position, and a pose.

The processor <NUM> performs the operations described with reference to <FIG>.

The memory <NUM> stores the plurality of frames captured by the first sensor <NUM> and/or the motion information sensed by the second sensor <NUM>. Also, the memory <NUM> stores rotation information estimated by the processor <NUM>, corrected rotation information, and translation information. The memory <NUM> stores various information generated during a processing operation of the processor <NUM>. Also, the memory <NUM> stores a variety of data and programs. The memory <NUM> includes a volatile memory or a non-volatile memory. The memory <NUM> includes a large-capacity storage medium such as a hard disk to store the variety of data. Further detail of the memory <NUM> is provided below.

The outputting apparatus <NUM> acquires sensing information of various sensors including the motion information and/or the plurality of frames through the communication interface <NUM>. Depending on an example, the communication interface <NUM> receives sensing information from other sensors located outside the outputting apparatus <NUM>.

The processor <NUM> outputs corrected rotation information and/or translation information using the communication interface <NUM> or displays a virtual object on the display <NUM> based on the corrected rotation information and/or translation information, thereby providing an augmented reality service. The processor <NUM> may render the virtual object on the display <NUM> based on the corrected rotation information and/or translation information and may also represent the virtual object together with the captured frames.

The term "processor," as used herein, is a hardware-implemented data processing device having a circuit that is physically structured to execute desired operations. For example, the desired operations include code or instructions included in a program. The hardware-implemented data processing device includes, but is not limited to, for example, a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA).

The processor <NUM> executes a program and controls the outputting apparatus <NUM>. Codes of the program executed by the processor <NUM> are stored in the memory <NUM>. Further details of the processor <NUM> is provided below.

The display <NUM> displays the corrected rotation information and/or translation information of the vehicle determined by the processor <NUM>. The display <NUM> displays a virtual object on the display <NUM> based on the corrected rotation information and/or translation information of the vehicle. In an example, the display <NUM> is a physical structure that includes one or more hardware components that provide the ability to render a user interface and/or receive user input. In an example, the corrected position of the vehicle is displayed on a wind shield glass or a separate screen of the vehicle using a head-up display (HUD) device or is displayed on an augmented reality head-up display (AR HUD). In an example, the outputting apparatus <NUM> transmits the localization information to an electronic control unit (ECU) or a vehicle control unit (VCU) of a vehicle. The ECU or the VCU displays the localization information on display device <NUM> of the vehicle.

However, the displaying of the corrected position of the vehicle is not limited to the example described above, and any other instrument cluster, vehicular infotainment system, screen in the vehicle, or display panel in the vehicle may perform the display function. Other displays, such as, for example, smart phone and eye glass display (EGD) that are operatively connected to the outputting apparatus <NUM> may be used without departing from the scope of the illustrative examples described.

The outputting apparatus <NUM> is an apparatus for performing the aforementioned method of outputting the pose information and may be, for example, a vehicle and a user device such as a navigation and a smartphone.

The outputting apparatus <NUM>, outputting apparatus <NUM>, filter <NUM>, point selector <NUM>, rotation information acquirer, VIO rotation acquirer <NUM>, translation information acquirer, VIO translation acquirer <NUM>, apparatuses, units, modules, devices, and other components described herein are implemented by hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

The methods that perform the operations described in this application are performed by computing hardware, for example, by one or more processors or computers, implemented as described above executing instructions or software to perform the operations described in this application that are performed by the methods.

Instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above are written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the processor or computer to operate as a machine or special-purpose computer to perform the operations performed by the hardware components and the methods as described above. In an example, the instructions or software includes at least one of an applet, a dynamic link library (DLL), middleware, firmware, a device driver, an application program storing the method of preventing the collision. In one example, the instructions or software include machine code that is directly executed by the processor or computer, such as machine code produced by a compiler. In another example, the instructions or software include higher-level code that is executed by the processor or computer using an interpreter. Programmers of ordinary skill in the art can readily write the instructions or software based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions in the specification, which disclose algorithms for performing the operations performed by the hardware components and the methods as described above.

Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card type memory such as multimedia card, secure digital (SD) card, or extreme digital (XD) card, magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and providing the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions.

Claim 1:
A computer-implemented visual inertial odometry method of obtaining pose information, the method comprising:
selecting (<NUM>) first points from frames captured by a first sensor, wherein the frames comprise a first frame captured at a first time point and a second frame captured at a second time point;
estimating (<NUM>) initial rotation information between the frames by:
estimating (<NUM>) a current state of an ego-motion, using a filter, based on a previous state of the ego-motion and motion information sensed by a second sensor; and
calculating the initial rotation information as an amount of rotation change between a first output of the filter at the previous state and a second output of the filter at the current state;
correcting (<NUM>) the estimated rotation information by:
prior to translation information between the frames being obtained, obtaining third points as a result of removing (<NUM>), from the first points, based on the estimated initial rotation information, second points corresponding to a dynamic object;
estimating additional rotation information between the frames based on the third points;
estimating a corrected current state of the ego-motion, using the filter, based on the current state of the ego-motion and the additional rotation information; and
calculating the corrected rotation information as an amount of rotation change between the first output of the filter at the previous state and a third output of the filter at the corrected current state;
obtaining (<NUM>) the translation information between the frames based on the third points and the corrected rotation information; and
outputting (<NUM>) the corrected rotation information and the translation information.