Patent ID: 12188771

DETAILED DESCRIPTION

FIG.1is a flow chart of an inertial measurement unit evaluating method100according to the 1st embodiment of the present disclosure.FIG.2Ais a block diagram of an inertial measurement unit evaluating system200according to the 2nd embodiment of the present disclosure.FIG.2B,FIG.2CandFIG.2Dare a top schematic view, a side schematic view and a front schematic view, respectively, of an autonomous driving vehicle800including the inertial measurement unit evaluating system200inFIG.2A. With reference toFIG.1toFIG.2D, the inertial measurement unit evaluating method100of the 1st embodiment is described with the assistance of the inertial measurement unit evaluating system200of the 2nd embodiment of the present disclosure. An inertial measurement unit evaluating method100is for evaluating an accuracy of an inertial measurement unit230included in the autonomous driving vehicle800. The autonomous driving vehicle800further includes a left camera210and a right camera220, and the left camera210and the right camera220both face a front direction of the autonomous driving vehicle800, e.g., the left camera210and the right camera220may be placed on the windshield of the front portion880of the autonomous driving vehicle800to face the front direction. The inertial measurement unit evaluating method100includes an image capturing step120, a vanishing line calculating step140, an image parameter determining step160, a unit parameter obtaining step170and a parameter comparing step180. Furthermore, the autonomous driving vehicle800may be an autonomous driving vehicle, or a vehicle of self-driving, semi-self-driving, or assisted driving, e.g., an autonomous car, an automated guided vehicle (AGV), etc.

FIG.5DandFIG.5Eare schematic views of the first vanishing point457and the second vanishing point458of the left image451and the right image452, respectively, of the vanishing line459for determining the image rollover tilt angle ϕ (clockwise direction or counterclockwise direction) in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.5Fis a schematic view of the vanishing line459for determining the image rollover tilt angle ϕ in the inertial measurement unit evaluating method100of the 1st embodiment. With reference toFIG.5DtoFIG.5F, the image capturing step120includes capturing the left image451and the right image452by the left camera210and the right camera220, respectively, at a time point, and each of the left image451and the right image452may be an image frame. The left image451includes the first left lane line453, the first right lane line455and the first vanishing point457, and the first vanishing point457is an intersection of the first left lane line453and the first right lane line455. The right image452includes the second left lane line454, the second right lane line456and the second vanishing point458, and the second vanishing point458is an intersection of the second left lane line454and the second right lane line456. The vanishing line calculating step140includes calculating to generate a vanishing line equation of the vanishing line459. The vanishing line459is a line connecting the first vanishing point457and the second vanishing point458, or a line connecting the first vanishing point457and the second vanishing point458t, which is the second vanishing point458after being transformed into the image coordinate system, as shown inFIG.5F.

With reference toFIG.1, the image parameter determining step160includes determining an image parameter set of the autonomous driving vehicle800according to the vanishing line equation. The unit parameter obtaining step170includes obtaining a unit parameter set (i.e., an inertial measurement unit parameter set) of the autonomous driving vehicle800from the inertial measurement unit230. The parameter comparing step180includes comparing the image parameter set and the unit parameter set to generate a comparison result. Therefore, based on the increasing image processing requirements for the vehicles, the number of cameras thereon has changed from single one to the multiple ones for the applications. The inertial measurement unit evaluating method100of the present disclosure applying the lane recognition technology will not cause too much load on the hardware and computing system of the autonomous driving vehicle800, so as to be easy to system integration and to effectively evaluate the accuracy of the inertial measurement unit230.

FIG.3A,FIG.3B,FIG.3C,FIG.3D,FIG.3EandFIG.3Fare schematic views of the images511,521,531,541,551,561, respectively, corresponding to the sub-steps of the image capturing step120inFIG.1. With reference toFIG.3AtoFIG.3F, in the image capturing step120, the lane line recognizing sub-step uses the Hough transform feature detection method. First, the image511of the original data (YUV) is captured by the left camera210or the right camera220, then the image521is generated from the image511after the grayscale conversion being performed and the image coordinate system being defined, and then the image531is generated from the image521after the edge processing being performed by Canny algorithm. Next, the region of interest (ROI) is set, such as the triangular frame548in the image541, which is formed by connecting the bottom line and the two oblique lines of the image541. The inside of the triangular frame548is the recognition region, the outside thereof is the non-recognition region, and the outside of the triangular frame548, e.g., the image551, is filled with a specific color block (e.g., a black block). Next, the Hough transform feature detection method is used to recognize the lane lines, such as the left lane line563and the right lane line565in the image561, and then a plurality of coordinate values of the left lane line563and the right lane line565are obtained. Any two of the coordinate values of one line can be used to obtain the slope and the linear line equation of the line, and the vanishing point567being the intersection of the left lane line563and the right lane line565can be obtained. For example, the linear line equation of the left lane line563is represented as Y=a1×X+b1, the linear line equation of the right lane line565is represented as Y=a2×X+b2, a1, a2, b1, and b2 are coefficients, and the vanishing point567is represented as (K1, K2).

Furthermore, the vanishing point is an intersection point intersected by all parallel lines in the camera coordinate system of the three-dimensional space. The two lanes are parallel in the camera coordinate system of the three-dimensional space, however, the two lanes are eventually intersected as the vanishing point in the image coordinate system of the two-dimensional space, e.g., the first vanishing point457inFIG.5Dmay have the coordinate value represented as (K1, K2). Moreover, the union of all vanishing points constitutes a vanishing line.

The vanishing line calculating step140may further include transforming one image coordinate system of an image coordinate system of the left image451and an image coordinate system of the right image452into the other image coordinate system thereof. A coordinate of a horizontal direction of the other image coordinate system is represented as X, a coordinate of a vertical direction of the other image coordinate system is represented as Y, the vanishing line equation is represented as Y=a×X+b, and a and b are coefficients of the vanishing line equation. Therefore, the coordinate system transformation and subsequent calculating steps are advantageous in reducing the amount of computation and accurately determining the three-axis angle (i.e., the three-axis tilt angle or the three-axis rotation angle) of the autonomous driving vehicle800with the dual images.

FIG.4AandFIG.4Bare reference views of the transformations of the image coordinate systems in the vanishing line calculating step140inFIG.1. With reference toFIG.4A, furthermore, the transformation of the camera coordinate system (Xc, Yc, Zc) into the image coordinate system (X, Y) is transformed from a three-dimensional space to a two-dimensional space, which belongs to the perspective projection relationship. As shown inFIG.4A, the camera coordinate system is the coordinate system of the directions (axes) Xc, Yc, Zc and the origin Oc, and the image coordinate system is the coordinate system of the horizontal direction X, the vertical direction Y and the origin O. InFIG.4A, it also shows the point P to be measured, the focal length f and the vertices A, B, C, and the coordinate value of the point P to be measured in the camera coordinate system is P(Xcp, Ycp, Zcp) and transformed to the coordinate value P′(Xp, Yp) of the image point P′ to be measured in the image coordinate system. InFIG.4A, there are the trigonometric relationships ΔABOc˜ΔOCOc and ΔPBOc˜ΔP′COc, so there are further the line segment geometric relationships in formula (1), formula (2), formula (3) and the transformation relationship of formula (4) from the camera coordinate system to image coordinate system as the following:

ABOC=AOcOOc=PBP′⁢C=XcpXp=Zcpf=YcpYp;(1)Xp=f×XcpZcp;(2)Yp=f×YcpZcp;and(3)Zcp[XpYp1]=[f0000f000010][XcpYcpZcp].(4)

With reference toFIG.4B, which shows the schematic view of dual cameras (e.g., the left camera210and the right camera220) represented in the image coordinate system. InFIG.4B, it shows the optical centers O1 and O2 of the dual cameras, the image point P′ to be measured (the coordinate value in the image coordinate system is P′(Xp, Yp)), the imaging point P1 (with the coordinate value (Xp1, Yp1)) and the imaging point P2 (with the coordinate value (Xp2, Yp2)) on the imaging surface670of the image sensors of the dual cameras, the focal length f, the baseline distance b of the dual cameras and the depth distance s0. InFIG.4B, there are the line segment geometric relationship in formula (5), the coordinate value relationship of the imaging points P1, P2 in the horizontal direction X in formula (6), the coordinate value relationship of the imaging points P1 and P2 in the vertical direction Y in formula (7), and the depth distance s0 in formula (8) as the following:

s⁢0f=XpXp⁢1=(Xp-b)Xp⁢2=YpYp⁢1=YpYp⁢2;(5)Xp⁢2=((Xp-b)Xp)×Xp⁢1=M×Xp⁢1;(6)Yp⁢2=Yp⁢1;and(7)s⁢0f=f×b(Xp⁢1-Xp⁢2).(8)

FIG.5AandFIG.5Bare schematic views of the first vanishing point357and the second vanishing point358of the left image351and the right image352, respectively, of the reference line359for determining the image rollover tilt angle ϕ in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.5Cis a schematic view of the reference line359for determining the image rollover tilt angle ϕ in the inertial measurement unit evaluating method100of the 1st embodiment. With reference toFIG.5AtoFIG.5C, the inertial measurement unit evaluating method100may further include a reference line defining step110, when the autonomous driving vehicle800is in a non-tilted state (may be determined by the inertial measurement unit230) at another time point, the another time point is earlier than the time point, and the left image351and the right image352are captured by the left camera210and the right camera220, respectively, at the another time point. The left image351includes the first left lane line353, the first right lane line355and the first vanishing point357, and the right image352includes the second left lane line354, the second right lane line356and the second vanishing point358. Next, the second vanishing point358in the image coordinate system of the right image352is transformed into the second vanishing point358tin the image coordinate system of the left image351(as shown inFIG.5C). The first vanishing point357and the second vanishing point358tin the left image351are connected to form a vanishing line having a vanishing line equation, which is defined as the reference line359having the reference line equation. In the 1 st embodiment, the reference line equation is Y=K5, and K5 is a constant of the reference line equation. Therefore, the reference line in the non-tilted state and the subsequent calculating steps are advantageous in reducing the amount of computation and accurately determining the three-axis angle of the autonomous driving vehicle800with the dual images.

Furthermore, in the camera coordinate system, when the camera is in a horizontal state and not tilted, the slope is equal to 0. Because the slope is equal to 0, only the single coordinate value of the vanishing point can be used to obtain the vanishing line equation, which is represented as Y=K0 (or shown in another constant symbol) and can be used as the original linear line equation of the three-axis of the vehicle body in the camera coordinate system. On the contrary, when the camera is tilted (the slope is not equal to 0), the intersection of the two lane lines in the image coordinate system of the two-dimensional space is the vanishing point, the coordinate value of the vanishing point can be obtained, but the vanishing line equation cannot be obtained, and a coordinate value of another point on the vanishing line is needed for obtaining the vanishing line equation.

FIG.5Gis a schematic view of the image rollover tilt angle ϕ in the inertial measurement unit evaluating method100of the 1st embodiment. With reference toFIG.1andFIG.5CtoFIG.5G, in the image capturing step120after the reference line defining step110, the left image451and the right image452are captured by the left camera210and the right camera220, respectively, at the time point.

In the vanishing line calculating step140, the second vanishing point458in the image coordinate system of the right image452is transformed into the second vanishing point458tin the image coordinate system of the left image451(as shown inFIG.5F). The first vanishing point457and the second vanishing point458tin the left image451are connected to form the vanishing line459having a vanishing line equation Y=a5×X+b5, and a5 and b5 are constants of the vanishing line equation.

In the image parameter determining step160, the image parameter set may include the image rollover tilt angle ϕ, as shown inFIG.5G. The image rollover tilt angle ϕ is an angle between the vanishing line459inFIG.5Fand the reference line359inFIG.5C. The angle that can be obtained from the vanishing line equation Y=a5×X+b5 and the reference line equation Y=K5 is the image rollover tilt angle ϕ, which is the angle at which the autonomous driving vehicle800tilts clockwise or counterclockwise relative to the normal direction Z. In the unit parameter obtaining step170, the unit parameter set includes a unit rollover tilt angle, which corresponds to the image rollover tilt angle ϕ. The parameter comparing step180further include comparing the image rollover tilt angle ϕ and the unit rollover tilt angle. Therefore, the image rollover tilt angle ϕ can be accurately calculated for comparison with the unit rollover tilt angle.

FIG.6AandFIG.6Bare schematic views of the first vanishing point367and the second vanishing point368of the left image361and the right image362, respectively, of the reference line369for determining the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.6Cis a schematic view of the reference line369for determining the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.6DandFIG.6Eare schematic views of the first vanishing point467and the second vanishing point468of the left image461and the right image462, respectively, of the vanishing line469for determining the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.6Fis a schematic view of the vanishing line469for determining the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.6Gis a schematic view of the parameters for determining the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.6His a schematic view of the image up-down tilt angle ω in the inertial measurement unit evaluating method100of the 1st embodiment. With reference toFIG.1andFIG.6AtoFIG.6H, in the reference line defining step110, when the autonomous driving vehicle800is in the non-tilted state, the left image361and the right image362are captured by the left camera210and the right camera220, respectively. The left image361includes the first left lane line363, the first right lane line365and the first vanishing point367, and the right image362includes the second left lane line364, the second right lane line366and the second vanishing point368. Next, the second vanishing point368in the image coordinate system of the right image362is transformed into the second vanishing point368tin the image coordinate system of the left image361(as shown inFIG.6C). The first vanishing point367and the second vanishing point368tin the left image361are connected to form the reference line369having a reference line equation Y=K62, and K62 is a constant of the reference line equation.

In the image capturing step120after the reference line defining step110, the left image461and the right image462are captured by the left camera210and the right camera220, respectively. The left image461includes the first left lane line463, the first right lane line465and the first vanishing point467, and the right image462includes the second left lane line464, the second right lane line466and the second vanishing point368. In the vanishing line calculating step140, the second vanishing point468in the image coordinate system of the right image462is transformed into the second vanishing point468tin the image coordinate system of the left image461(as shown inFIG.6F). The first vanishing point467and the second vanishing point468tin the left image461are connected to form the vanishing line469having a vanishing line equation Y=K63, and K63 is a constant of the vanishing line equation.

The vanishing line calculating step140may further include calculating to generate a vanishing distance s6 (as shown inFIG.6H) parallel to the normal direction Z between the vanishing line469and one of the left camera210and the right camera220, the vanishing distance s6 is a distance from the one of the left camera210and the right camera220to the first vanishing point467, and the horizontal direction X, the vertical direction Y and the normal direction Z are perpendicular to each other.

In the image parameter determining step160, the image parameter set may include the image up-down tilt angle ω (as shown inFIG.6H), which is obtained according to a difference d6 in the vertical direction Y between the vanishing line469and the reference line369, and the vanishing distance s6, that is, d6=K63-K62, and via tan(ω)=d6/s6, the image up-down tilt angle ω can be obtained. The image up-down tilt angle ω is the upward or downward angle of the body of the autonomous driving vehicle800. In the unit parameter obtaining step170, the unit parameter set includes a unit up-down tilt angle, which corresponds to the image up-down tilt angle ω. The parameter comparing step180further includes comparing the image up-down tilt angle ω and the unit up-down tilt angle. Therefore, the image up-down tilt angle ω can be accurately calculated for comparison with the unit up-down tilt angle.

FIG.7AandFIG.7Bare schematic views of the first vanishing point377and the second vanishing point378of the left image371and the right image372, respectively, of the reference line379for determining the image left-right tilt angle T in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.7Cis a schematic view of the reference line379for determining the image left-right tilt angle τ in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.7DandFIG.7Eare schematic views of the first vanishing point477and the second vanishing point478of the left image471and the right image472, respectively, for determining the image left-right tilt angle τ in the inertial measurement unit evaluating method100of the 1st embodiment.FIG.7Fis a schematic view of the image left-right tilt angle τ in the inertial measurement unit evaluating method100of the 1st embodiment. With reference toFIG.1andFIG.7AtoFIG.7F, in the reference line defining step110, when the autonomous driving vehicle800is in the non-tilted state, the left image371and the right image372are captured by the left camera210and the right camera220, respectively. The left image371includes the first left lane line373, the first right lane line375and the first vanishing point377, and the right image372includes the second left lane line374, the second right lane line376and the second vanishing point378. Next, the second vanishing point378in the image coordinate system of the right image372is transformed into the second vanishing point378tin the image coordinate system of the left image371(as shown inFIG.7C). The coordinate value of the first vanishing point377in the left image371is (Xp1, Yp1), and the coordinate value of the transformed second vanishing point378tis (Xp2, Yp2). The first vanishing point377and the second vanishing point378tin the left image371are connected to form the reference line379having a reference line equation Y=K7, and K7 is a constant of the reference line equation. In practice, it should be noted that the reference line in one ofFIG.5C,FIG.6CandFIG.7Ccan be defined as a reference line in another one ofFIG.5C,FIG.6CandFIG.7C, that is, the reference lines with the reference line equations of the one and the another one ofFIG.5C,FIG.6CandFIG.7Care the same.

In the image capturing step120after the reference line defining step110, the left image471and the right image472are captured by the left camera210and the right camera220, respectively. The left image471includes the first left lane line473, the first right lane line475and the first vanishing point477, and the right image472includes the second left lane line474, the second right lane line476and the second vanishing point378. In the vanishing line calculating step140, the second vanishing point478in the image coordinate system of the right image472is transformed into the second vanishing point (not shown in drawings) in the image coordinate system of the left image471. The coordinate value of the first vanishing point477in the left image471is (Xq1, Yq1), and the coordinate value of the transformed second vanishing point in the left image471is (Xq2, Yq2). The vanishing line calculating step140further includes calculating to generate a vanishing distance s7 parallel to the normal direction Z between the vanishing line and one of the left camera210and the right camera220, as shown inFIG.7F.

In the image parameter determining step160, the image parameter set may include the image left-right tilt angle τ, as shown inFIG.7F. The image left-right tilt angle τ is obtained according to an absolute value d7 of a difference in the horizontal direction X between the transformed second vanishing point in the left image471and the transformed second vanishing point378t, which is for defining the reference line379, in the left image371, that is, d7=|Xp2−Xq2|, and via tan(τ)=d7/s7, the image left-right tilt angle τ can be obtained. The image left-right tilt angle τ is the left or right tilt angle of the body of the autonomous driving vehicle800. In the unit parameter obtaining step170, the unit parameter set includes a unit left-right tilt angle, which corresponds to the image left-right tilt angle τ. The parameter comparing step180further includes comparing the image left-right tilt angle τ and the unit left-right tilt angle. Therefore, the image left-right tilt angle τ can be accurately calculated for comparison with the unit left-right tilt angle. In another embodiment of the present disclosure (not shown in drawings), an image left-right tilt angle may be obtained according to an absolute value of a difference in the horizontal direction between the first vanishing point, which is for defining the vanishing line, and another first vanishing point, which is for defining the reference line.

In practice, at least two of the three-axis angles (i.e., the image rollover angle ϕ, the image up-down tilt angle ω and the image left-right tilt angle τ) at a time point of the autonomous driving vehicle800calculated by the inertial measurement unit evaluating method100may be not zero. For example, the image rollover tilt angle ϕ towards the clockwise direction and the image up-down tilt angle ω towards the up direction may both exist at a time point, or the image up-down tilt angle ω towards the up direction and the image left-right tilt angle τ towards the left direction may both exist at a time point, or the image rollover tilt angle ϕ towards the counterclockwise direction, the image up-down tilt angle ω towards the down direction and the image left-right tilt angle τ towards the right direction may all exist at a time point, and it is not limited thereto. Furthermore, when the vanishing line equation Y=a5×X+b5 related to the image rollover tilt angle ϕ has been determined, the image rollover tilt angle D can be determined, and the image up-down tilt angle ω and the image left-right tilt angle τ can further be obtained based on the vanishing line equation Y=a5×X+b5.

With reference toFIG.1, the inertial measurement unit evaluating method100may further include a calibration requesting step190, which includes requesting to perform a calibration of the inertial measurement unit230when the comparison result satisfies a threshold condition, e.g., the signals for requesting the calibration of the inertial measurement unit230is generated and transmitted. Therefore, the inertial measurement unit evaluating method100is advantageous in evaluating whether to further perform a computationally intensive calibration (e.g., a Kalman filter algorithm) for the inertial measurement unit230. When the comparison result does not satisfy the threshold condition, that is, when the unit parameter set measured by the inertial measurement unit230is evaluated within an acceptable range of the accuracy by the inertial measurement unit evaluating method100, the computationally intensive calibration can be not needed, and the calibration accuracy of the inertial measurement unit230can be simultaneously ensured.

With reference toFIG.2AtoFIG.2D, the inertial measurement unit evaluating system200is disposed on the autonomous driving vehicle800and includes the inertial measurement unit230, the left camera210, the right camera220, a storage medium280and a processor290.

The left camera210faces a front direction of the autonomous driving vehicle800, and the right camera220faces the front direction. The storage medium280is configured to provide an inertial measurement unit evaluating program282. The processor290is communicatively coupled to the inertial measurement unit230, the left camera210, the right camera220and the storage medium280. Based on the inertial measurement unit evaluating program282, the processor290is configured to capture the left image451and the right image452by the left camera210and the right camera220, respectively, at a time point, the left image451includes the first left lane line453, the first right lane line455and the first vanishing point457, the first vanishing point457is the intersection of the first left lane line453and the first right lane line455, the right image452includes the second left lane line454, the second right lane line456and the second vanishing point458, and the second vanishing point458is the intersection of the second left lane line454and the second right lane line456. Based on the inertial measurement unit evaluating program282, the processor290is further configured to: calculate to generate the vanishing line459, which is a line connecting the first vanishing point457and the second vanishing point458or the second vanishing point458ttransformed into the image coordinate system; determine the image parameter set of the autonomous driving vehicle800according to the vanishing line459; obtain the unit parameter set of the autonomous driving vehicle800from the inertial measurement unit230; and compare the image parameter set and the unit parameter set to generate the comparison result. Therefore, the inertial measurement unit evaluating system200is advantageous in evaluating whether to further perform the computationally intensive calibration for the inertial measurement unit230.

With reference toFIG.2BandFIG.2D, the left camera210and the right camera220may be disposed symmetrically to a (virtual) longitudinal center plane of the autonomous driving vehicle800. Therefore, it is beneficial to reduce the calculation amount required by the inertial measurement unit evaluating program282. Furthermore, the left camera and the right camera according to the present disclosure can be dual cameras on the same device, or each can be an independent single camera device.

Regarding other details of the inertial measurement unit evaluating system200of the 2nd embodiment, the contents of the inertial measurement unit evaluating method100of the 1st embodiment can be referred, and the details are not described again herein.

Although the present disclosure has been described in considerable detail with reference to certain embodiments thereof, other embodiments are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the embodiments contained herein. It will be apparent to those skilled in the art that various modifications and variations can be made to the structure of the present disclosure without departing from the scope or spirit of the disclosure. In view of the foregoing, it is intended that the present disclosure cover modifications and variations of this disclosure provided they fall within the scope of the following claims.