Apparatus and method for estimating road geomeiry

A processing device includes a first processor configured to detect a bounding box of a distant vehicle, in an input image generated by imaging the distant vehicle, and extract at least one feature of the distant vehicle. A second processor is configured to estimate a geometry of a road on which the distant vehicle is located, based on a position of at least one feature relative to at least a portion of the bounding box.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No. 10-2020-0013304, filed on Feb. 4, 2020, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The disclosure relates to a technique of collecting information about the driving of a vehicle, and more particularly, to an apparatus and method for estimating a road geometry.

Various pieces of information may be collected for autonomous driving and/or driving assistance. For example, a vehicle may include various sensors configured to sense a state of the vehicle and a surrounding state of the vehicle, and useful information may be generated from outputs of the sensors. The collected information may be utilized in various ways. For instance, the collected information may be used to control the vehicle or provided to a driver of the vehicle. Not only user convenience but also safety may be critical during the driving of the vehicle. Thus, information collected to assist the driving of the vehicle should have high accuracy.

SUMMARY

The disclosure provides an apparatus and method for precisely and easily estimating a geometry of a road on which a vehicle travels.

According to an aspect of the disclosure, a method includes obtaining an input image, the input image being generated by imaging a distant vehicle, detecting a bounding box of the distant vehicle from the input image, extracting at least one feature of the distant vehicle from the input image, and estimating a geometry of a road on which the distant vehicle is located, based on a position of the at least one feature relative to at least a portion of the bounding box.

According to another aspect of the disclosure, there is provided a processing device including a first processor configured to detect a bounding box of a distant vehicle, in an input image generated by imaging the distant vehicle, and extract at least one feature of the distant vehicle. A second processor is configured to estimate a geometry of a road on which the distant vehicle is located, based on a position of at least one feature relative to at least a portion of the bounding box.

According to another aspect of the disclosure, there is provided a vehicle including a camera module configured to image a distant vehicle and generate an input image, a processing device configured to detect a bounding box and at least one feature of the distant vehicle in the input image and estimate a geometry of a road on which the distant vehicle is located, based on a position of the at least one feature relative to at least a portion of the bounding box, and a controller configured to generate a control signal for controlling the vehicle, based on the geometry of the road.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1is a block diagram of a vehicle100according to an example embodiment. The vehicle100may refer to an arbitrary movable object that travels on a road. For example, the vehicle100may refer to an object (e.g., a bicycle, a car, a motorcycle, a train, and the like) designed to transport a person or an object or refer to an object designed to be movable for purposes different from transportation. As used herein, a car will mainly be described as an example of the vehicle100, but it will be understood that example embodiments are not limited thereto. As shown inFIG. 1, the vehicle100may include a processing device110, a camera module120, and at least one sensor130. In some embodiments, as described below with reference toFIG. 15, the vehicle100may further include various machine parts for driving operations.

The camera module120may image (or shoot) another vehicle, which is spaced apart from the vehicle100, and generate image data IMG. For example, the camera module120may be installed to shoot the front of the vehicle100and generate image data IMG corresponding to an image including a front vehicle. Also, the camera module120may be installed to shoot the rear of the vehicle100and generate image data IMG corresponding to an image including a rear vehicle. In some embodiments, the camera module120may include an image sensor capable of sensing visible light, and the image data IMG may indicate a visible light image. In some embodiments, the image data IMG may indicate an infrared (IR) image, a grayscale image, and/or a depth image. In some embodiments, the vehicle100may include two or more camera modules, and pieces of image data corresponding to various images may be generated. As used herein, a vehicle, which is spaced apart from the vehicle100and shot by the camera module120, may be referred to as a distant vehicle or another vehicle. A distant vehicle located at the front of the vehicle100may be referred to as a front vehicle and a distant vehicle located at the rear of the vehicle100may be referred to as a rear vehicle. Also, the vehicle100including the camera module120configured to shoot the distant vehicle may be referred to as a host vehicle.

The at least one sensor130may sense a state of the vehicle100or a surrounding state of the vehicle100and generate a sense signal SEN. In some embodiments, the at least one sensor130, which is a distance sensor configured to measure a distance from the distant vehicle, may include a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a Time of Flight (ToF) sensor, an ultrasonic sensor, and/or an IR sensor. In some embodiments, to sense a state of the vehicle100, the at least one sensor130may include a geomagnetic sensor, a global positioning system (GPS) sensor, an acceleration sensor, and/or a gyro sensor. Furthermore, in some embodiments, the at least one sensor130may further include a pressure sensor and a temperature/humidity sensor. As described below, in some embodiments, the sense signal SEN may be used for the processing device110to estimate a geometry of the road.

The processing device110may communicate with the camera module120and the at least one sensor130. For example, as shown inFIG. 1, the processing device110may receive image data IMG from the camera module120or receive a sense signal SEN from the at least one sensor130. As used herein, the image data IMG received by the processing device110from the camera module120may be referred to as an input image. The processing device110may be an arbitrary electronic device, which may process data and/or a signal. For example, the processing device110may be an integrated circuit (IC), which is manufactured using a semiconductor process, or a module including at least two semiconductor packages and a board on which the semiconductor packages are mounted. As shown inFIG. 1, the processing device110may include a first processor111, a second processor112, a memory113, and an input/output (I/O) interface114, which may communicate with each other through a bus115. In some embodiments, at least two of the first processor111, the second processor112, the memory113, and the I/O interface114may directly communicate with each other without using the bus115. In some embodiments, the bus115may be omitted.

The first processor111may process the image data IMG and be also referred to as an image processor. For example, the first processor111may detect a bounding box of the distant vehicle from the image data IMG. Also, the first processor111may extract at least one feature of the distant vehicle from the image data IMG. In some embodiments, the first processor111may detect the bounding box and/or extract at least one feature, based on a model learned by a plurality of vehicle images. To this end, in some embodiments, the first processor111may include a neural network processing unit (NPU). As described below, a posture of the distant vehicle may be determined based on the bounding box detected by the first processor111and the at least one feature extracted by the first processor111.

The second processor112may estimate a geometry of a road on which the distant vehicle is located. For example, the second processor112may obtain the bounding box and the at least one feature of the distant vehicle from the first processor111, determine the posture of the distant vehicle, based on a position of at least one feature relative to at least a portion of the bounding box, and estimate the geometry of the road on which the distant vehicle is located, based on the posture of the distant vehicle. Also, the second processor112may compensate for the estimated geometry of the road, based on the state of the vehicle100. As described below with reference toFIGS. 15 and 16, the geometry of the road, which is estimated by the second processor112, may be utilized for various functions that are useful for driving the vehicle100. For instance, the second processor112may assist lane detection based on the estimated geometry of the road.

In some embodiments, each of the first processor111and the second processor112may include a hardware logic designed by logic synthesis. Also, in some embodiments, each of the first processor111and the second processor112may include at least one core, which executes instructions stored in an internal memory of the first processor111and/or the memory113. For example, each of the first processor111and the second processor112may refer to an arbitrary hardware-implemented data processing device including a circuit that is physically configured to execute predetermined operations including operations expressed by instructions and/or code, which are included in a program. For example, the data processing device may include a microprocessor (MP), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a processor core, a multi-core processor, a multi-processor, an application-specific integrated circuit (ASIC), an application-specific instruction-set processor (ASIP), and a field programmable gate array (FPGA). In some embodiments, unlike that shown inFIG. 1, the processing device110may include a single processor configured to perform all operations of the first processor111and the second processor112. As used herein, operations performed by the first processor111and/or the second processor112may also be referred to as operations performed by the processing device110.

The memory113may store data, which is processed or to be processed by the first processor111and the second processor112or data, which is received through the I/O interface114or is to be transmitted to the outside. For example, the memory113may store image data IMG, which is provided by the camera module120, store data about the bounding box and the at least one feature of the distant vehicle, which are generated by the first processor111, or store data about the geometry of the road, which is generated by the second processor112. In some embodiments, the memory113may store a series of instructions, which are executed by the first processor111and the second processor112. In some embodiments, the memory113may include a volatile memory device, such as dynamic random access memory (DRAM) and static RAM (SRAM). In some embodiments, the memory113may include a non-volatile memory device, such as electrically erasable programmable read-only memory (EEPROM), silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory, magnetic RAM (MRAM), phase-change RAM (PRAM), and/or resistive RAM (RRAM).

The I/O interface114may provide an interface with an external component of the processing device110. For example, as shown inFIG. 1, the I/O interface114may provide an interface with the camera module120or provide an interface with the at least one sensor130. In some embodiments, the I/O interface114may provide an interface for outputting the geometry of the road, which is estimated by the second processor112.

FIG. 2is a flowchart of a method of estimating a road geometry, according to an example embodiment. As shown inFIG. 2, the method of estimating the road geometry may include a plurality of operations (e.g., S20, S40, S60, and S80). In some embodiments, the method ofFIG. 2may be performed by the processing device110ofFIG. 1. Hereinafter,FIG. 2will be described with reference toFIG. 1.

In operation S20, an operation of obtaining image data IMG may be performed. For example, the processing device110may receive image data IMG, which is generated by shooting a distant vehicle by using the camera module120, from the camera module120. In some embodiments, operation S20may be performed after operation S40or performed in parallel with operation S40.

In operation S40, an operation of detecting a bounding box of the distant vehicle may be performed. For example, the processing device110may detect the bounding box of the distant vehicle in the image data IMG. In some embodiments, the bounding box of the distant vehicle may include a pair of lateral lines and a pair of vertical lines and define a minimum region including the distant vehicle. Examples of the bounding box will be described below with reference toFIG. 3. The bounding box of the distant vehicle may be detected from the image data IMG, based on an arbitrary method of detecting a specific object from an image, and an example of operation S40will be described below with reference toFIG. 4.

In operation S60, an operation of extracting a feature of the distant vehicle may be performed. For example, the processing device110may extract at least one feature of the distant vehicle from the bounding box of the distant vehicle. In some embodiments, the processing device110may extract a headlamp, a tail lamp, a license plate, a side mirror, and a wheel of the distant vehicle as the feature of the distant vehicle. The feature of the distant vehicle may be extracted from the image data IMG, based on an arbitrary method of extracting a feature from an image, and examples of the extracted feature will be described below with reference toFIGS. 5A to 5C.

In operation S80, an operation of estimating a geometry of a road may be performed. For example, the processing device110may estimate a geometry of a road on which the distant vehicle is located, based on a position of at least one feature relative to at least a portion of the bounding box of the distant vehicle. The position of the at least one feature may indicate a posture of the distant vehicle, and the processing device110may estimate the geometry of the road, based on the posture of the distant vehicle. Thus, the geometry of the road may be estimated based on one piece of image data IMG, and a plurality of camera modules may be prevented from being included in the vehicle100to estimate the geometry of the road. Also, when the distant vehicle is preceding the vehicle100on a route in which the vehicle100travels, a geometry of a road on which the vehicle100will travel may be estimated in advance. Furthermore, the geometry of the road may be accurately estimated based on not only the position of the distant vehicle but also the posture of the distant vehicle. In some embodiments, the geometry of the road may include a profile (or height profile) of the road, which indicates road undulations, and/or a cross slope (or road angle) of the road. Examples of operation S80will be described below with reference toFIGS. 6, 8, 10, 12, and 13. In some embodiments, operation S80may include all operations shown inFIGS. 6, 8, 10, 12, and 13.

FIG. 3illustrates examples of image data and a bounding box according to an example embodiment. As described above with reference toFIG. 1, image data IMG′ ofFIG. 3may be generated by shooting a distant vehicle by using the camera module120, and the processing device110may detect a bounding box of the distant vehicle in the image data IMG′. Hereinafter,FIG. 3will be described with reference toFIG. 1. Also, it is assumed that the camera module120is arranged to shoot the front of the vehicle100and the distant vehicle is a front vehicle.

In some embodiments, the image data IMG′ may be generated by shooting a plurality of front vehicles, and the processing device110may detect a plurality of bounding boxes corresponding respectively to the plurality of front vehicles. For example, as shown inFIG. 3, the processing device110may detect bounding boxes BOX1, BOX5, and BOX6of front vehicles located on the same lane as the vehicle100, detect bounding boxes BOX2, BOX3, BOX4, BOX7, and BOX8of front vehicles located on different lanes from the vehicle, or detect a bounding box BOX9of a front vehicle that travels in an opposite direction to the vehicle100. As described above with reference toFIG. 2, each of the bounding boxes BOX1to BOX9may include a pair of lateral lines and a pair of vertical lines. Thus, each of the bounding boxes BOX1to BOX9may be defined by a pair of points facing each other in a diagonal direction. As shown inFIG. 3, each of the bounding boxes BOX1to BOX9may have different sizes due to a distance from the vehicle100to the front vehicle and a size of the front vehicle.

FIG. 4is a flowchart of a method of estimating a road geometry, according to an example embodiment. Specifically, the flowchart ofFIG. 4illustrates an example of operation S40ofFIG. 2. As described above with reference toFIG. 2, an operation of detecting a bounding box of a distant vehicle may be performed in operation S40aofFIG. 4. As shown inFIG. 4, operation S40amay include operations S42and S44.FIG. 4will be described with reference toFIG. 1.

Referring toFIG. 4, in operation S42, an operation of providing image data IMG to a machine learning model ML learned from vehicle images may be performed. That is, a bounding box of a distant vehicle may be detected from the image data IMG, based on machine learning, and the machine learning model ML learned from the vehicle images may have an arbitrary structure for machine learning. For example, the machine learning model ML may include an artificial neural network, a convolution neural network, a deep neural network, a decision tree, a support vector machine, a Bayesian network, and/or a genetic algorithm. In some embodiments, the first processor111may include a component (e.g., an NPU) configured to implement at least a portion of the machine learning model ML.

In operation S44, an operation of obtaining a bounding box BOX from the machine learning model ML may be performed. For example, the machine learning model ML may generate coordinates of a pair of points defining a bounding box BOX of a distant vehicle in the image data IMG. In some embodiments, as described above with reference toFIG. 3, a plurality of bounding boxes corresponding to a plurality of distant vehicles in the image data IMG may be obtained from the machine learning model ML.

FIGS. 5A to 5Care diagrams of examples of a bounding box and a feature according to an example embodiment. Specifically,FIG. 5Aillustrates a bounding box BOX5aand features F51ato F55a, which correspond to a rear surface of a distant vehicle,FIG. 5Billustrates a bounding box BOX5band features F51bto F55b, which correspond to a front surface of the distant vehicle, andFIG. 5Cillustrates a bounding box BOX5cand features F51cto F56c, which correspond to the rear surface and a side surface of the distant vehicle. Hereinafter,FIGS. 5A to 5Cwill be described with reference toFIG. 1. A repeated description will be omitted in the description ofFIGS. 5A to 5C.

Referring toFIG. 5A, the bounding box BOX5amay include the rear surface of the distant vehicle. For example, the camera module120may be arranged to shoot the front of the vehicle100. The camera module120may shoot a rear surface of the front vehicle preceding the vehicle100and generate image data IMG. Thus, the processing device110may detect the bounding box BOX5aincluding the rear surface of the front vehicle and extract the features F51ato F55ain the bounding box BOX5a. For example, as shown inFIG. 5A, the processing device110may extract the features F51aand F52acorresponding to a side mirror, the features F53aand F54acorresponding to a tail lamp, and the feature F55acorresponding to a license plate from the image data IMG. In some embodiments, the processing device110may extract only some of the features F51ato F55aofFIG. 5Aor further extract additional features (e.g., a wheel or the like) in addition to the features F51ato F55aofFIG. 5A.

Referring toFIG. 5B, the bounding box BOX5bmay include the front surface of the distant vehicle. For example, the camera module120may be arranged to shoot the rear of the vehicle100. The camera module120may shoot a front surface of a rear vehicle following the vehicle100and generate image data IMG. Thus, the processing device110may detect the bounding box BOX5bincluding the front surface of the rear vehicle and extract the features F51bto F55bfrom the bounding box BOX5b. For example, as shown inFIG. 5B, the processing device110may extract the features F51band F52bcorresponding to the side mirror, the features F53band F54bcorresponding to a headlamp, and the feature F55bcorresponding to the license plate from the image data IMG. In some embodiments, the processing device110may extract only some of the features F51bto F55bofFIG. 5Bor further extract additional features (e.g., a wheel, a fog light, or the like) in addition to the features F51bto F55bofFIG. 5B.

Referring toFIG. 5C, the bounding box BOX5cmay include the side surface and the rear surface of the distant vehicle. For example, the camera module120may be arranged to shoot the front of the vehicle100. The camera module120may shoot a distant vehicle, which is traveling on a different lane from the vehicle100or stopped on the side of a road and generate image data IMG. Thus, the processing device110may detect the bounding box BOX5cincluding the rear surface and the side surface of the distant vehicle and extract the features F51cto F56cin the bounding box BOX5c. For example, as shown inFIG. 5C, the processing device110may extract the features F51cand F52ccorresponding to the tail lamp, the feature F53ccorresponding to the side mirror, the features F54cand F55ccorresponding to a wheel, and the feature F56ccorresponding to the license plate from the image data IMG. In some embodiments, the processing device110may extract only some of the features F51cto F56cofFIG. 5Cor further extract additional features in addition to the features F51cto F56cofFIG. 5C.

As described above with reference toFIGS. 1 and 2, the extracted features of the distant vehicle may be used to determine the posture of the distant vehicle and used to estimate the geometry of the road on which the distant vehicle is located. Hereinafter, a feature corresponding to the license plate will be mainly described as an example of the feature of the distant vehicle, but it will be understood that example embodiments are not limited thereto. In addition, although not shown inFIGS. 5A to 5C, in some embodiments, a bounding box including the front surface and the side surface of the distant vehicle may be detected from image data IMG, which is generated by the camera module120located to shoot the rear of the vehicle100, and features may be featured in the bounding box. Hereinafter, a feature corresponding to a specific portion of the distant vehicle may be simply referred to as the corresponding specific portion. For example, a feature corresponding to the license plate of the distant vehicle may be simply referred to as the license plate of the distant vehicle.

FIG. 6is a flowchart of a method of estimating a road geometry, according to an example embodiment.FIGS. 7A to 7Care diagrams of examples of a road geometry and image data according to example embodiments. Specifically, the flowchart ofFIG. 6illustrates an example of operation S80ofFIG. 2.FIGS. 7A to 7Cillustrate examples of the arrangement of host vehicles71a,73a,71b,73b,71c, and73cand front vehicles72a,74a,72b,74b,72c, and74c, and bounding boxes BOX7a, BOX7b, and BOX7cand features F7a, F7b, and F7c, which correspond thereto. As described above with reference toFIG. 2, an operation of estimating a geometry of a road may be performed in operation S80aofFIG. 6. Hereinafter, the host vehicles71a,73a,71b,73b,71c, and73cofFIGS. 7A to 7Cwill be assumed to include the processing device110and the camera module120ofFIG. 1.FIGS. 6 and 7A to 7Cwill be described with reference toFIG. 1.

Referring toFIG. 6, operation S80amay include operations S82aand S84a. In operation S82a, an operation of measuring a longitudinal position of a feature in a bounding box may be performed. Hereinafter, as described below with reference toFIGS. 7A to 7C, a posture of a distant vehicle may be changed due to the geometry of the road, and thus, the longitudinal position of the feature of the distant vehicle may be changed due to a profile of the road.

Referring to the upper portion ofFIG. 7A, in Case A, the host vehicle71aand the front vehicle72amay travel on a flat, horizontal road. Also, in Case B, the host vehicle73aand the front vehicle74amay travel on a flat, uphill road. That is, in Cases A and B ofFIG. 7A, the host vehicles71aand73aand the front vehicles72aand74amay be traveling on a road having the same geometry. When rear surfaces of the front vehicles72aand74aare shot at the host vehicles71aand73a, similar images may be obtained.

Referring to the lower portion ofFIG. 7A, in Case A or Case B ofFIG. 7A, the processing device110may detect the bounding box BOX7aincluding the rear surface of the front vehicle72aor74aand extract the license plate F7aas a feature. The bounding box BOX7amay have a height H7a, and the processing device110may measure a first distance Y1afrom a top of the bounding box BOX7ato a bottom of the license plate F7aand/or a second distance Y2afrom the bottom of the license plate F7ato a bottom of the bounding box BOX7a. In some embodiments, unlike that shown inFIG. 7A, the processing device110may measure a longitudinal position of a center or a top of the license plate F7ain the bounding box BOX7a.

The processing device110may define the longitudinal position of the feature by various metrics. For example, the processing device110may define the longitudinal position of the license plate F7aby the first distance Y1aor the second distance Y2a, a ratio between the first distance Y1aand the second distance Y2a, or a ratio of the first distance Y1aor the second distance Y2ato the height H7aof the bounding box BOX7a. In some embodiments, the processing device110may measure each of longitudinal positions of a plurality of features (e.g., a license plate and a tail lamp) and calculate one metric from the measured longitudinal positions. Hereinafter, it is assumed that the processing device110measures a ratio Y2a/Y1aof the second distance Y2ato the first distance Y1aas the longitudinal position of the license plate F7a. However, example embodiments are not limited thereto.

Referring to the upper portion ofFIG. 7B, in Case A, the host vehicle71bmay be traveling at a flat, horizontal position of the road, while the front vehicle72bmay be traveling on at a flat, uphill position of the road. Also, in Case B, the host vehicle73bmay be traveling at the flat, horizontal position of the road, while the front vehicle74bmay be traveling on a bump71of the road, and a front wheel of the front vehicle74bmay be on the bump71. Thus, each of the front vehicles72band74bmay have a posture with a front portion elevated. In Cases A and B ofFIG. 7B, when rear surfaces of the front vehicles72band74bare shot at the host vehicles71band73b, similar images may be obtained.

Referring to the lower portion ofFIG. 7B, in Cases A and B ofFIG. 7B, the processing device110may detect the bounding box BOX7bincluding the rear surface of the front vehicle72bor74band extract the license plate F7bas a feature. The bounding box BOX7bmay have a height H7b, and the processing device110may measure a first distance Y1bfrom a top of the bounding box BOX7bto a bottom of the license plate F7band/or a second distance Y2bfrom a bottom of the bounding box BOX7bto the bottom of the license plate F7b. As compared to the bounding box BOX7aofFIG. 7A, the bounding box BOX7bofFIG. 7Bmay further include a portion of a top surface of the front vehicle72bor74bdue to the posture of the front vehicle72bor74b. In addition, as compared to the license plate F7aofFIG. 7A, the license plate F7bofFIG. 7Bmay be located relatively lower in the bounding box BOX7bdue to the posture of the front vehicle72bor74b. Thus, a longitudinal position (Y2b/Y1b) of the license plate F7bofFIG. 7Bmay be lower than the longitudinal position (Y2a/Y1a) of the license plate F7aofFIG. 7A(Y2b/Y1b<Y2a/Y1a).

Referring to the upper portion ofFIG. 7C, in Case A, the host vehicle71cmay be traveling at a flat, horizontal position of the road, while the front vehicle72cmay be traveling at a flat, downhill position of the road. Also, in Case B, the host vehicle73cmay be traveling at a flat, horizontal position of the road, while the front vehicle74cmay be traveling on a bump72of the road, and a rear wheel of the front vehicle74cmay be on the bump72. Thus, each of the front vehicles72cand74cmay have a posture with the rear portion elevated. In Cases A and B ofFIG. 7C, when rear surfaces of the front vehicles72cand74care shot at the host vehicles71cand73c, similar images may be obtained.

Referring to the lower portion ofFIG. 7C, in Cases A and B ofFIG. 7C, the processing device110may detect the bounding box BOX7cincluding a rear surface of the front vehicle72cor74cand extract the license plate F7cas a feature. The bounding box BOX7cmay have a height H7c, and the processing device110may measure a first distance Y1cfrom a top of the bounding box BOX7cto a bottom of the license plate F7cand/or a second distance Y2cfrom a bottom of the bounding box BOX7cto the bottom of the license plate F7c. As compared to the bounding box BOX7aofFIG. 7A, the bounding box BOX7cofFIG. 7Cmay further include a portion of a bottom surface of the front vehicle72cor74cdue to the posture of the front vehicle72cor74c. In addition, as compared to the license plate F7aofFIG. 7A, the license plate F7cofFIG. 7Cmay be located relatively higher in the bounding box BOX7cdue to the posture of the front vehicle72cor74c. Thus, a longitudinal position (Y2c/Y1c) of the license plate F7cofFIG. 7may be higher than the longitudinal position (Y2a/Y1a) of the license plate F7aofFIG. 7A(Y2c/Y1c>Y2a/Y1a).

Referring back toFIG. 6, in operation S84a, an operation of estimating a profile of a road may be performed. For example, as described above with reference toFIGS. 7A to 7C, the processing device110may estimate the profile of the road, which indicates road undulations, based on a longitudinal position of a license plate that varies depending on a posture of a front vehicle. For example, the processing device110may estimate that a profile of a point on which the front vehicle is located is the same as a profile of a point on which a host vehicle is located, based on the longitudinal position (Y2a/Y1a) of the license plate F7aofFIG. 7A. Also, based on the longitudinal position (Y2b/Y2a) of the license plate F7bofFIG. 7B, the processing device110may estimate that the point on which the front vehicle is located has an ascent or a bump. Furthermore, based on the longitudinal position (Y2c/Y1c) of the license plate F7cofFIG. 7C, the processing device110may estimate that the point on which the front vehicle is located has a descent or a bump. In some embodiments, as described below with reference toFIG. 10, the processing device110may estimate a front position of the estimated profile of the road, based on a distance from the front vehicle. As described below with reference toFIG. 12, the profile of the road may be compensated for based on a state of a host vehicle. As described below with reference toFIG. 13, the profile of the road may be estimated based on a difference between a previous longitudinal position and a current longitudinal position of the license plate.

FIG. 8is a flowchart of a method of estimating a road geometry, according to an example embodiment.FIG. 9illustrates an example of image data according to an example embodiment. Specifically, the flowchart ofFIG. 8illustrates an example of operation S80ofFIG. 2, andFIG. 9illustrates a bounding box BOX9and a license plate F9serving as a feature in image data that is generated by shooting a front vehicle. As described above with reference toFIG. 2, an operation of estimating a geometry of a road may be performed in operation S80bofFIG. 8. Hereinafter,FIGS. 8 and 9will be described with reference toFIG. 1.

Referring toFIG. 8, operation S80bmay include operations S82band S84b. In operation S82b, an operation of detecting a slope of a feature in a bounding box may be performed. For example, referring toFIG. 9, the bounding box BOX9including a rear surface of the front vehicle may be detected, and the license plate F9may be extracted as the feature in the bounding box BOX9. As shown inFIG. 9, the license plate F9may have an inclination, which is rotated by an angle θ from the horizon (or a horizontal line of the bounding box BOX9) in a counterclockwise direction, and the front vehicle may also be determined to be rotated by an angle θ from the horizon in the counterclockwise direction. In some embodiments, the processing device110may detect a first line91parallel to the license plate F9and measure an angle θ formed by a second line92corresponding to the horizon with the first line91.

In operation S84b, an operation of estimating a cross slope of a road may be performed. For example, as shown inFIG. 9, when the front vehicle is rotated by an angle θ in the counterclockwise direction, the processing device110may estimate a cross slope of a road, which is inclined by an angle θ in the counterclockwise direction. In some embodiments, as described below with reference toFIG. 10, the processing device110may estimate a front position at which the cross slope of the road is estimated, based on a distance from the front vehicle. As described below with reference toFIG. 12, the processing device110may estimate the cross slope of the road, based on a state of the host vehicle. As described below with reference toFIG. 13, the cross slope of the road may be estimated based on a difference between a previous inclination and a current inclination of the license plate F9.

FIG. 10is a flowchart of a method of estimating a road geometry, according to an example embodiment. Specifically, the flowchart ofFIG. 10illustrates an example of operation S80ofFIG. 2. As described above with reference toFIG. 2, an operation of estimating a geometry of a road may be performed in operation S80cofFIG. 10. As shown inFIG. 10, operation S80cmay include operations S82cand S84cand will be described below with reference toFIG. 1.

In operation S82c, an operation of obtaining a distance from a distant vehicle may be performed. In some embodiments, the at least one sensor130may include a distance sensor configured to measure a distance from the distant vehicle, and the processing device110may obtain the distance from the distant vehicle, based on a sense signal SEN provided by the at least one sensor130. In some embodiments, the processing device110may estimate the distance from the distant vehicle, based on image data IMG provided by the camera module120. For example, a size and a position of a bounding box of the distant vehicle in the image data IMG may depend on a distance from the distant vehicle, and thus, the processing device110may estimate the distance from the distant vehicle, based on at least one of the size and the position of the bounding box. In some embodiments, the processing device110may obtain the distance from the distant vehicle, based on both the sense signal SEN and the image data IMG.

In some embodiments, the processing device110may convert the image data IMG into data corresponding to a top-view image. In the top-view image, distant vehicles may be arranged according to a distance from a host vehicle. For example, the processing device110may convert the image data IMG into the data corresponding to the top-view image, based on a homography matrix and inverse perspective mapping.

In operation S84c, an operation of generating three-dimensional (3D) data may be performed. For example, the processing device110may generate 3D data indicating the estimated geometry of the road and store the 3D data in the memory113. As described above, 3D data indicating a road geometry may be utilized for useful functions. Examples of the road geometry indicated by the 3D data will be described below with reference toFIGS. 11A and 11B.

FIGS. 11A and 11Billustrate examples of a road geometry that is estimated according to example embodiments. Specifically,FIGS. 11A and 11Bschematically illustrate examples of the road geometry indicated by the 3D data generated due to operation S80cofFIG. 10. As described above with reference toFIG. 10, the estimated road geometry may be expressed as the 3D data. Hereinafter,FIGS. 11A and 11Bwill be described with reference toFIG. 1.

Referring toFIG. 11A, a road geometry11a, which is estimated, may indicate a profile and a cross slope of a road. For example, as shown inFIG. 11A, the road geometry11amay include a bump11_2, and a left side of the road geometry11amay have a higher shape than a right side thereof in a direction in which a vehicle travels.

Referring toFIG. 11B, a road geometry11b, which is estimated, may indicate a state of the road or a state of pavement. For example, as shown inFIG. 11B, the road geometry11bmay include a region11_4indicating cobblestone pavement and a region11_6indicating even pavement.

FIG. 12is a flowchart of a method of estimating a road geometry, according to an example embodiment. Specifically, the flowchart ofFIG. 12illustrates an example of operation S80ofFIG. 2. As described above with reference toFIG. 2, an operation of estimating a geometry of a road may be performed in operation S80dofFIG. 12. As shown inFIG. 12, operation S80dmay include operations S82dand S84d. Hereinafter, the flowchart ofFIG. 12will be described with reference toFIG. 1.

In operation S82d, an operation of obtaining state information during the shooting of a distant vehicle may be performed. For example, the processing device110may obtain a state of the vehicle100(i.e., a state of the camera module120) during the shooting of the distant vehicle, based on a sense signal SEN provided by the at least one sensor130. As in Cases A and B ofFIG. 7A, a distant vehicle may be equally or similarly represented in image data IMG despite different geometries of roads. Thus, the processing device110may obtain state information during the shooting of the distant vehicle to compensate for geometric information about a road on which the vehicle100is located, in a geometry of the road on which the distant vehicle is located. For example, the at least one sensor130may include an acceleration sensor or a gyro sensor, which may sense a posture of the vehicle100(or the camera module120), and the processing device110may obtain information about the state of the vehicle100during the shooting of the distant vehicle, based on the sense signal SEN.

In operation S84d, an operation of compensating for the estimated geometry of the road may be performed. In some embodiments, the processing device110may determine a profile of the point on which the vehicle100is located, based on the state information obtained in operation S82d, and compensate for a profile of the point, which has been estimated based on the image data IMG, based on the determined profile. For example, as in Case B ofFIG. 7A, while both the vehicle100and the distant vehicle are traveling on an uphill road having substantially the same slope angle, the processing device110may determine a first slope angle of the uphill road on which the vehicle100is located, based on the sense signal SEN, and estimate a slope angle of the point on which the distant vehicle is located, by adding the first slope angle to a second slope angle (e.g., approximately zero (0) degrees) of the estimated road, based on the image data IMG.

In some embodiments, the processing device110may determine a cross slope of the point on which the vehicle100is located, based on the state information obtained in operation S83d, and compensate for a cross slope of the road, which is estimated based on the image data IMG, based on the determined cross slope. For example, the processing device110may determine a first cross slope of the point on which the vehicle100is located, based on the sense signal SEN, and estimate the cross slope of the point on which the distant vehicle is located, by adding the first cross slope to a second cross slope (e.g., the angle θ ofFIG. 9) of the road, which is estimated based on the image data IMG.

FIG. 13is a flowchart of a method of estimating a road geometry, according to an example embodiment. Specifically, the flowchart ofFIG. 13illustrates an example of operation S80ofFIG. 2. As described above with reference toFIG. 2, an operation of estimating a geometry of a road may be performed in operation S80eofFIG. 13. As shown inFIG. 13, operation S80emay include operations S82eand S84e. Hereinafter,FIG. 13will be described with reference toFIG. 1.

In operation S82e, an operation of obtaining a previous position of a feature in a bounding box may be performed. For example, the processing device110may store data about a bounding box detected in image data IMG and a feature extracted in the bounding box in the memory113. To estimate the geometry of the road from image data IMG that is currently received, the processing device110may read data (e.g., the data about the bounding box and the feature), which is generated due to previously received image data IMG, from the memory113. Thus, the processing device110may obtain a previous position of a feature relative to at least a portion of the bounding box.

In operation S84e, an operation of estimating the geometry of the road, based on a difference between the previous position and a current position of the feature, may be performed. In some embodiments, the processing device110may estimate the geometry based on a variation between the previous position and the current position of the feature. For example, when a longitudinal position of a license plate of a distant vehicle becomes higher than a previous position thereof, the processing device110may estimate a profile of a road having a lower slope angle than a profile of a previously estimated road. A decrement in slope angle may be determined based on a difference between the previous position and a current position of the license plate.

FIG. 14is a flowchart of a method of estimating a road geometry, according to an example embodiment. Similarly to the method ofFIG. 2, the method ofFIG. 14may include operations S20′, S40′, S60′, and S80′ and further include operation S90. Hereinafter, the same description as inFIG. 2will be omitted from a description ofFIG. 14, and the flowchart ofFIG. 14will be described with reference toFIG. 1.

In operation S20′, an operation of obtaining image data IMG generated by shooting a distant vehicle may be performed. In operation S40′, an operation of detecting a bounding box of the distant vehicle in the image data IMG may be performed. In operation S60′, an operation of extracting a feature of the distant vehicle from the image data IMG may be performed. In operation S80′, an operation of estimating a geometry of a road on which the distant vehicle is located, based on the bounding box and the feature, may be performed.

In operation S90, an operation of assisting lane detection may be performed. The lane detection may be utilized for various functions, such as autonomous driving, adaptive cruise control (ACC), lane departure warning (LDW), a lane keeping assistance system (LKAS), lane centering control (LCC), and the like. A shape of a lane shown in the image data IMG may vary depending on a geometry of a road in which the lane is located. Thus, the processing device110may assist the lane detection based on the geometry of the road, which is estimated in operation S80′. As a result, the accuracy of the lane detection may be improved.

FIG. 15is a block diagram of a vehicle200according to an example embodiment. As shown inFIG. 15, the vehicle200may include a propulsion device220, an electronic device240, a peripheral device260, and a driving device280.

The propulsion device220may include an engine/motor221, an energy source222, a transmission223, a wheel/tire224, a suspension225, and a shock absorber226. The engine/motor221may include any combination of an internal combustion engine, an electric motor, a steam engine, and a Stirling engine. In some embodiments, when the vehicle200is a gas-electric hybrid car, the engine/motor221may include a gasoline engine and an electric motor. The energy source222may be an energy source that at least partially provides power to the engine/motor221, and the engine/motor221may convert the energy source222into kinetic energy.

Non-limiting examples of the energy source222may include at least one of gasoline, diesel, propane, other compressed gas-based fuels, ethanol, a solar panel, a battery, and other electrical power sources. In some embodiments, the energy source222may include at least one of a fuel tank, a battery, a capacitor, and a flywheel. Also, the energy source222may provide energy not only to the engine/motor221but also to other components of the vehicle200.

The transmission223may transmit mechanical power from the engine/motor221to the wheel/tire224. For example, the transmission223may include at least one of a gearbox, a clutch, a differential, and a drive shaft. When the transmission223includes drive shafts, the drive shafts may include at least one wheel axle coupled to the wheel/tire224. The wheel/tire224may have various structures for a bicycle, a motorcycle, a four-wheeled vehicle, and the like and be in contact with pavement.

The suspension225, which is a device configured to support a weight of the vehicle200, may adjust a ground clearance from pavement to the vehicle200or adjust vibration transmitted from the pavement to the vehicle200. The shock absorber226may control the vibration of a spring, which is received from the pavement during the driving of the vehicle200and assist in restoring the spring to an original state. For example, the shock absorber226may generate damping force for stopping the vibration of the spring and control the elasticity of the spring. In some embodiments, the shock absorber226may be included in the suspension225.

The electronic device240may include a controller241, a processing device242, a storage243, a user interface244, at least one sensor245, and a power supply246. The controller241may control the vehicle200and be also referred to as an electronic control unit (ECU). For example, the controller241may control the propulsion device220and the driving device280to control the driving of the vehicle200and may control the peripheral device260. In some embodiments, as described below with reference toFIG. 16, the controller241may control the vehicle200based on a road geometry provided by the processing device242.

The processing device242may perform various operations for assisting the driving of the vehicle200. For example, as described above with reference to the drawings, the processing device242may receive image data from an image sensor included in the at least one sensor245and estimate a road geometry based on the image data. Also, the processing device242may store data indicating the estimated road geometry in the storage243or provide the data to the controller241. In some embodiments, as described above with reference toFIG. 1, the processing device242may include a first processor and a second processor.

The storage243may store data and include, for example, a non-volatile semiconductor memory device, a volatile semiconductor memory device, and/or a disk drive. The user interface244may include an input device configured to receive a user input and an output device configured to provide an output signal to a user. For example, the input device may include a keypad, a dome switch, a touch pad, a jog wheel, a jog switch, and/or a microphone. Also, the output device may include a speaker and/or a buzzer configured to output an audio signal and a display device and/or a light-emitting diode (LED) configured to output a video signal.

The at least one sensor245may include a sensor configured to sense a state of the vehicle200. For example, the at least one sensor245may include a motion sensor, such as a geomagnetic sensor, an acceleration sensor, and a gyro sensor, or include a GPS sensor configured to estimate a position of the vehicle200. Also, the at least one sensor245may include a sensor configured to sense a surrounding state of the vehicle200. For example, the at least one sensor245may include a RADAR sensor configured to sense presence and/or speeds of objects around the vehicle200using a radio signal or include a LIDAR sensor configured to sense presence and/or speeds of objects around the vehicle200using laser. Furthermore, the at least one sensor245may include at least one image sensor (or a camera module including an image sensor) configured to shoot the vicinity of the vehicle200. The image sensor may provide image data, which is generated by shooting a distant vehicle, to the processing device242.

The power supply246may provide power to at least some of components of the vehicle200. For example, the power supply246may include a generator configured to generate power due to the driving of the vehicle200or a battery configured to store electric power.

The peripheral device260may include a headlamp261, a tail lamp262, a blinker263, an internal lamp264, a wiper265, and a conditioner266. The headlamp261may be arranged on a front surface of the vehicle200, and the tail lamp262may be arranged on a rear surface of the vehicle200. As described above with reference toFIGS. 5A to 5C, the headlamp261and/or the tail lamp262may be extracted as features by another vehicle. The blinker263may be arranged on the front surface, the rear surface, and a side surface of the vehicle200, and the internal lamp264may be arranged in a driver's space. The wiper265may reciprocate on glass arranged on the front surface and/or the rear surface of the vehicle200. The conditioner266may include an air conditioner and/or a heater.

The driving device280may include a brake unit281, a steering unit282, and a throttle283. The brake unit281may be implemented as a combination of mechanisms configured to decelerate the vehicle200. For example, the brake unit281may use friction to reduce a rotational speed of the wheel/tire224. The steering unit282may be implemented as a combination of mechanisms configured to adjust a direction in which the vehicle200travels. The throttle283may be implemented as a combination of mechanisms configured to control an operation speed of the engine/motor221. For example, the throttle283may adjust the amount of a gas mixture of fuel air flowing into the engine/motor221and control power and thrust.

FIG. 16is a flowchart of a method of estimating a road geometry, according to an example embodiment. Specifically, the flowchart ofFIG. 16illustrates a method S100of controlling a vehicle based on the estimated road geometry. As shown inFIG. 16, the method S100of controlling the vehicle may include a plurality of operations S110, S130, S150, S170, and S190. In some embodiments, the method S100of controlling the vehicle may include only some of the plurality of operations S110, S130, S150, S170, and S190. In some embodiments, the method S100of controlling the vehicle may be performed by the controller241ofFIG. 15. Hereinafter,FIG. 16will be described with reference toFIG. 15.

In operation S110, an operation of regulating the suspension225may be performed. In operation S130, an operation of regulating a damper of the shock absorber226may be performed. For example, when an ascending profile is estimated by the processing device242, the controller241may regulate the suspension225and the shock absorber226to reduce vibration transmitted from pavement to the vehicle200. Also, when a bump is estimated, the controller241may regulate the suspension225so that a ground clearance of the vehicle200may be elevated.

In operation S150, an operation of regulating the transmission223may be performed. For example, when an ascending profile is estimated by the processing device242, the controller241may regulate the transmission223to increase a transmission gear ratio. In operation S170, a steering regulation operation may be performed. For example, when a curved road having a cross slope is estimated by the processing device242, the controller241may regulate the steering unit282along the curved road. In operation S190, an operation of regulating the headlamp261may be performed. For example, when a descending profile is estimated by the processing device242, the controller241may regulate the headlamp261so that light emitted by the headlamp261may be directed downward.