Source: https://patents.google.com/patent/US10032083B2/en
Timestamp: 2020-01-24 05:32:04
Document Index: 125832301

Matched Legal Cases: ['§ 119', 'Application No. 10', 'Application No. 15003302', 'Application No. 15003302', 'Application No. 10', 'Application No. 15003302']

US10032083B2 - Driver assistance for a vehicle - Google Patents
Driver assistance for a vehicle Download PDF
US10032083B2
US10032083B2 US14/964,995 US201514964995A US10032083B2 US 10032083 B2 US10032083 B2 US 10032083B2 US 201514964995 A US201514964995 A US 201514964995A US 10032083 B2 US10032083 B2 US 10032083B2
US14/964,995
US20160171315A1 (en
Woonki PARK
2014-12-10 Priority to KR1020140177580A priority Critical patent/KR101641490B1/en
2014-12-10 Priority to KR10-2014-0177580 priority
2015-12-10 Application filed by LG Electronics Inc filed Critical LG Electronics Inc
2016-06-16 Publication of US20160171315A1 publication Critical patent/US20160171315A1/en
2018-06-26 Assigned to LG ELECTRONICS INC. reassignment LG ELECTRONICS INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PARK, Woonki, PARK, Yeonchool, YI, Chuho
2018-07-24 Publication of US10032083B2 publication Critical patent/US10032083B2/en
Systems and techniques are disclosed for providing driver assistance in a vehicle. A driver assistance apparatus includes a stereo camera configured to acquire stereo images of a view ahead of a vehicle, and a processor configured to generate a depth map based on the acquired stereo images and to determine whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map.
Pursuant to 35 U.S.C. § 119(a), this application claims the benefit of an earlier filing date and right of priority to Korean Patent Application No. 10-2014-0177580, filed on Dec. 10, 2014 in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference.
Systems and techniques are disclosed that enable a driver assistance apparatus that determines whether a road ahead of a vehicle, on which the vehicle will travel, is an uphill road or a downhill road based on stereo images captured by a camera in the vehicle.
In one aspect, a driver assistance apparatus includes a stereo camera configured to acquire stereo images of a view ahead of a vehicle and a processor configured to generate a depth map based on the acquired stereo images and determine whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map.
Implementations may include one or more of the following features. For example, the processor may be further configured to detect a road surface based on the generated depth map and determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected road surface. The processor may be further configured to detect a lane from the stereo images, determine a shape of the detected lane, and determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined shape of the detected lane.
In some examples, the processor may be further configured to determine a disappearance point indicated in the depth map and determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined disappearance point indicated in the depth map. Also, the processor may be further configured to detect a plurality of fixed objects around the road segment indicated in the generated depth map and determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected plurality of fixed objects around the road segment indicated in the generated depth map.
In some implementations, the driver assistance apparatus may include a stereo camera drive unit configured to drive the stereo camera and the processor may be configured to control the stereo camera drive unit based on a determination of whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment. In these implementations, the processor may be further configured to determine that the road segment that is within the view ahead of the vehicle is an uphill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment, determine a tilt of the uphill road segment and control the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
In some examples, the processor may be further configured to determine that the road segment that is within the view ahead of the vehicle is an uphill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment, determine a travelling speed of the vehicle and control the stereo camera to be tilted upward at a speed that is proportional to the determined traveling speed of the vehicle. In these examples, the processor may be further configured to determine that a distance between the vehicle and an inflection point of the uphill road segment is at least a predetermined distance and, based on a determination that the distance between the vehicle and the inflection point of the uphill road segment is at least the predetermined distance, control the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
Also, the processor may be further configured to determine that the vehicle has entered the uphill road segment and, based on a determination that the vehicle has entered the uphill road segment, control the stereo camera to return to a state in which the stereo camera is not tilted. In addition, the processor may be configured to determine a travelling speed of the vehicle and control, in proportion to the determined traveling speed of the vehicle and according to the determined tilt of the uphill road segment, a speed of the stereo camera to return to the state in which the stereo camera is not tilted. Further, the processor may be further configured to determine that the road segment that is within the view ahead of the vehicle is a downhill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment, determine a tilt of the downhill road segment and control the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
In some implementations, the processor may be further configured to determine that the road segment that is within the view ahead of the vehicle is a downhill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment, determine a travelling speed of the vehicle and control the stereo camera to be tilted downward at a speed that is proportional to the determined traveling speed of the vehicle. In these implementations, the processor may be configured to determine that a distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance and, based on a determination that the distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance, control the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
In some examples, the processor may be configured to determine that the vehicle has entered the downhill road segment and the stereo camera drive unit may be configured to, based on a determination that the vehicle has entered the downhill road segment, drive the stereo camera to return to a state in which the stereo camera is not tilted. In these examples, the processor may be configured to determine a travelling speed of the vehicle and control, according to the determined tilt of the downhill road segment or in proportion to the determined traveling speed of the vehicle, the stereo camera to return to the state in which the stereo camera is not tilted.
In some implementations, the processor may be further configured to detect lines in the acquired stereo images and generate, based on the detected lines, information regarding a curve of the road segment that is within the view ahead of the vehicle. In some implementations, the driver assistance apparatus may include a stereo camera drive unit configured to drive the stereo camera. In these implementations, the processor may be further configured to determine, based on the generated information regarding a curve of the road segment that is within the view ahead of the vehicle, that the road segment that is within the view ahead of the vehicle is curved to the right or the left in a direction of travel of the vehicle, detect an outermost line among the detect lines that is most distant from a center of curvature of the road segment that is within the view ahead of the vehicle, detect a tilt of the outermost line, and control the stereo camera drive unit to tilt the stereo camera upward or downward according to the detected tilt of the outermost line.
Also, the driver assistance apparatus may include a display unit configured to output information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle. In addition, the driver assistance apparatus may include a first memory, and the processor may be further configured to cumulatively store, in the first memory, information regarding a state of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle.
In another aspect, a vehicle includes a brake drive unit configured to drive a brake of the vehicle and a stereo camera configured to acquire stereo images for a view ahead of the vehicle. The vehicle also includes a processor configured to generate a depth map based on the acquired stereo images, determine, based on the generated depth map, whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment, generate road surface information based on the acquired stereo images, and output a control signal configured to control the brake drive unit based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the generated road surface information.
Implementations may include one or more of the following features. For example, the vehicle may include a navigation system configured to provide a map and information regarding a position of the vehicle on the map. In this example, the processor may be configured to receive, from the navigation system, information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images and estimate, based on the received information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images, a state of the road segment.
In some implementations, the vehicle may include a slip determination unit configured to determine a slip degree of the vehicle. In these implementations, the processor may be further configured to output the control signal to control the brake drive unit of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the determined slip degree of the vehicle. Also, in these implementations, the vehicle may includes a first memory and a power source drive unit configured to control an output of an engine of the vehicle, where the output of the engine is configured to limit a speed of the vehicle. Further, in these implementations, the processor may be further configured to cumulatively store, in the first memory, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle, apply a Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment, estimate the slip degree of the vehicle based on an application of the Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment, and output the control signal to control, based on the estimated slip degree of the vehicle, the power source drive unit or the brake drive unit to drive the brake of the vehicle at a level equivalent to 90% or less than a full brake level.
In yet another aspect, a driver assistance method includes acquiring, by a stereo camera of a vehicle, stereo images of a view ahead of the vehicle and generating a depth map based on the acquired stereo images. The method also includes determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map.
Implementations may include one or more of the following features. For example, the method may include detecting a road surface from the generated depth map and determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected road surface. Also, the method may include detecting a lane from the stereo images, determining a shape of the detected lane, and determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined shape of the detected lane.
In addition, the method may include determining a disappearance point indicated in the depth map and determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined disappearance point indicated in the depth map. Further, the method may include detecting a plurality of fixed objects around the road segment indicated in the depth map and determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected plurality of fixed objects around the road segment indicated in the depth map.
In some implementations, the method may include controlling a stereo camera drive unit to drive the stereo camera based on a determination of whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment. In these implementations, the method may include determining that the road segment that is within the view ahead of the vehicle is an uphill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment, determining a tilt of the uphill road segment. Further, in these implementations, the method may include driving the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
In some examples, the method may include determining that the road segment that is within the view ahead of the vehicle is an uphill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment, determining a travelling speed of the vehicle and driving the stereo camera to be tilted upward at a speed that is proportional to the determined traveling speed of the vehicle. In these examples, the method may include determining that a distance between the vehicle and an inflection point of the uphill road segment is at least a predetermined distance and, based on a determination that the distance between the vehicle and the inflection point of the uphill road segment is at least the predetermined distance, driving the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
In some implementations, the method may include determining that the vehicle has entered the uphill road segment and, based on a determination that the vehicle has entered the uphill road segment, driving the stereo camera to return to a state in which the stereo camera is not tilted. In these implementations, the method may include determining a travelling speed of the vehicle and driving, in proportion to the determined traveling speed of the vehicle and according to the determined tilt of the uphill road segment, the stereo camera to return to the state in which the stereo camera is not tilted.
Also, the method may include determining that the road segment that is within the view ahead of the vehicle is a downhill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment, determining a tilt of the downhill road segment. The method further may include driving the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
In some examples, the method may include determining that the road segment that is within the view ahead of the vehicle is a downhill road segment and, based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment, determining a travelling speed of the vehicle and driving the stereo camera to be tilted downward at a speed that is proportional to the determined traveling speed of the vehicle. In addition, the method may include determining that a distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance and, based on a determination that the distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance, driving the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
In some implementations, the method may include determining that the vehicle has entered the downhill road segment and, based on a determination that the vehicle has entered the downhill road segment, driving the stereo camera to return to a state in which the stereo camera is not tilted. In these implementations, the method may includes determining a travelling speed of the vehicle and driving, according to the determined tilt of the downhill road segment or in proportion to the determined traveling speed of the vehicle, the stereo camera to return to the state in which the stereo camera is not tilted.
In some examples, the method may include detecting lines in the acquired stereo images and generating, based on the detected lines, information regarding a curve of the road segment that is within the view ahead of the vehicle. In these examples, the method may include controlling a stereo camera drive unit to drive the stereo camera, determining, based on the generated information regarding a curve of the road segment that is within the view ahead of the vehicle, that the road segment that is within the view ahead of the vehicle is curved to the right or the left in a direction of travel of the vehicle, detecting an outermost line among the detect lines that is most distant from a center of curvature of the road segment that is within the view ahead of the vehicle, detecting a tilt of the outermost line, and controlling the stereo camera drive unit to tilt the stereo camera upward or downward according to the detected tilt of the outermost line.
The method may include outputting, to a display unit, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle. The method also may include cumulatively storing, in a first memory, information regarding a state of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle. The method further may include controlling a brake drive unit to drive a brake of the vehicle, generating road surface information based on the acquired stereo images, and outputting a control signal configured to control the brake drive unit based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the generated road surface information. In addition, the method may include receiving, from a navigation system that is configured to provide a map and information regarding a position of the vehicle on the map, information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images from a navigation system and estimating, based on the received information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images, a state of the road segment.
In some implementations, the method may include determining a slip degree of the vehicle and outputting the control signal to control the brake drive unit of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the determined slip degree of the vehicle. In these implementations, the method may include controlling an output of an engine of the vehicle through a power source drive unit of the vehicle, wherein the output of the engine is configured to limit a speed of the vehicle, cumulatively storing, in a first memory, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle, applying a Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment, estimating the slip degree of the vehicle based on an application of the Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment, and outputting the control signal to control, based on the estimated slip degree of the vehicle, the power source drive unit or the brake drive unit to drive the brake of the vehicle at a level equivalent to 90% or less than a full brake level.
FIG. 1 is a diagram illustrating an example of a vehicle equipped with a stereo camera;
FIG. 2 is a diagram illustrating an example of a stereo camera attached to a vehicle;
FIGS. 4A, 4B, and 4C are internal block diagrams illustrating various examples of a processor of a driver assistance apparatus;
FIGS. 5A and 5B are diagrams illustrating examples of operations of a processor of a driver assistance apparatus;
FIGS. 8, 9A, and 9B are flowcharts illustrating examples of an operation of a driver assistance apparatus;
FIGS. 10A, 10B, 11A, and 11B are diagrams illustrating examples of a determination of an uphill road and a downhill road;
FIGS. 12A, 12B, 13A, 13B, 14A, 14B, 15A, and 15B are diagrams illustrating examples of a motion of a stereo camera corresponding to an uphill road and a downhill road; and
FIGS. 16A and 16B are diagrams illustrating examples of an operation of a driver assistance apparatus in a scenario in which a road ahead of a vehicle is a curved uphill road or a curved downhill road.
Information regarding a road ahead of a vehicle during traveling of the vehicle may be acquired through a navigation system. However, it is not possible to acquire correct information regarding whether the road ahead of the vehicle is an uphill or a downhill. Consequently, there is a high necessity for research on acquisition of information regarding whether the road ahead of the vehicle is the uphill or the downhill and controlling of a unit equipped in the vehicle.
Systems and techniques are disclosed that enable a driver assistance apparatus to determine whether a road ahead of a vehicle, on which the vehicle will travel, is an uphill road or a downhill road based on stereo images captured by a camera in the vehicle.
For ease of description in the present disclosure, an X axis may refer to a direction in which an overall width of a vehicle is measured, a Y axis may refer to a direction in which an overall height of the vehicle is measured, and a Z axis may refer to a direction in which an overall length of the vehicle is measured.
FIG. 1 is a diagram illustrating an example of a vehicle equipped with a stereo camera.
The stereo camera 195 may include one or more cameras. Stereo images acquired by the cameras may be signal-processed in a driver assistance apparatus 100 (see, e.g., FIG. 3).
The example of FIG. 1 shows, by way of example, that the stereo camera 195 includes two cameras.
In present disclosure, the stereo camera 195 may also be referred to as a stereo camera module 195.
FIG. 2 is a diagram illustrating an example of a stereo camera attached to a vehicle (e.g., the vehicle shown in FIG. 1).
Referring to the example in FIG. 2, the stereo camera module 195 may include a first camera 195 a having a first lens 193 a and a second camera 195 b having a second lens 193 b.
The stereo camera module 195 may also be referred to as a stereo image acquisition device.
In some implementations, the stereo camera module 195 may further include a first light shield unit 192 a to shield light incident upon the first lens 193 a and a second light shield unit 192 b to shield light incident upon the second lens 193 b.
The stereo camera module 195 shown in FIG. 2 may be detachably attached to a ceiling or a windshield of the vehicle 200.
A driver assistance apparatus 100 (see, e.g., FIG. 3) equipped with the stereo camera module 195 with the above-stated construction may acquire stereo images for a view ahead of the vehicle from the stereo camera module 195, detect disparity based on the stereo images, detect an object for at least one of the stereo images based on disparity information, and continuously track motion of the object after detection of the object.
Referring to the examples in FIGS. 3A and 3B, a driver assistance apparatus 100 may perform signal-processing of stereo images received from the stereo camera 195 (e.g., based on computer vision) to create vehicle-related information. The vehicle-related information may include vehicle control information for direct control of the vehicle or driver assistance information for driving guidance provided to a driver.
Referring first to the example in FIG. 3A, the driver assistance apparatus 100 of FIG. 3A may include a communication unit 120, an interface unit 130, a first memory 140, a processor 170, an electric power supply unit 190, a stereo camera 195, and a stereo camera drive unit 196. The stereo camera drive unit 196 may also be referred to as a stereo camera drive device.
The communication unit 120 may exchange data with a mobile terminal 600 or a server 500 in a wireless fashion. For example, the communication unit 120 may exchange data with a mobile terminal of the driver (e.g., in a wireless fashion). Any suitable wireless data communication protocol may be used, such as Bluetooth, Wi-Fi, Wi-Fi Direct, and/or Automotive Pixel Link (APiX).
The communication unit 120 may receive various types of information, such as weather information and/or road traffic state information such as Transport Protocol Expert Group (TPEG) information, from the mobile terminal 600 or the server 500. In some implementations, the communication unit 120 may transmit real-time traffic information acquired by the driver assistance apparatus 100 based on stereo images to the mobile terminal 600 or the server 500.
The interface unit 130 may receive vehicle-related data or transmit a signal processed or created by the processor 170 to the outside of the vehicle. To this end, the interface unit 130 may perform data communication with a control unit 770, an audio video navigation (AVN) apparatus 400, and/or a sensor unit 760 in the vehicle in a wired communication fashion or a wireless communication fashion.
The interface unit 130 may receive map information related to vehicle travel through data communication with the AVN apparatus 400. For example, the AVN apparatus 400 may include a navigation system, and the interface unit 130 may receive a map and information regarding the position of the vehicle on the map from the navigation system and transmit the map and the vehicle position information to the processor 170.
The sensor information may include information regarding at least one of slip vehicle degree, vehicle heading information, vehicle position information (global positioning system (GPS) information), vehicle angle information, vehicle speed information, vehicle acceleration information, vehicle tilt information, vehicle forward movement/backward movement information, battery information, fuel information, tire information, vehicle lamp information, in-vehicle temperature information, and/or in-vehicle humidity information.
In addition, the processor 170 may, in some implementations, calculate the distance to the detected object (e.g., an adjacent vehicle), a speed of the detected adjacent vehicle, and a difference in speed with the detected adjacent vehicle.
In some implementations, the processor 170 may create a depth map based on stereo images and determine whether a road ahead of the vehicle, on which the vehicle will travel, is an uphill or a downhill, based on the created depth map. In this case, the processor 170 may function as an uphill/downhill determination device.
For example, the processor 170 may detect a road surface from the stereo images or the depth map and determine whether the road ahead of the vehicle is the uphill or the downhill based on the detected road surface.
As another example, the processor 170 may detect a lane from the stereo images or the depth map and determine whether the road ahead of the vehicle is the uphill or the downhill based on the shape of the detected lane.
As another example, the processor 170 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a disappearance point indicated in the stereo images or the depth map.
As another example, the processor 170 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a plurality of fixed objects around the road indicated in the stereo images or the depth map.
In some implementations, the processor 170 may detect a lane or a line from the stereo images or the depth map and create information regarding a curve of the road based on the detected lane or line. In this case, the processor 170 may function as a curve information creation device. For example, the processor 170 may detect a lane based on lane-dividing lines located at opposite sides of the vehicle which is traveling on the road and create information regarding a curve of the road. Alternatively, the processor 170 may detect a lane based on the central line of the road and create information regarding a curve of the road. For example, the processor 170 may detect a lane through Hough transformation and create information regarding a curve of the road. The information regarding the curve of the road may include curvature of the road.
In some implementations, the processor 170 may cumulatively store information regarding the determined uphill or downhill of the road ahead of the vehicle in the first memory 140. In this case, the processor 170 may function as a storage device. The processor 170 may apply one or more processing techniques, such as a Kalman filter, to the cumulatively stored information regarding the uphill or the downhill of the road ahead of the vehicle. In this case, the processor 170 may estimate a vehicle slip degree based on the uphill or downhill information to which the Kalman filter has been applied. In this case, the processor 170 may function as a vehicle slip degree estimation device. For example, the processor 170 may receive data cumulatively stored in a state in which the uphill or downhill information is matched with the slip information through the interface unit 130. The processor 170 may estimate a vehicle slip degree corresponding to the uphill or downhill to which the Kalman filter has been applied based on the received matching information. The processor 170 may output a control signal to control a power source drive unit or a brake drive unit to the control unit 770 such that a brake can be driven at a level equivalent to a fraction (e.g., 90% or less) of a full brake (which means a case in which the brake is fully driven) based on the estimated slip degree.
In some implementations, the processor 170 may output the information regarding the uphill or the downhill of the road ahead of the vehicle through a display unit 180. In this case, the processor 170 may function as an output device. The processor 170 may graphically process the motion of the stereo camera 195 corresponding to the uphill or the downhill of the road ahead of the vehicle and output the graphically-processed motion of the stereo camera 195 through the display unit 180.
The processor 170 may cumulatively store a state of the vehicle based on the uphill or the downhill of the road ahead of the vehicle in the first memory 140. For example, the processor 170 may cumulatively store the motion of the stereo camera 195 corresponding to a tilt of the uphill of the road ahead of the vehicle or a tilt of the downhill of the road ahead of the vehicle in the first memory 140. In this case, the processor 170 may apply a Kalman filter based on the cumulatively stored information regarding the motion of the stereo camera 195. In a case in which the Kalman filter is applied, it the processor 170 may be able to accurately control the motion of the stereo camera 195.
In some implementations, the processor 170 may generate information regarding the road surface on which the vehicle is traveling based on the stereo images. For example, the processor 170 may classify a state of the road surface into one of several states, such as a dry state, a wet state, a snowy state, and a frozen state based on the stereo images. In some implementations, this classification may be performed based on differences in brightness data of the stereo images. For example, the snowy state may have the highest brightness level, and the dry state, the frozen state, and the wet state may have sequentially lower brightness levels. As described above, therefore, the state of the road surface may be classified into the dry state, the wet state, the snowy state, and the frozen state based on the brightness. In another example, the processor 170 of the driver assistance apparatus 100 may classify the road surface data into the dry state, the wet state, the snowy state, and the frozen state based on intensity and exposure of the images. The processor 170 may output a control signal to control the brake drive unit to the control unit 770 based on the uphill or the downhill of the road ahead of the vehicle and road surface information.
In addition, the processor 170 may receive information regarding the road on which the vehicle 200 is traveling excluding a region of the road displayed in the stereo images from the navigation system to estimate a state of the road. In this case, the processor 170 may function as a road state estimation device. As a specific example, the processor 170 may estimate a state of the road ahead of the vehicle or at the rear of the vehicle which is not displayed in the stereo images. The road state may include a curve of the road, a tunnel, and/or a number of lanes in the road.
In some cases, the processor 170 may be implemented using digital signal processors (DSPs), application specific integrated circuits (ASICs), or micro-controllers. The processor 170 may be mounted on one surface of a predetermined circuit board.
The stereo camera 195 may be detachably attached to a ceiling or a windshield of the vehicle 200. The stereo camera 195 may include a first camera 195 a having a first lens 193 a and a second camera 195 b having a second lens 193 b.
In some implementations, the stereo camera 195 may further include a first light shield unit 192 a to shield light incident upon the first lens 193 a and a second light shield unit 192 b to shield light incident upon the second lens 193 b.
The stereo camera drive unit 196 may drive the stereo camera 195 under control of the processor 170. For example, the stereo camera drive unit 196 may include a drive unit, such as a motor, which may move the stereo camera 195 upward and downward and from side to side under control of the processor 170. In this case, the stereo camera drive unit 196 may include a drive force transmission unit to transmit drive force generated by the drive means to the stereo camera 195.
For example, the stereo camera drive unit 196 may tilt or pan the stereo camera 195.
Referring now to the example in FIG. 3B, a driver assistance apparatus 100 of FIG. 3B may further include an input unit 110, a display unit 180, and an audio output unit 185 as compared with the example of a driver assistance apparatus 100 of FIG. 3A. Hereinafter, a description will be given of only the input unit 110, the display unit 180, and the audio output unit 185.
The display unit 180 may display an image related to the operation of the driver assistance apparatus. In order to display such an image, the display unit 180 may include a cluster or a head up display (HUD) provided at the inside front of the vehicle. In a case in which the display unit 180 is a HUD, the display unit 180 may include a projection module to project an image on the windshield of the vehicle 200.
FIGS. 4A to 4C are internal block diagrams showing various examples of a processor (e.g., the processors shown in FIGS. 3A and 3B). FIGS. 5A and 5B are reference views illustrating examples of operations of a processor (e.g., the processors shown in FIGS. 4A and 4B).
For example, the object verification unit 436 may verify various types of objects, such as an adjacent vehicle, a lane, a road surface, a traffic sign, a dangerous zone, a tunnel, etc. located around the vehicle.
FIG. 4C is an internal block diagram showing a further example of the processor 170.
Referring to FIG. 4C, the processor 170 of FIG. 4C is different from the processor 170 of FIG. 4A in that the processor 170 of FIG. 4C further includes other units which are not included in the processor 170 of FIG. 4A. Hereinafter, a description will be given of only the difference between the processor 170 of FIG. 4C and the processor 170 of FIG. 4A.
The processor 170 of the driver assistance apparatus 100 may further include a depth map creation unit 460, a road profile unit 464, and a lane detection unit 468.
The depth map creation unit 460 may create a depth map based on preprocessed images. To this end, various methods, such as dynamic programming (DP), belief propagation (BP), and graph-cut, may be used.
The road profile unit 464 may profile a road surface based on the depth map. The road profile unit 464 may analyze a state of the road on a YZ plane based on the depth map. The road profile unit 464 may analyze a change of the road surface based on the depth map.
The lane detection unit 468 may detect a lane on the road based on the image segment from the segmentation unit 432 or the depth map. Lane detection may be executed using a well-known algorithm.
The application unit 450 may determine whether a road ahead of the vehicle is an uphill or a downhill.
For example, the application unit 450 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a state or change of the road surface analyzed by the road profile unit 464. The application unit 450 may calculate X, Y, and Z axis reference data of the road from the depth map. In particular, the application unit 450 may calculate road surface data on the YZ plane. At this time, rise and fall of the road surface on the YZ plane may be detected. For example, in a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a positive direction of the Y axis, the application unit 450 may determine that the road ahead of the vehicle is the uphill. On the other hand, in a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a negative direction of the Y axis, the application unit 450 may determine that the road ahead of the vehicle is the downhill. In addition, the application unit 450 may determine a tilt of the uphill or the downhill of the road ahead of the vehicle based on the length of the road surface on the Z axis and the length of the road surface on the Y axis.
For example, the application unit 450 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a change of the detected lane. In a case in which the lane is detected based on the image segment from the segmentation unit 432, the application unit 450 may analyze a change of the lane (e.g. whether the lane extends while having a predetermined angle in the positive direction of the Y axis or the lane extends while having a predetermined angle in the negative direction of the Y axis) to determine whether the road ahead of the vehicle is the uphill or the downhill. On the other hand, in a case in which the lane is detected based on the depth map, the application unit 450 may determine whether the road ahead of the vehicle is the uphill or the downhill through the same process as for the road surface.
In another example, the application unit 450 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a plurality of fixed objects around the road indicated in the stereo images or the depth map. The fixed objects around the road may include streetlights, roadside tree, or electric poles. The application unit 450 may track positions, relative motions, or sizes of objects continuously arranged along the road to determine whether the road ahead of the vehicle is the uphill or the downhill.
In a further example, the application unit 450 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a disappearance point indicated in the stereo images or the depth map. In a case in which the disappearance point in the stereo images is located above a reference line, the application unit 450 may determine that the road ahead of the vehicle is the uphill. On the other hand, in a case in which the disappearance point in the stereo images is located below the reference line, the application unit 450 may determine that the road ahead of the vehicle is the downhill.
The disparity map 520 shows a disparity between the stereo images FR1 a and FR1 b as different levels. When a disparity level of a region (e.g., an object) in the image is higher, the distance from the region to the vehicle may be calculated as being smaller. When a disparity level is lower, on the other hand, the distance to the vehicle may be calculated as being greater.
FIG. 5A shows, by way of example, that, in the disparity map 520, first to fourth lanes 528 a, 528 b, 528 c, and 528 d have their own disparity levels, and a construction zone 522, a first preceding vehicle 524, and a second preceding vehicle 526 have their own disparity levels.
The segmentation unit 432, the object detector 434, and the object verification unit 436 may respectively perform segmentation, object detection, and object verification for at least one of the stereo images FR1 a and FR1 b based on the disparity map 520.
FIG. 5A shows, by way of example, that object detection and verification for the second stereo image FR1 b are performed using the disparity map 520.
For example, object detection and verification for first to fourth lanes 538 a, 538 b, 538 c, and 538 d, a construction zone 532, a first preceding vehicle 534, and a second preceding vehicle 536 in an image 530 may be performed.
Referring now to FIG. 5B, the stereo camera 195 acquires stereo images during the second frame period.
The disparity calculator 420 of the processor 170 receives stereo images FR2 a and FR2 b signal-processed by the image preprocessor 410 and performs stereo matching for the received stereo images FR2 a and FR2 b to acquire a disparity map 540.
FIG. 5B shows, by way of example, that, in the disparity map 540, first to fourth lanes 548 a, 548 b, 548 c, and 548 d have their own disparity levels, and a construction zone 542, a first preceding vehicle 544, and a second preceding vehicle 546 have their own disparity levels.
The segmentation unit 432, the object detector 434, and the object verification unit 436 respectively perform segmentation, object detection, and object verification for at least one of the stereo images FR2 a and FR2 b based on the disparity map 540.
FIG. 5B shows, by way of example, that object detection and verification for the second stereo image FR2 b are performed using the disparity map 540.
For example, object detection and verification for first to fourth lanes 558 a, 558 b, 558 c, and 558 d, a construction zone 552, a first preceding vehicle 554, and a second preceding vehicle 556 in an image 550 may be performed.
Referring to FIG. 6A, a first lane 642 a, a second lane 644 a, a third lane 646 a, and a fourth lane 648 a are arranged from the left side to the right side of the image. A construction zone 610 a is located between the first lane 642 a and the second lane 644 a. A first preceding vehicle 620 a is located between the second lane 644 a and the third lane 646 a. A second preceding vehicle 630 a is located between the third lane 646 a and the fourth lane 648 a.
Referring to FIG. 6B, a first lane 642 b, a second lane 644 b, a third lane 646 b, and a fourth lane 648 b are arranged from the left side to the right side of the image. A construction zone 610 b is located between the first lane 642 b and the second lane 644 b. A first preceding vehicle 620 b is located between the second lane 644 b and the third lane 646 b. A second preceding vehicle 630 b is located between the third lane 646 b and the fourth lane 648 b.
The driver assistance apparatus 100 may perform signal processing based on stereo images taken by the stereo camera 195 to verify objects for the construction zone 610 b, the first preceding vehicle 620 b, and the second preceding vehicle 630 b. In addition, the driver assistance apparatus 100 may verify the first lane 642 b, the second lane 644 b, the third lane 646 b, and the fourth lane 648 b.
FIG. 6B shows, by way of example, that, in order to indicate that the objects for the construction zone 610 b, the first preceding vehicle 620 b, and the second preceding vehicle 630 b have been verified, borders of the objects are highlighted.
In some implementations, the driver assistance apparatus 100 may calculate distance information for the construction zone 610 b, the first preceding vehicle 620 b, and the second preceding vehicle 630 b based on stereo images taken by the stereo camera 195.
FIG. 6B shows, by way of example, that calculated first distance information 611 b, calculated second distance information 621 b, and calculated third distance information 631 b respectively corresponding to the construction zone 610 b, the first preceding vehicle 620 b, and the second preceding vehicle 630 b are displayed.
In some implementations, the driver assistance apparatus 100 may receive speed limit information for a road on which the vehicle is traveling through the communication unit 120 or the interface unit 130. FIG. 6B shows, by way of example, that speed limit information 640 b is displayed.
For example, in a case in which the power source is an engine using fossil fuel, the power source drive unit 754 may electronically control the engine. Consequently, the power source drive unit 754 may control output torque of the engine. In a case in which the power source is the engine, the power source drive unit 754 may restrict output torque of the engine to limit the vehicle speed under control of the control unit 770. The power source drive unit 754 may correspond to a power source drive device.
The control unit 770 may receive information regarding an uphill or a downhill of a road ahead of the vehicle from the driver assistance apparatus 100. The control unit 770 may store the information regarding the uphill or the downhill of the road ahead of the vehicle and slip information created by a slip determination unit 771 in the second memory 740 in a state in which the information regarding the uphill or the downhill of the road ahead of the vehicle is matched with the slip information. In such cases, the control unit 770 may cumulatively store the matched information.
In some implementations, the control unit 770 may receive the information regarding the uphill or the downhill of the road ahead of the vehicle cumulatively stored in the first memory 140. The control unit 770 may apply a Kalman filter to the cumulatively stored information regarding the uphill or the downhill of the road ahead of the vehicle. In this case, the control unit 770 may estimate a vehicle slip degree based on the uphill or downhill information to which the Kalman filter has been applied. For example, the control unit 770 may estimate a vehicle slip degree corresponding to the uphill or downhill to which the Kalman filter has been applied based on the slip information stored while being matched with the uphill or downhill information. The control unit 770 may control the power source drive unit or the brake drive unit such that the brake can be driven at a level equivalent to a fraction (e.g., 90% or less) of the full brake based on the estimated slip degree.
In some implementations, the control unit 770 may apply the Kalman filter to the slip information cumulatively stored in the second memory 740 to estimate a slip degree of the vehicle. The control unit 770 may control the power source drive unit 754 or the brake drive unit 753 such that the brake can be driven at a level equivalent to 90% or less the full brake based on the estimated slip degree and the information regarding the uphill or the downhill of the road ahead of the vehicle.
The electric power supply unit 790 may supply electric power to the respective components under control of the control unit 770. For example, electric power from an in-vehicle battery may be supplied to the electric power supply unit 790.
In some implementations, the electronic control apparatus 700 of the vehicle 200 may further include a slip determination unit 771. The slip determination unit 771 may be included in the control unit 770 as a subcomponent. For example, the sensor unit 760 may include a wheel speed sensor. The wheel speed sensor may sense a wheel rotational speed corresponding to vehicle acceleration or deceleration. The wheel rotational speed sensed by the wheel speed sensor may be input to the control unit 770. The control unit 770 may compare a wheel rotational speed calculated by the power source with the wheel rotational speed sensed by the wheel speed sensor to determine whether the vehicle has slipped.
In another example, the slip determination unit 771 may determine whether the vehicle has slipped using odometry. The slip determination unit 771 may compare a distance by which the vehicle 200 is to be moved using power generated by the power source drive unit 754 with a distance by which the vehicle 200 has been actually moved to determine whether the vehicle has slipped. Specifically, in a case in which the distance by which the vehicle 200 has been actually moved is shorter than the distance by which the vehicle 200 is to be moved, the slip determination unit 771 may determine that the vehicle has slipped. In addition, the slip determination unit 771 may determine a slip degree of the vehicle in proportion to the distance difference. At this time, the distance by which the vehicle 200 has been actually moved may be calculated based on speed information and moving time.
The slip determination unit 771 may, in some implementations, cumulatively store the slip degree of the vehicle in the second memory 740.
FIGS. 8 to 9B are flowcharts illustrating examples of an operation of a driver assistance apparatus.
Referring to FIG. 8, the processor 170 receives stereo images from the stereo camera 195 (which corresponds to a stereo image acquisition step) (S810).
In a state in which the processor 170 receives the stereo images, the processor 170 determines an uphill of the road ahead of the vehicle, a downhill of the road ahead of the vehicle, or a tilt of the road ahead of the vehicle based on the stereo images (S830).
For example, the processor 170 may create a depth map based on preprocessed stereo images. The processor 170 may profile a road surface based on the depth map. The processor 170 may determine whether the road ahead of the vehicle is the uphill or the downhill based on profiling of the road surface. For example, the profiled road surface may provide rise and fall of the road surface on the YZ plane. In a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a positive direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the uphill. On the other hand, in a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a negative direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the downhill. In addition, the processor 170 may determine a tilt of the uphill or the downhill based on the profiled road surface. For example, the processor 170 may determine the tilt of the uphill or the downhill based on the length of the road surface on the Z axis and the length of the road surface on the Y axis. In this case, the processor 170 may function as an uphill tilt determination device or a downhill tilt determination device.
For example, the processor 170 may detect a lane from the stereo images. The processor 170 may determine whether the road ahead of the vehicle is the uphill or the downhill based on the detected lane. In a case in which the lane extends to a predetermined length or more while having a predetermined angle in the positive direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the uphill. On the other hand, in a case in which the lane extends to a predetermined length or more while having a predetermined angle in the negative direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the downhill. In addition, the processor 170 may determine a tilt of the uphill or the downhill based on the lane. For example, the processor 170 may determine the tilt of the uphill or the downhill based on the length of the lane on the Z axis and the length of the lane on the Y axis.
In another example, the processor 170 may detect a plurality of fixed objects around the road from the stereo images. The objects may include streetlights, roadside tree, or electric poles. The processor 170 may track positions, relative motions, or sizes of objects continuously arranged along the road to determine whether the road ahead of the vehicle is the uphill or the downhill. For example, in a case in which streetlights having the same size are arranged along one side of the road, the processor 170 may calculate the distances to the streetlights through disparity calculation. The distances to first and second streetlights on the Z axis may be compared with the distances to the first and second streetlights on the Y axis to determine whether the road ahead of the vehicle is the uphill or the downhill. In addition, the processor 170 may determine a tilt of the uphill or the downhill based on the objects. For example, the processor 170 may determine the tilt of the uphill or the downhill based on the distances to the first and second streetlights on the Z axis and the distances to the first and second streetlights on the Y axis.
In a further example, the processor 170 may determine whether the road ahead of the vehicle is the uphill or the downhill based on a disappearance point indicated in the stereo images. In a case in which the disappearance point in the stereo images is located above a reference line, the processor 170 may determine that the road ahead of the vehicle is the uphill. On the other hand, in a case in which the disappearance point in the stereo images is located below the reference line, the processor 170 may determine that the road ahead of the vehicle is the downhill.
In a state in which it is determined whether the road ahead of the vehicle is the uphill or the downhill, the processor 170 controls the stereo camera drive unit 196 based on determination as to whether the road ahead of the vehicle is the uphill or the downhill.
FIGS. 9A and 9B are flowcharts illustrating further details of step S850 of FIG. 8.
Referring to the example in FIG. 9A, in a case in which it is determined that the road ahead of the vehicle is the uphill (S910), the processor 170 outputs a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward (S915). In this case, the processor 170 may function as a control signal output device. The processor 170 controls the stereo camera 195 to be moved upward in proportion to a tilt of the uphill. For example, the processor 170 may control the stereo camera 195 to be moved upward in response to the tilt of the uphill. As a result, the stereo camera 195 may be moved upward before the vehicle 200 enters the uphill, whereby it is possible to adaptively acquire images for a view ahead of the vehicle 200.
On the other hand, in a case in which it is determined that the road ahead of the vehicle is not the uphill but is the downhill (S910 and S920), the processor 170 outputs a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward (S925). The processor 170 controls the stereo camera 195 to be moved downward in response to a tilt of the downhill. For example, the processor 170 may control the stereo camera 195 to be moved downward in response to the tilt of the downhill. As a result, the stereo camera 195 may be moved downward before the vehicle 200 enters the downhill, whereby it is possible to adaptively acquire images for a view ahead of the vehicle 200.
In a case in which it is determined that the road ahead of the vehicle is neither the uphill nor the downhill (S910 and S920), the processor 170 maintains the attitude of the stereo camera 195 (S930).
Referring to the example in FIG. 9B, in a case in which it is determined that the road ahead of the vehicle is the uphill (S950), the processor 170 determines whether the vehicle has reached a predetermined distance from an inflection point (S955). The inflection point may be a start point of the uphill. The processor 170 may set an inflection point based on the depth map and determine whether the vehicle 200 has reached a predetermined distance from the inflection point based on information regarding the distance to the inflection point.
In a case in which the vehicle 200 has reached the predetermined distance from the inflection point, the processor 170 outputs a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward (S960). The processor 170 controls the stereo camera 195 to be moved upward in response to a tilt of the uphill. The stereo camera 195 may be moved upward before the vehicle 200 enters the uphill, whereby it is possible to adaptively acquire images for a view ahead of the vehicle 200.
On the other hand, in a case in which it is determined that the road ahead of the vehicle is not the uphill but is the downhill (S950 and S970), the processor 170 determines whether the vehicle has reached a predetermined distance from an inflection point (S975). The inflection point may be a start point of the downhill. The processor 170 may set an inflection point based on the depth map and determine whether the vehicle 200 has reached a predetermined distance from the inflection point based on information regarding the distance to the inflection point.
In a case in which the vehicle 200 has reached the predetermined distance from the inflection point, the processor 170 outputs a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward (S980). The processor 170 controls the stereo camera 195 to be moved downward in response to a tilt of the downhill. The stereo camera 195 may be moved downward before the vehicle 200 enters the downhill, whereby it is possible to adaptively acquire images for a view ahead of the vehicle 200.
In a case in which it is determined that the road ahead of the vehicle is neither the uphill nor the downhill (S950 and S970), the processor 170 maintains the attitude of the stereo camera 195 (S965). In some implementations, in a case in which the road ahead of the vehicle is the uphill but the vehicle 200 has not reached the predetermined distance from the inflection point (S950 and S955), the processor 170 maintains the attitude of the stereo camera 195 (S965). On the other hand, in a case in which the road ahead of the vehicle is the downhill but the vehicle 200 has not reached the predetermined distance from the inflection point (S970 and S975), the processor 170 maintains the attitude of the stereo camera 195 (S965).
FIGS. 10A to 11B are diagrams illustrating examples of an operation of a processor (e.g., the processor 170 of FIG. 4C) of a driver assistance apparatus.
FIGS. 10A and 10B are diagrams showing examples of images for a view ahead of the vehicle 200 that can be acquired by the stereo camera 195. FIGS. 11A and 11B are views showing the images for the view ahead of the vehicle shown in FIG. 10A on a YZ plane and an XZ plane.
FIGS. 10A and 10B are based on an XY plane. FIG. 10A shows an uphill of the road ahead of the vehicle, and FIG. 10B shows a downhill of the road ahead of the vehicle.
Referring first to FIGS. 10A, 11A and 11B, the stereo camera 195 acquires stereo images 1000R and 1000L, and the processor 170 receives the stereo images 1000R and 1000L.
The processor 170 matches the stereo images 1000R and 1000L using global matching or local matching to create a depth map 1000D.
The depth map 1000D may include a plurality of roadside trees 1010 a, 1010 b, 1010 c, and 1010 d, a plurality of streetlights 1015 a, 1015 b, 1015 c, and 1015 d, a road surface 1020, and lanes 1030 a, 1030 b, and 1030 c.
The processor 170 may detect the road surface 1020 from the depth map and determine whether the road ahead of the vehicle is the uphill based on the detected road surface. The processor 170 may calculate X, Y, and Z axis reference data of the road from the depth map. In particular, the processor 170 may calculate data of the road surface 1020 on the YZ plane. A rise and fall of the road surface 1020 on the YZ plane may be detected. In a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a positive direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the uphill. In some implementations, the processor 170 may determine a tilt of the uphill of the road ahead of the vehicle based on profiling of the road surface. For example, the processor 170 may determine the tilt of the uphill of the road ahead of the vehicle based on a length (a) of a portion of the road surface on the Z axis and a length (b) of the portion of the road surface on the Y axis.
The processor 170 may detect the lanes 1030 a, 1030 b, and 1030 c based on the stereo images 1000R and 1000L or the depth map 1000D. In a case in which the detected lanes 1030 a, 1030 b, and 1030 c extend to a predetermined length or more while having a predetermined angle in the positive direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the uphill. In some implementations, the processor 170 may determine a tilt of the uphill of the road ahead of the vehicle based on the lanes. For example, the processor 170 may determine the tilt of the uphill of the road ahead of the vehicle based on a length (a) of a portion of each lane on the Z axis and a length (b) of the portion of each lane on the Y axis.
The processor 170 may determine whether the road ahead of the vehicle is the uphill based on a plurality of fixed objects 1010 a, 1010 b, 1010 c, 1010 d, 1015 a, 1015 b, 1015 c, and 1015 d around the road indicated in the stereo images 1000R and 1000L or the depth map 1000D. For example, in a case in which streetlights 1015 a, 1015 b, 1015 c, and 1015 d having the same size are arranged along one side of the road, the processor 170 may calculate the distances to the streetlights 1015 a, 1015 b, 1015 c, and 1015 d through disparity calculation. At this time, the distances to first and second streetlights 1015 a and 1015 d on the Z axis may be compared with the distances to the first and second streetlights 1015 a and 1015 d on the Y axis to determine whether the road ahead of the vehicle is the uphill. As a specific example, in a case in which the distance between the second streetlight 1015 d and the vehicle 200 on the Z axis is greater than that between the first streetlight 1015 a and the vehicle 200 on the Z axis, and the distance between the second streetlight 1015 d and the vehicle 200 on the Y axis is greater than that between the first streetlight 1015 a and the vehicle 200 on the Y axis, the processor 170 may determine that the road ahead of the vehicle is the uphill. On the other hand, the processor 170 may determine a tilt of the uphill of the road ahead of the vehicle based on the objects. For example, the processor 170 may determine the tilt of the uphill of the road ahead of the vehicle based on distances (a) to the first and second streetlights on the Z axis and distances (b) to the first and second streetlights on the Y axis.
Although not shown, in a case in which a disappearance point in the stereo images is located above a reference line, the processor 170 may determine that the road ahead of the vehicle is the uphill.
Referring now to the example in FIG. 10B, the stereo camera 195 acquires stereo images 1050R and 1050L, and the processor 170 receives the stereo images 1050R and 1050L.
The processor 170 matches the stereo images 1050R and 1050L using global matching or local matching to create a depth map 1050D.
The depth map 1050D may include a plurality of roadside trees 1060 a, 1060 b, 1060 c, and 1060 d, a plurality of streetlights 1065 a, 1065 b, 1065 c, and 1065 d, a road surface 1070, and lanes 1080 a, 1080 b, and 1080 c.
The processor 170 may detect the road surface 1070 from the depth map and determine whether the road ahead of the vehicle is the downhill based on the detected road surface. The processor 170 may calculate X, Y, and Z axis reference data of the road from the depth map. In particular, the processor 170 may calculate data of the road surface 1080 on the YZ plane. A rise and fall of the road surface 1080 on the YZ plane may be detected. In a case in which the road surface extends to a predetermined length or more while having a predetermined angle in a negative direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the downhill. In some implementations, the processor 170 may determine a tilt of the downhill of the road ahead of the vehicle based on profiling of the road surface. For example, the processor 170 may determine the tilt of the downhill of the road ahead of the vehicle based on the length of the road surface on the Z axis and the length of the road surface on the Y axis.
The processor 170 may detect the lanes 1080 a, 1080 b, and 1080 c based on the stereo images 1050R and 1050L or the depth map 1050D. In a case in which the detected lanes 1080 a, 1080 b, and 1080 c extend to a predetermined length or more while having a predetermined angle in the negative direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the downhill. In some implementations, the processor 170 may determine a tilt of the downhill of the road ahead of the vehicle based on the lanes. For example, the processor 170 may determine the tilt of the downhill of the road ahead of the vehicle based on the length of each lane on the Z axis and the length of each lane on the Y axis.
The processor 170 may determine whether the road ahead of the vehicle is the downhill based on a plurality of fixed objects 1060 a, 1060 b, 1060 c, 1060 d, 1065 a, 1065 b, 1065 c, and 1065 d around the road indicated in the stereo images 1050R and 1050L or the depth map 1050D. For example, in a case in which streetlights 1065 a, 1065 b, 1065 c, and 1065 d having the same size are arranged along one side of the road, the processor 170 may calculate the distances to the streetlights 1065 a, 1065 b, 1065 c, and 1065 d through disparity calculation. The distances to first and second streetlights 1065 a and 1065 d on the Z axis may be compared with the distances to the first and second streetlights 1065 a and 1065 d on the Y axis to determine whether the road ahead of the vehicle is the downhill.
For example, in a case in which the distance between the second streetlight 1065 d and the vehicle 200 on the Z axis is lesser than that between the first streetlight 1065 a and the vehicle 200 on the Z axis, and the distance between the second streetlight 1065 d and the vehicle 200 on the Y axis is greater than that between the first streetlight 1065 a and the vehicle 200 on the Y axis, the processor 170 may determine that the road ahead of the vehicle is the downhill. In some implementations, the processor 170 may determine a tilt of the downhill of the road ahead of the vehicle based on the objects. For example, the processor 170 may determine the tilt of the downhill of the road ahead of the vehicle based on the distances to the first and second streetlights on the Z axis and the distances to the first and second streetlights on the Y axis.
Although not shown, in a case in which a disappearance point in the stereo images is located below a reference line, the processor 170 may determine that the road ahead of the vehicle is the downhill.
FIGS. 12A to 15B are diagrams illustrating examples of a motion of a stereo camera corresponding to an uphill and a downhill road.
FIGS. 12A and 12B are diagrams illustrating examples of motions of the stereo camera 195 and the display unit 180 when the vehicle enters an uphill from a level ground in accordance with one embodiment of the present invention.
Referring to the example in FIG. 12A, when the vehicle 200 travels on a level ground 1210 in a positive direction of the Z axis, the processor 170 may determine that a road ahead of the vehicle is an uphill 1220. The processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. In this case, the processor 170 may control the stereo camera 195 to be moved upward in response to a tilt α of the uphill. For example, the processor 170 may control the stereo camera 195 to be moved upward in proportion to the tilt α of the uphill. Specifically, the processor 170 may control the stereo camera 195 to be tilted upward in proportion to the tilt α of the uphill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
In some implementations, the processor 170 may control the stereo camera 195 to be moved upward in response to a traveling speed of the vehicle 200. For example, the processor 170 may control an upward speed of the stereo camera 195 in proportion to the traveling speed of the vehicle 200.
In some implementations, when the vehicle 200 travels on the level ground 1210 in the positive direction of the Z axis, the processor 170 may determine that the road ahead of the vehicle is the uphill 1220. In a case in which it is determined that the road ahead of the vehicle is the uphill, the processor 170 determines whether the vehicle has reached a predetermined distance 1230 from an inflection point 1215. The inflection point 1215 may be a start point of the uphill. The processor 170 may set the inflection point 1215 based on the depth map and determine whether the vehicle 200 has reached the predetermined distance 1230 from the inflection point 1215 based on information regarding the distance to the inflection point 1215. The predetermined distance 1230 from the inflection point 1215 may be preset as a point at which the stereo camera 195 can be properly moved.
In a case in which the vehicle 200 has reached the predetermined distance 1230 from the inflection point 1215, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. The processor 170 may control the stereo camera 195 to be moved upward in response to the tilt α of the uphill. For example, the processor 170 may control the stereo camera 195 to be moved upward in proportion to the tilt α of the uphill. Specifically, the processor 170 may control the stereo camera 195 to be tilted upward in proportion to the tilt α of the uphill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
After the vehicle 200 has entered the uphill 1220, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. For example, after the vehicle 200 has entered the uphill 1220, the processor 170 may return the stereo camera 195 to the original position thereof. In this case, the processor 170 may return the stereo camera 195 to the original position thereof in response to the traveling speed of the vehicle 200. For example, the processor 170 may return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200. Specifically, the processor 170 may tilt the stereo camera 195 downward to return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200.
Referring to the example in FIG. 12B, when the vehicle 200 travels on the level ground 1210 in the positive direction of the Z axis, the stereo camera 195 assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200. As shown in the example of the left-most part of FIG. 12B, the processor 170 graphically processes an image 1210 a corresponding to the stereo camera 195 which assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200 and displays the graphically-processed image through the display unit 180. In addition, the processor 170 may display information 1201 indicating that there is an uphill predetermined meters ahead of the vehicle on a portion of the display unit 180.
When the vehicle 200 enters the uphill 1220 from the level ground 1210, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward immediately before the vehicle 200 enters the uphill 1220. As shown in the example of the middle part of FIG. 12B, the processor 170 graphically processes an image 1220 b corresponding to the stereo camera 195 which is moved upward and displays the graphically-processed image through the display unit 180.
After the vehicle 200 has entered the uphill 1220, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 returns to the original position thereof. As shown in the right-most part of FIG. 12B, the processor 170 graphically processes an image 1230 c corresponding to the stereo camera 195 which returns to the original position thereof and displays the graphically-processed image through the display unit 180.
In some implementations, the images 1210 a, 1220 b, and 1230 c corresponding to the stereo camera 195 shown in FIG. 12B may be three-dimensionally displayed or displayed as a video. This may enable the driver of the vehicle 200 to intuitively recognize a current state of the stereo camera 195 through the images 1210 a, 1220 b, and 1230 c displayed on the display unit 180.
FIGS. 13A and 13B are diagrams illustrating the motions of the stereo camera 195 and the display unit 180 when the vehicle enters a level ground from a downhill road.
Referring to FIG. 13A, when the vehicle 200 travels on a downhill 1310 in a positive direction of the Z axis and in a negative direction of the Y axis, the processor 170 may determine that a road ahead of the vehicle is a level ground 1320. The processor 170 may determine the level ground 1320 based on determination as to whether the road ahead of the vehicle is the uphill or the downhill as described above. In this case, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. For example, the processor 170 may control the stereo camera 195 to be moved upward in response to a tilt β of the downhill. As a specific example, the processor 170 may control the stereo camera 195 to be moved upward in proportion to the tilt β of the downhill. The processor 170 may control the stereo camera 195 to be tilted upward in proportion to the tilt β of the downhill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
In some implementations, when the vehicle 200 travels on the downhill 1310 in the positive direction of the Z axis and in the negative direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the level ground 1320. The processor 170 may determine the level ground 1320 based on determination as to whether the road ahead of the vehicle is the uphill or the downhill as described above. In a case in which it is determined that the road ahead of the vehicle is the level ground, the processor 170 determines whether the vehicle has reached a predetermined distance 1330 from an inflection point 1315. The inflection point 1315 may be a start point of the level ground. The processor 170 may set the inflection point 1315 based on the depth map and determine whether the vehicle 200 has reached the predetermined distance 1330 from the inflection point 1315 based on information regarding the distance to the inflection point 1315. In some implementations, the predetermined distance 1330 from the inflection point 1315 may be preset as a point at which the stereo camera 195 can be properly moved.
In a case in which the vehicle 200 has reached the predetermined distance 1330 from the inflection point 1315, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. In this case, the processor 170 may control the stereo camera 195 to be moved upward in response to the tilt β of the downhill. For example, the processor 170 may control the stereo camera 195 to be moved upward in proportion to the tilt β of the downhill. Specifically, the processor 170 may control the stereo camera 195 to be tilted upward in proportion to the tilt β of the downhill. In some implementations, the processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
After the vehicle 200 has entered the level ground 1320, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. For example, after the vehicle 200 has entered the level ground 1320, the processor 170 may return the stereo camera 195 to the original position thereof. In this case, the processor 170 may return the stereo camera 195 to the original position thereof in response to the traveling speed of the vehicle 200. For example, the processor 170 may return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200. Specifically, the processor 170 may tilt the stereo camera 195 downward to return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200.
Referring to the example in FIG. 13B, when the vehicle 200 travels on the downhill 1310 in the positive direction of the Z axis and in the negative direction of the Y axis, the stereo camera 195 assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200. In this case, as shown in the example of the left-most part of FIG. 13B, the processor 170 graphically processes an image 1310 a corresponding to the stereo camera 195 which assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200 and displays the graphically-processed image through the display unit 180. In addition, the processor 170 may display information 1301 indicating that there is a level ground predetermined meters ahead of the vehicle on a portion of the display unit 180.
When the vehicle 200 enters the level ground 1320 from the downhill 1310, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward (e.g., immediately) before the vehicle 200 enters the level ground 1320. In this case, as shown in the example of the middle part of FIG. 13B, the processor 170 graphically processes an image 1320 b corresponding to the stereo camera 195 which is moved upward and displays the graphically-processed image through the display unit 180.
After the vehicle 200 has entered the level ground 1320, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 returns to the original position thereof. For example, as shown in the right-most part of FIG. 13B, the processor 170 graphically processes an image 1330 c corresponding to the stereo camera 195 which returns to the original position thereof and displays the graphically-processed image through the display unit 180.
In some implementations, the images 1310 a, 1320 b, and 1330 c corresponding to the stereo camera 195 shown in FIG. 13B may be three-dimensionally displayed or displayed as a video.
Consequently, this may enable the driver of the vehicle 200 may intuitively recognize a current state of the stereo camera 195 through the images 1310 a, 1320 b, and 1330 c displayed on the display unit 180.
FIGS. 14A and 14B are diagrams illustrating examples of the motions of the stereo camera 195 and the display unit 180 when the vehicle enters a downhill from a level ground.
Referring to the example in FIG. 14A, when the vehicle 200 travels on a level ground 1410 in a positive direction of the Z axis, the processor 170 may determine that a road ahead of the vehicle is a downhill 1420. In this case, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. In some implementations, the processor 170 may control the stereo camera 195 to be moved downward in response to a tilt γ of the downhill. For example, the processor 170 may control the stereo camera 195 to be moved downward in proportion to the tilt γ of the downhill. Specifically, the processor 170 may control the stereo camera 195 to be tilted downward in proportion to the tilt γ of the downhill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
In some implementations, when the vehicle 200 travels on the level ground 1410 in the positive direction of the Z axis, the processor 170 may determine that the road ahead of the vehicle is the downhill 1420. In a case in which it is determined that the road ahead of the vehicle is the downhill, the processor 170 determines whether the vehicle has reached a predetermined distance 1430 from an inflection point 1415. The inflection point 1415 may be a start point of the downhill. The processor 170 may set the inflection point 1415 based on the depth map and determine whether the vehicle 200 has reached the predetermined distance 1430 from the inflection point 1415 based on information regarding the distance to the inflection point 1415. The predetermined distance 1430 from the inflection point 1415 may be preset as a point at which the stereo camera 195 can be properly moved.
In a case in which the vehicle 200 has reached the predetermined distance 1430 from the inflection point 1415, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. In this case, the processor 170 may control the stereo camera 195 to be moved downward in response to the tilt γ of the downhill. For example, the processor 170 may control the stereo camera 195 to be moved downward in proportion to the tilt γ of the downhill. Specifically, the processor 170 may control the stereo camera 195 to be tilted downward in proportion to the tilt γ of the downhill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
After the vehicle 200 has entered the downhill 1420, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. For example, after the vehicle 200 has entered the downhill 1420, the processor 170 may return the stereo camera 195 to the original position thereof. In this case, the processor 170 may return the stereo camera 195 to the original position thereof in response to the traveling speed of the vehicle 200. For example, the processor 170 may return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200. Specifically, the processor 170 may tilt the stereo camera 195 upward to return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200.
Referring to the example in FIG. 14B, when the vehicle 200 travels on the level ground 1410 in the positive direction of the Z axis, the stereo camera 195 assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200. In this case, as shown in the example of the left-most part of FIG. 14B, the processor 170 graphically processes an image 1410 a corresponding to the stereo camera 195 which assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200 and displays the graphically-processed image through the display unit 180. In addition, the processor 170 may display information 1401 indicating that there is a downhill predetermined meters ahead of the vehicle on a portion of the display unit 180.
When the vehicle 200 enters the downhill 1420 from the level ground 1210, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward immediately before the vehicle 200 enters the downhill 1420. For example, as shown in the middle part of FIG. 14B, the processor 170 graphically processes an image 1420 b corresponding to the stereo camera 195 which is moved downward and displays the graphically-processed image through the display unit 180.
After the vehicle 200 has entered the downhill 1420, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 returns to the original position thereof. For example, as shown in the right-most part of FIG. 14B, the processor 170 graphically processes an image 1430 c corresponding to the stereo camera 195 which returns to the original position thereof and displays the graphically-processed image through the display unit 180.
In some implementations, the images 1410 a, 1420 b, and 1430 c corresponding to the stereo camera 195 shown in the examples of FIG. 14B may be three-dimensionally displayed or displayed as a video. Consequently, this may enable the driver of the vehicle 200 may intuitively recognize a current state of the stereo camera 195 through the images 1410 a, 1420 b, and 1430 c displayed on the display unit 180.
FIGS. 15A and 15B are diagrams illustrating examples of motions of the stereo camera 195 and the display unit 180 when the vehicle enters a level ground from an uphill road.
Referring to the example in FIG. 15A, when the vehicle 200 travels on an uphill 1510 in a positive direction of the Z axis and in a positive direction of the Y axis, the processor 170 may determine that a road ahead of the vehicle is a level ground 1520. The processor 170 may determine the level ground 1520 based on determination as to whether the road ahead of the vehicle is the uphill or the downhill as described above. In this case, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. For example, the processor 170 may control the stereo camera 195 to be moved downward in response to a tilt δ of the uphill. As a specific example, the processor 170 may control the stereo camera 195 to be moved downward in response to the tilt δ of the uphill. The processor 170 may control the stereo camera 195 to be tilted downward in response to the tilt δ of the uphill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
In some implementations, when the vehicle 200 travels on the uphill 1510 in the positive direction of the Z axis and in the positive direction of the Y axis, the processor 170 may determine that the road ahead of the vehicle is the level ground 1520. The processor 170 may determine the level ground 1520 based on determination as to whether the road ahead of the vehicle is the uphill or the downhill as described above. In a case in which it is determined that the road ahead of the vehicle is the level ground, the processor 170 determines whether the vehicle has reached a predetermined distance 1530 from an inflection point 1515. The inflection point 1515 may be a start point of the level ground. The processor 170 may set the inflection point 1515 based on the depth map and determine whether the vehicle 200 has reached the predetermined distance 1530 from the inflection point 1515 based on information regarding the distance to the inflection point 1515. In some implementations, the predetermined distance 1530 from the inflection point 1515 may be preset as a point at which the stereo camera 195 can be properly moved.
In a case in which the vehicle 200 has reached the predetermined distance 1530 from the inflection point 1515, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward. For example, the processor 170 may control the stereo camera 195 to be moved downward in response to the tilt δ of the uphill. As a specific example, the processor 170 may control the stereo camera 195 to be moved downward in response to the tilt δ of the uphill. Specifically, the processor 170 may control the stereo camera 195 to be tilted downward in response to the tilt δ of the uphill. The processor 170 may control the stereo camera drive unit 196 to control the motion of the stereo camera 195.
After the vehicle 200 has entered the level ground 1520, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved upward. For example, after the vehicle 200 has entered the level ground 1520, the processor 170 may return the stereo camera 195 to the original position thereof. In this case, the processor 170 may return the stereo camera 195 to the original position thereof in response to the traveling speed of the vehicle 200. For example, the processor 170 may return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200. Specifically, the processor 170 may tilt the stereo camera 195 upward to return the stereo camera 195 to the original position thereof in proportion to the traveling speed of the vehicle 200.
Referring to the example of FIG. 15B, when the vehicle 200 travels on the uphill 1510 in the positive direction of the Z axis and in the positive direction of the Y axis, the stereo camera 195 assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200. In this case, as shown in the example of the left-most part of FIG. 15B, the processor 170 graphically processes an image 1510 a corresponding to the stereo camera 195 which assumes an attitude suitable for acquiring images for a view ahead of the vehicle 200 and displays the graphically-processed image through the display unit 180. In addition, the processor 170 may display information 1501 indicating that there is a level ground predetermined meters ahead of the vehicle on a portion of the display unit 180.
When the vehicle 200 enters the level ground 1520 from the uphill 1510, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 is moved downward immediately before the vehicle 200 enters the level ground 1520. In this case, as shown in the middle part of FIG. 15B, the processor 170 graphically processes an image 1520 b corresponding to the stereo camera 195 which is moved downward and displays the graphically-processed image through the display unit 180.
After the vehicle 200 has entered the level ground 1520, the processor 170 may output a control signal to the stereo camera drive unit 196 to control the stereo camera drive unit 196 such that the stereo camera 195 returns to the original position thereof. In this case, as shown in the right-most part of FIG. 15B, the processor 170 graphically processes an image 1530 c corresponding to the stereo camera 195 which returns to the original position thereof and displays the graphically-processed image through the display unit 180.
In some implementations, the images 1510 a, 1520 b, and 1530 c corresponding to the stereo camera 195 shown in FIG. 15B may be three-dimensionally displayed or displayed as a video.
Consequently, this may enable the driver of the vehicle 200 may intuitively recognize a current state of the stereo camera 195 through the images 1510 a, 1520 b, and 1530 c displayed on the display unit 180.
FIGS. 16A and 16B are diagrams illustrating an example of an operation of the driver assistance apparatus in a case in which a road ahead of the vehicle is a curved uphill or a curved downhill road.
Referring to the example of FIG. 16A, the processor 170 may detect lines on the road from the stereo images or the depth maps and create information regarding a curve of the road based on the detected lines.
In a case in which there is an uphill or a downhill of the road ahead of the vehicle in a state in which the uphill or the downhill is curved to the left in a travel direction of the vehicle, the processor 170 detects an outermost one 1620 of the lines ahead of the vehicle. The outermost line 1620 may be a line most distant from a point 1610, which is the center of the curve. The processor 170 may detect a tilt of the outermost line 1620 formed in a positive direction of the Z axis and in a positive direction of the Y axis from the stereo images or the depth maps. The processor 170 may control the stereo camera 195 to be moved upward or downward through control of the stereo camera drive unit 196 in proportion to the tilt of the outermost line 1620.
Referring to the example of FIG. 16B, in a case in which there is an uphill or a downhill of the road ahead of the vehicle in a state in which the uphill or the downhill is curved to the right in a travel direction of the vehicle, the processor 170 detects an outermost one 1640 of the lines ahead of the vehicle. The outermost line 1640 may be a line most distant from a point 1630, which is the center of the curve. The processor 170 may detect a tilt of the outermost line 1640 formed in the positive direction of the Z axis and in the positive direction of the Y axis from the stereo images or the depth maps. The processor 170 may control the stereo camera 195 to be moved upward or downward through control of the stereo camera drive unit 196 in proportion to the tilt of the outermost line 1640.
An operation method of the driver assistance apparatus may be implemented as code that can be written on a program recorded medium and thus read by a computer. The computer-readable medium may be any type of recording device in which data is stored in a computer-readable manner. The computer-readable medium may include, for example, a hard disk drive (HDD), a solid state disk (SSD), a silicon disk drive (SDD), a read only memory (ROM), a random access memory (RAM), a compact disc read only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device. In addition, the computer-readable medium may be implemented in the form of a carrier wave (for example, a transmission over the Internet). The computer may include the processor 170 or the control unit 770.
a stereo camera configured to acquire stereo images of a view ahead of a vehicle;
a stereo camera drive unit configured to drive the stereo camera; and
generate a depth map based on the acquired stereo images; and
determine whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map,
wherein the at least one processor is configured to control the stereo camera drive unit based on a determination of whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment by:
determining that the road segment that is within the view ahead of the vehicle is an uphill road segment; and
based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment:
determining a travelling speed of the vehicle; and
controlling the stereo camera to be tilted upward at a speed that is proportional to the determined traveling speed of the vehicle,
wherein the at least one processor is further configured to, based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment:
determine a tilt of the uphill road segment;
determine that a distance between the vehicle and an inflection point of the uphill road segment is at least a predetermined distance; and
based on a determination that the distance between the vehicle and the inflection point of the uphill road segment is at least the predetermined distance, control the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
2. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
detect a road surface based on the generated depth map; and
determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected road surface.
3. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
detect a lane from the stereo images;
determine a shape of the detected lane; and
determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined shape of the detected lane.
4. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
determine a disappearance point indicated in the depth map; and
determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined disappearance point indicated in the depth map.
5. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
detect a plurality of fixed objects around the road segment indicated in the generated depth map; and
determine whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected plurality of fixed objects around the road segment indicated in the generated depth map.
6. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
determine that the vehicle has entered the uphill road segment; and
based on a determination that the vehicle has entered the uphill road segment, control the stereo camera to return to a state in which the stereo camera is not tilted.
7. The driver assistance apparatus according to claim 6, wherein the at least one processor is configured to:
determine a travelling speed of the vehicle; and
control, in proportion to the determined traveling speed of the vehicle and according to the determined tilt of the uphill road segment, a speed of the stereo camera to return to the state in which the stereo camera is not tilted.
8. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
determine that the road segment that is within the view ahead of the vehicle is a downhill road segment; and
based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment:
determine a tilt of the downhill road segment; and
control the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
9. The driver assistance apparatus according to claim 8, wherein the at least one processor is configured to:
determine that a distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance; and
based on a determination that the distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance, control the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
10. The driver assistance apparatus according to claim 8, wherein:
the at least one processor is configured to determine that the vehicle has entered the downhill road segment, and
the stereo camera drive unit is configured to, based on a determination that the vehicle has entered the downhill road segment, drive the stereo camera to return to a state in which the stereo camera is not tilted.
11. The driver assistance apparatus according to claim 10, wherein the at least one processor is configured to:
control, according to the determined tilt of the downhill road segment or in proportion to the determined traveling speed of the vehicle, the stereo camera to return to the state in which the stereo camera is not tilted.
12. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
control the stereo camera to be tilted downward at a speed that is proportional to the determined traveling speed of the vehicle.
13. The driver assistance apparatus according to claim 1, wherein the at least one processor is further configured to:
detect lines in the acquired stereo images; and
generate, based on the detected lines, information regarding a curve of the road segment that is within the view ahead of the vehicle.
14. The driver assistance apparatus according to claim 1, further comprising a display unit configured to output information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle.
15. The driver assistance apparatus according to claim 1, further comprising:
the at least one processor is further configured to cumulatively store, in the first memory, information regarding a state of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle.
a power source drive unit configured to control an output of an engine of the vehicle, wherein the output of the engine is configured to limit a speed of the vehicle;
a brake drive unit configured to drive a brake of the vehicle;
a stereo camera configured to acquire stereo images for a view ahead of the vehicle;
generate a depth map based on the acquired stereo images;
determine, based on the generated depth map, whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment;
generate road surface information based on the acquired stereo images; and
output a control signal configured to control the brake drive unit based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the generated road surface information; and
a slip determination unit configured to determine a slip degree of the vehicle,
wherein the at least one processor is further configured to output the control signal to control the brake drive unit of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the determined slip degree of the vehicle,
cumulatively store, in the first memory, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle;
apply a Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment;
estimate the slip degree of the vehicle based on an application of the Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment; and
output the control signal to control, based on the estimated slip degree of the vehicle, the power source drive unit or the brake drive unit to drive the brake of the vehicle at a level equivalent to 90% or less than a full brake level.
a navigation system configured to provide a map and information regarding a position of the vehicle on the map, wherein
receive, from the navigation system, information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images; and
estimate, based on the received information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images, a state of the road segment.
18. A driver assistance method comprising:
acquiring, by a stereo camera of a vehicle, stereo images of a view ahead of the vehicle;
generating a depth map based on the acquired stereo images;
determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map; and
controlling a stereo camera drive unit to drive the stereo camera, based on a determination of whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment,
wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
determining that a distance between the vehicle and an inflection point of the uphill road segment is at least a predetermined distance; and
based on a determination that the distance between the vehicle and the inflection point of the uphill road segment is at least the predetermined distance, driving the stereo camera to be tilted upward according to a tilt of the uphill road segment.
19. The driver assistance method according to claim 18, wherein determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map comprises:
detecting a road surface from the generated depth map; and
determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected road surface.
20. The driver assistance method according to claim 18, wherein determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map comprises:
detecting a lane from the stereo images;
determining a shape of the detected lane; and
determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined shape of the detected lane.
21. The driver assistance method according to claim 18, wherein determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map comprises:
determining a disappearance point indicated in the depth map; and
determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the determined disappearance point indicated in the depth map.
22. The driver assistance method according to claim 18, wherein determining whether a road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the generated depth map comprises:
detecting a plurality of fixed objects around the road segment indicated in the depth map; and
determining whether the road segment that is within the view ahead of the vehicle is an uphill road segment or a downhill road segment based on the detected plurality of fixed objects around the road segment indicated in the depth map.
23. The driver assistance method according to claim 18, further comprising:
based on a determination that the road segment that is within the view ahead of the vehicle is an uphill road segment, determining a tilt of the uphill road segment, wherein
controlling the stereo camera drive unit to drive the stereo camera comprises driving the stereo camera to be tilted upward according to the determined tilt of the uphill road segment.
24. The driver assistance method according to claim 23, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
determining that the vehicle has entered the uphill road segment; and
based on a determination that the vehicle has entered the uphill road segment, driving the stereo camera to return to a state in which the stereo camera is not tilted.
25. The driver assistance method according to claim 24, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
driving, in proportion to the determined traveling speed of the vehicle and according to the determined tilt of the uphill road segment, the stereo camera to return to the state in which the stereo camera is not tilted.
26. The driver assistance method according to claim 18, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
driving the stereo camera to be tilted upward at a speed that is proportional to the determined traveling speed of the vehicle.
27. The driver assistance method according to claim 18, further comprising:
determining that the road segment that is within the view ahead of the vehicle is a downhill road segment; and
based on a determination that the road segment that is within the view ahead of the vehicle is a downhill road segment, determining a tilt of the downhill road segment, wherein
controlling the stereo camera drive unit to drive the stereo camera comprises driving the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
28. The driver assistance method according to claim 27, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
determining that a distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance; and
based on a determination that the distance between the vehicle and an inflection point of the downhill road segment is at least a predetermined distance, driving the stereo camera to be tilted downward according to the determined tilt of the downhill road segment.
29. The driver assistance method according to claim 27, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
determining that the vehicle has entered the downhill road segment; and
based on a determination that the vehicle has entered the downhill road segment, driving the stereo camera to return to a state in which the stereo camera is not tilted.
30. The driver assistance method according to claim 29, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
driving, according to the determined tilt of the downhill road segment or in proportion to the determined traveling speed of the vehicle, the stereo camera to return to the state in which the stereo camera is not tilted.
31. The driver assistance method according to claim 18, wherein controlling the stereo camera drive unit to drive the stereo camera comprises:
driving the stereo camera to be tilted downward at a speed that is proportional to the determined traveling speed of the vehicle.
32. The driver assistance method according to claim 18, further comprising:
detecting lines in the acquired stereo images; and
generating, based on the detected lines, information regarding a curve of the road segment that is within the view ahead of the vehicle.
33. The driver assistance method according to claim 18, further comprising outputting, to a display unit, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle.
34. The driver assistance method according to claim 18, further comprising cumulatively storing, in a first memory, information regarding a state of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle.
35. The driver assistance method according to claim 18, further comprising:
controlling a brake drive unit to drive a brake of the vehicle;
generating road surface information based on the acquired stereo images; and
outputting a control signal configured to control the brake drive unit based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the generated road surface information.
36. The driver assistance method according to claim 35, further comprising:
receiving, from a navigation system that is configured to provide a map and information regarding a position of the vehicle on the map, information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images from a navigation system; and
estimating, based on the received information regarding the road segment on which the vehicle is traveling excluding a region of the road segment displayed in the stereo images, a state of the road segment.
37. The driver assistance method according to claim 35, further comprising:
determining a slip degree of the vehicle, wherein
outputting a control signal configured to control the brake drive unit based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the generated road surface information comprises:
outputting the control signal to control the brake drive unit of the vehicle based on the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle and based on the determined slip degree of the vehicle.
38. The driver assistance method according to claim 37, further comprising:
controlling an output of an engine of the vehicle through a power source drive unit of the vehicle, wherein the output of the engine is configured to limit a speed of the vehicle;
cumulatively storing, in a first memory, information regarding the determined uphill road segment or the determined downhill road segment that is within the view ahead of the vehicle;
applying a Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment; and
estimating the slip degree of the vehicle based on an application of the Kalman filter to the cumulatively stored information regarding the determined uphill road segment or the determined downhill road segment, wherein
outputting the control signal to control, based on the estimated slip degree of the vehicle, the power source drive unit or the brake drive unit to drive the brake of the vehicle at a level equivalent to 90% or less than a full brake level.
US14/964,995 2014-12-10 2015-12-10 Driver assistance for a vehicle Active 2036-04-10 US10032083B2 (en)
KR1020140177580A KR101641490B1 (en) 2014-12-10 2014-12-10 Driver assistance apparatus and Vehicle including the same
KR10-2014-0177580 2014-12-10
US20160171315A1 US20160171315A1 (en) 2016-06-16
US10032083B2 true US10032083B2 (en) 2018-07-24
ID=54705342
US14/964,995 Active 2036-04-10 US10032083B2 (en) 2014-12-10 2015-12-10 Driver assistance for a vehicle
US (1) US10032083B2 (en)
EP (1) EP3032458B1 (en)
KR (1) KR101641490B1 (en)
CN (1) CN105691299B (en)
KR20180059224A (en) 2016-11-25 2018-06-04 현대자동차주식회사 Apparatus for controlling stop of vehicle and method thereof
KR20180061686A (en) 2016-11-30 2018-06-08 삼성전자주식회사 Method and apparatus for generating autonomous driving route
US10406917B2 (en) * 2017-08-28 2019-09-10 Ford Global Technologies, Llc Systems and methods for vehicle cruise control smoothness adaptation
KR102039801B1 (en) * 2017-12-18 2019-11-01 전자부품연구원 Road Gradient Estimation Method and System based on Stereo Camera
US20030028291A1 (en) 2001-07-31 2003-02-06 Omron Corporation Apparatus for and method of detecting object on road
JP2003139533A (en) 2001-10-30 2003-05-14 Mitsui Constr Co Ltd Method and apparatus for measuring road infrastructure
KR20070077293A (en) 2006-01-23 2007-07-26 주식회사 현대오토넷 System and method for displaying rear image
JP2010040015A (en) 2008-08-08 2010-02-18 Honda Motor Co Ltd Road shape detector for vehicle
US20100299109A1 (en) 2009-05-22 2010-11-25 Fuji Jukogyo Kabushiki Kaisha Road shape recognition device
JP2014006882A (en) 2012-05-31 2014-01-16 Ricoh Co Ltd Road surface slope recognition device, road surface slope recognition method, and road surface slope recognition program
WO2014030508A1 (en) 2012-08-22 2014-02-27 株式会社メガチップス Traffic lane situation determination device and method for determining traffic lane situation
2014-12-10 KR KR1020140177580A patent/KR101641490B1/en active IP Right Grant
2015-07-20 CN CN201510426976.XA patent/CN105691299B/en active IP Right Grant
2015-11-19 EP EP15003302.5A patent/EP3032458B1/en active Active
2015-12-10 US US14/964,995 patent/US10032083B2/en active Active
European Communication pursuant to Article 94(3) EPC in European Application No. 15003302.5, dated Mar. 31, 2017, 5 pages (with English translation).
Extended European Search Report issued in European Application No. 15003302.5 dated Jul. 12, 2016, 13 pages.
Lee et al., "Estimation of Longitudinal Profile of Road Surface from Stereo Disparity using Dijkstra Algorithm," International Journal of Control, Automation, and Systems, 12(4):895-903, Jul. 2014.
Office Action issued in Korean Application No. 10-2014-0177580, dated Jan. 5, 2016, 7 pages.
Partial European Search Report issued in European Application No. 15003302.5 dated Apr. 21, 2016, 9 pages.
EP3032458A2 (en) 2016-06-15
US20160171315A1 (en) 2016-06-16
KR20160070526A (en) 2016-06-20
CN105691299B (en) 2018-09-25
CN105691299A (en) 2016-06-22
KR101641490B1 (en) 2016-07-21
EP3032458A3 (en) 2016-08-10
EP3032458B1 (en) 2018-10-17
US8818708B2 (en) 2014-08-26 Optimum driving path on full windshield display
KR101551215B1 (en) 2015-09-18 Driver assistance apparatus and Vehicle including the same
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:YI, CHUHO;PARK, WOONKI;PARK, YEONCHOOL;REEL/FRAME:046202/0842