Patent ID: 12258011

DESCRIPTION OF EMBODIMENTS

A vehicle controller and a method for controlling a vehicle executed by the vehicle controller will now be described with reference to the attached drawings. The vehicle controller detects a lane being traveled by a vehicle (hereafter, a “travel lane”), and sets a planned trajectory to be traveled by the vehicle along the detected travel lane in a section from the current position to a predetermined distance away. Additionally, the vehicle controller detects a lane line dividing the travel lane and a lane adjacent thereto (hereafter, an “adjacent lane”) from an image that is generated by a camera mounted on the vehicle and that represents a region around the vehicle. Based on the detected lane line, the vehicle controller further identifies a section of the planned trajectory matching the detected lane line as an effective section, which is a section of the planned trajectory effective for determination of a collision with an obstacle. In the case that an obstacle detected by the sensor mounted on the vehicle is located on the planned trajectory and in the effective section, the vehicle controller controls motion of the vehicle to avoid a collision between the obstacle and the vehicle. In this way, the vehicle controller prevents unnecessary control of motion of the vehicle even if it has erroneously recognized an adjacent lane as the travel lane and set a planned trajectory along the adjacent lane.

FIG.1schematically illustrates the configuration of a vehicle control system equipped with the vehicle controller.FIG.2illustrates the hardware configuration of an electronic control unit, which is an embodiment of the vehicle controller. In this embodiment, the vehicle control system1, which is mounted on a vehicle10and controls the vehicle10, includes a GPS receiver2, a camera3, a storage device4, and an electronic control unit (ECU)5, which is an example of the vehicle controller. The GPS receiver2, the camera3, and the storage device4are connected to the ECU5via an in-vehicle network conforming to a standard, such as a controller area network, so that they can communicate with each other. The vehicle control system1may further include a distance sensor (not illustrated), such as LiDAR or radar, which measures the distances from the vehicle10to objects around the vehicle10; a navigation device (not illustrated) for searching for a planned travel route to a destination; and a wireless communication device (not illustrated) for wireless communication with another device.

The GPS receiver2, which is an example of the position measuring unit, receives GPS signals from GPS satellites at predetermined intervals, and determines the position of the vehicle10, based on the received GPS signals. The GPS receiver2outputs positioning information indicating the result of determination of the position of the vehicle10based on the GPS signals to the ECU5via the in-vehicle network at predetermined intervals. The vehicle10may include a receiver conforming to a satellite positioning system other than the GPS receiver2. In this case, this receiver determines the position of the vehicle10.

The camera3, which is an example of the image capturing unit, includes a two-dimensional detector constructed from an array of optoelectronic transducers, such as CCD or C-MOS, having sensitivity to visible light and a focusing optical system that forms an image of a target region on the two-dimensional detector. The camera3is mounted, for example, in the interior of the vehicle10so as to be oriented, for example, to the front of the vehicle10. The camera3captures a region in front of the vehicle10every predetermined capturing period (e.g., 1/30 to 1/10 seconds), and generates images in which this region is captured. The images obtained by the camera3may be color or grayscale images. The vehicle10may include multiple cameras taking pictures in different orientations or having different focal lengths.

Every time the camera3generates an image, the camera3outputs the generated image to the ECU5via the in-vehicle network.

The storage device4, which is an example of the storage unit, includes, for example, at least one of a hard disk drive, a nonvolatile semiconductor memory, and an optical recording medium and an access device therefor. The storage device4stores a high-precision map, which is an example of the map information. The high-precision map includes, for example, information indicating the presence or absence and the positions of lane lines at locations of a road included in a predetermined region represented in this map. The high-precision map may further include information indicating road markings other than lane lines, such as stop lines, and information indicating signposts.

The storage device4may further include a processor for executing, for example, a process to update the high-precision map and a process related to a request from the ECU5to read out the high-precision map. For example, every time the vehicle10moves a predetermined distance, the storage device4may transmit the current position of the vehicle10and a request to obtain a high-precision map to a map server via the wireless communication device (not illustrated), and receive a high-precision map of a predetermined region around the current position of the vehicle10from the map server via the wireless communication device. When receiving a request from the ECU5to read out the high-precision map, the storage device4cuts out that portion of the high-precision map stored therein which includes the current position of the vehicle10and which represents a region smaller than the predetermined region, and outputs the cut portion to the ECU5via the in-vehicle network.

The ECU5controls travel of the vehicle10to automatically drive the vehicle10.

As illustrated inFIG.2, the ECU5includes a communication interface21, a memory22, and a processor23. The communication interface21, the memory22, and the processor23may be separate circuits or a single integrated circuit.

The communication interface21includes an interface circuit for connecting the ECU5to the in-vehicle network. Every time the communication interface21receives positioning information from the GPS receiver2, the communication interface21passes the positioning information to the processor23. Every time the communication interface21receives an image from the camera3, the communication interface21passes the received image to the processor23. Additionally, the communication interface21passes the high-precision map read from the storage device4to the processor23.

The memory22, which is another example of the storage unit, includes, for example, volatile and nonvolatile semiconductor memories. The memory22stores various types of data used in a vehicle control process executed by the processor23of the ECU5. For example, the memory22stores images of surroundings of the vehicle10; the result of determination of the position of the vehicle; the high-precision map; internal parameters indicating the focal length, angle of view, orientation, and mounted position of the camera3; and a set of parameters for specifying a classifier used for detecting, for example, lane lines. Additionally, the memory22temporarily stores various types of data generated during the vehicle control process.

The processor23includes one or more central processing units (CPUs) and a peripheral circuit thereof. The processor23may further include another operating circuit, such as a logic-arithmetic unit, an arithmetic unit, or a graphics processing unit. The processor23executes the vehicle control process on the vehicle10at predetermined intervals.

FIG.3is a functional block diagram of the processor23, related to the vehicle control process. The processor23includes a lane detection unit31, a trajectory setting unit32, a detection unit33, an effective-section identifying unit34, a determining unit35, and a vehicle control unit36. These units included in the processor23are, for example, functional modules implemented by a computer program executed on the processor23, or may be dedicated operating circuits provided in the processor23.

The lane detection unit31detects a lane being traveled by the vehicle10(hereafter, a “travel lane”). For example, the lane detection unit31refers to the current position of the vehicle10measured by the GPS receiver2and the high-precision map to identify the road being traveled by the vehicle10, and detects a lane in the identified road on which the vehicle10can travel as the travel lane. For example, in the case that the road at the current position of the vehicle10is a two-lane road where vehicles keep to the left, the lane detection unit31detects the left lane with respect to the travel direction of the vehicle10as the travel lane.

Alternatively, the lane detection unit31may compare an image obtained by the camera3with the high-precision map to detect the travel lane. In this case, for example, the lane detection unit31inputs the image into a classifier to detect features on or near the road represented in the image. As such a classifier, the lane detection unit31may uses, for example, a deep neural network (DNN) having a convolutional neural network (CNN) architecture, e.g., Single Shot MultiBox Detector (SSD) or Faster R-CNN. Such a classifier is trained in advance to detect a detection target feature from an image. With an assumption about the position and orientation of the vehicle10, the lane detection unit31projects features detected from the image onto the high-precision map or features on or near the road around the vehicle10in the high-precision map onto the image by referring to internal parameters of the camera3. Then, the lane detection unit31estimates the current position and orientation of the vehicle10to be the position and orientation of the vehicle10for the case that the features detected from the image best match those represented in the high-precision map. Of the individual lanes represented in the high-precision map, the lane detection unit31detects the lane including the estimated current position of the vehicle10as the travel lane.

The lane detection unit31notifies information indicating the detected travel lane and the current position of the vehicle10to the trajectory setting unit32and the vehicle control unit36.

When receiving the information indicating the detected travel lane and the current position of the vehicle10from the lane detection unit31, the trajectory setting unit32sets a planned trajectory along the travel lane from the current position of the vehicle10to a predetermined distance away. For example, the trajectory setting unit32refers to the high-precision map to set a planned trajectory on the center of the travel lane.

The trajectory setting unit32notifies the set planned trajectory to the effective-section identifying unit34, the determining unit35, and the vehicle control unit36.

The detection unit33detects a lane line of the travel lane and an obstacle in an area around the vehicle10from an image obtained by the camera3. For example, the detection unit33inputs an image into a classifier to detect a lane line and an obstacle represented in the image. The obstacle is an object impeding travel of the vehicle10, such as a parked vehicle or a fallen object on the road. As such a classifier, the detection unit33may use a classifier similar to that described in relation to the lane detection unit31, e.g., SSD or Faster R-CNN. Alternatively, as such a classifier, the detection unit33may use a DNN for semantic segmentation that identifies, for each pixel, the type of object represented in the pixel, e.g., a fully convolutional network (FCN) or U-Net. Such a classifier is trained in advance to detect a lane line and an obstacle, which are detection targets, from an image.

In the case that the vehicle10is equipped with a distance sensor, the detection unit33may detect an obstacle, based on a ranging signal obtained by the distance sensor. In this case also, the detection unit33inputs a ranging signal into a classifier that has been trained to detect an obstacle from a ranging signal, thereby detecting an obstacle. In this case, since the direction to the obstacle is identified in the ranging signal by the classifier, the detection unit33can estimate the distance to the detected obstacle by referring to the distance value of the ranging signal in the identified direction.

The detection unit33notifies the effective-section identifying unit34of information indicating the position of the lane line in the image, and the determining unit35of information indicating the position of the obstacle in the image or indicating the distance and direction to the obstacle detected by the distance sensor.

The effective-section identifying unit34identifies a section of the planned trajectory matching the lane line detected by the detection unit33between the travel lane and an adjacent lane as an effective section, based on the detected lane line.

FIGS.4A to4Dillustrate examples of the effective section.

In the example illustrated inFIG.4A, a planned trajectory401is curved relative to a real travel lane400from the middle because, for example, an adjacent lane is erroneously detected as the travel lane; the difference between the curvature of the planned trajectory401and the curvature of lane line411between the travel lane and the adjacent lane increases with the distance from the vehicle10. In this case, the effective-section identifying unit34identifies a section of the planned trajectory401from the current position of the vehicle10to a position P where the difference between the curvatures of the planned trajectory401and the lane line411exceeds a predetermined threshold (a first section) as an effective section421. Such a position P will be referred to as a “bending position” below for convenience of description. The planned trajectory401, which is set along the detected travel lane by referring to the high-precision map, can be set as far as a position beyond the range of the camera3or another sensor. For this reason, in the case that the section from the current position of the vehicle10to a bending position P is an effective section, the effective-section identifying unit34can set a relatively long effective section, which enables the vehicle control unit36to start controlling motion of the vehicle10to avoid a collision between the vehicle10and an obstacle at a relatively early stage.

In the example illustrated inFIG.4B, a planned trajectory402crosses the left lane line412of the travel lane400in the middle because, for example, an adjacent lane is erroneously detected as the travel lane. In this case, the effective-section identifying unit34identifies a section of the planned trajectory402from the current position of the vehicle10to a position Q where the planned trajectory402and the lane line412cross (a second section) as an effective section422. Such a position Q will be referred to as a “crossing position” below for convenience of description. To reduce the influence of an error in recognition of the position of the lane line412on control of the vehicle10, the effective-section identifying unit34may detect the position where the planned trajectory402crosses a virtual line (indicated by a broken line412a) located a predetermined offset distance inwardly of the travel lane relative to the lane line412, as the crossing position Q. The error in the position of a lane line is likely to increase with the curvature of the lane line or with the distance from the vehicle10. Thus the effective-section identifying unit34may increase the predetermined offset distance with the curvature of the lane line crossing the planned trajectory or with the distance from the vehicle10. Alternatively, the effective-section identifying unit34may refer to the high-precision map and the current position of the vehicle10indicated by the latest positioning information to identify the standard of the road being traveled by the vehicle10, and set the predetermined offset distance depending on the standard. Since a higher-standard road generally has fewer curves with large curvatures, the effective-section identifying unit34may shorten the predetermined offset distance as the standard of the road being traveled by the vehicle10is higher. In this case, the effective-section identifying unit34can appropriately set the offset distance irrespective of the result of detection of a lane line from an image.

In the example illustrated inFIG.4C, a planned trajectory403is set longer than a lane line413detected by the detection unit33between the travel lane and an adjacent lane, and has neither bending position P nor crossing position Q. When neither bending position P nor crossing position Q is detected, the effective-section identifying unit34identities a section of the planned trajectory403from the current position of the vehicle10to the farthest point R of the detected lane line413(a third section) as an effective section423. In other words, the effective section423has the same length as the section from the current position of the vehicle10to the farthest point R of the detected lane line413. In the case that the detected farthest points of the left and right lane lines of the travel lane differ, the effective-section identifying unit34may identify an effective section, based on the lane line whose detected farthest point is farther from the current position of the vehicle10.

In the example illustrated inFIG.4D, no lane line is detected between the travel lane and an adjacent lane. Thus, no effective section is identified for a planned trajectory404. In other words, the length of the effective section is 0.

The positions of pixels in an image correspond one-to-one to the directions from the camera3to objects represented in the respective pixels. Thus, for each pixel representing a lane line in an image, the effective-section identifying unit34can estimate the actual real-space position of the lane line represented in the pixel by referring to the position of the pixel as well as internal parameters such as the focal length and the mounted position of the camera3. The real space herein is represented by a coordinate system relative to a predetermined point of the vehicle10(e.g., the center of the front end of the vehicle10). In this way, the effective-section identifying unit34can estimate the real-space position of a lane line detected from an image, and thus determine the position of the furthest end of the detected lane line or the position where the planned trajectory and the lane line cross, based on the estimated position. To calculate the difference between the curvatures of the planned trajectory and the detected lane line, the effective-section identifying unit34divides, for example, the planned trajectory and the detected lane line into subsections each having a predetermined length. Then, for each divided subsection, the effective-section identifying unit34determines the curvature of the planned trajectory in the subsection from the positions of the planned trajectory at both ends and the midpoint of the subsection. Similarly, for each divided subsection, the effective-section identifying unit34determines the curvature of the lane line in the subsection from the positions of the lane line at both ends and the midpoint of the subsection.

When both the bending position P illustrated inFIG.4Aand the crossing position Q illustrated inFIG.4Bare determined, the effective-section identifying unit34may set a section between the closer to the vehicle10of these positions and the current position of the vehicle10to be an effective section. For example, referring back toFIG.4A, the planned trajectory401crosses the lane line411farther from the vehicle10than the bending position P. Hence the effective section421is identified as the section from the current position of the vehicle10to the bending position P, as described above. Referring back toFIG.4B, the difference between the curvatures of the planned trajectory402and the lane line412exceeds the threshold farther from the vehicle10than the crossing position Q. Hence the effective section422is identified as the section from the current position of the vehicle10to the crossing position Q, as described above.

The effective-section identifying unit34notifies information indicating the effective section of the planned trajectory to the determining unit35.

The determining unit35determines whether the obstacle detected by the detection unit33is located in the effective section of the planned trajectory and whether the obstacle lies on the planned trajectory.

In the case that the obstacle is detected from an image generated by the camera3, the determining unit35estimates, for each pixel representing the obstacle in the image, the actual real-space position of the obstacle represented in the pixel by referring to the position of the pixel as well as internal parameters such as the focal length and the mounted position of the camera3. The real space herein is represented by a coordinate system relative to a predetermined point of the vehicle10. In the case that the obstacle is detected from a ranging signal generated by the distance sensor, the direction and distance to the obstacle are determined from the position in the vehicle10where the distance sensor is mounted, and thus the actual position of the obstacle is estimated on the basis of this direction and distance. The determining unit35compares the estimated real-space position of the obstacle with the effective section to determine whether the obstacle is located in the elective section.

When it is determined that the obstacle is located in the effective section of the planned trajectory, the determining unit35determines whether the obstacle lies on the planned trajectory, and if so, determines that the obstacle is located on the planned trajectory. When no obstacle is detected, the determining unit35determines that no obstacle is located on the planned trajectory.

The determining unit35may first determine whether the obstacle lies on the planned trajectory, and if so, determine whether the obstacle is located in the effective section of the planned trajectory.

Even if the obstacle is not located in the effective section of the planned trajectory, the determining unit35may determine whether the obstacle lies on the planned trajectory in the case that the distance from the vehicle10to the obstacle is not greater than a predetermined distance threshold. The predetermined distance threshold may be, for example, the shortest distance such that the vehicle10can avoid a collision with the obstacle by a motion for collision avoidance, or the shortest distance multiplied by a predetermined safety factor, e.g., the distance corresponding to a time to collision (TTC) of 1.4 seconds. Thus the determining unit35may increase the predetermined distance threshold with the speed of the vehicle10measured by a vehicle speed sensor (not illustrated) of the vehicle10.

The determining unit35notifies the vehicle control unit36of the results of determination whether the obstacle is located in the effective section of the planned trajectory and whether the obstacle lies on the planned trajectory. The determining unit35may further notify the vehicle control unit36of the result of determination whether the distance from the vehicle10to the obstacle is not greater than the predetermined distance threshold.

The vehicle control unit36executes automated driving control of the vehicle10so that it will travel along the planned trajectory. For example, the vehicle control unit36determines the steering angle for the vehicle10to travel along the planned trajectory by referring to the current position of the vehicle10and the planned trajectory, and outputs a control signal depending on the steering angle to an actuator (not illustrated) that controls the steering wheel of the vehicle10. The vehicle control unit36also determines target acceleration of the vehicle10according to a target speed of the vehicle10and its current speed measured by the vehicle speed sensor (not illustrated), and sets the degree of accelerator opening or the amount of braking so that the acceleration of the vehicle10will be equal to the target acceleration. The vehicle control unit36then determines the amount of fuel injection according to the set degree of accelerator opening, and outputs a control signal depending on the amount of fuel injection to a fuel injector of an engine of the vehicle10. Alternatively, the vehicle control unit36outputs a control signal depending on the set amount of braking to the brake of the vehicle10. The vehicle control unit36refers the high-precision map to identify the legally permitted speed of the road being traveled by the vehicle10, and sets the target speed of the vehicle10according to the legally permitted speed. Alternatively, the vehicle control unit36may set the target speed of the vehicle10so as to keep the distance to a vehicle traveling ahead of the vehicle10constant.

When receiving from the determining unit35the result of determination that an obstacle is located in the effective section of the planned trajectory and lies on the planned trajectory, the vehicle control unit36controls motion of the vehicle10to avoid a collision between the vehicle10and the obstacle. For example, the vehicle control unit36sets the amount of braking so that the vehicle10will stop before colliding with the obstacle, and outputs a control signal depending on the set amount of braking to the brake of the vehicle10to decelerate the vehicle10. Alternatively, the vehicle control unit36may determine the steering angle so that the vehicle10will change its direction before colliding with the obstacle, and output a control signal depending on the steering angle to the actuator (not illustrated) that controls the steering wheel of the vehicle10.

As described above, the presence of an obstacle in the effective section of the planned trajectory is set as one of conditions for controlling motion of the vehicle10to avoid a collision between the vehicle10and the obstacle, which enables the vehicle control unit36to prevent controlling motion of the vehicle10for collision avoidance although the vehicle10is unlikely to collide with the obstacle. For example, referring back toFIG.4A, there is a detected obstacle430on the planned trajectory401but outside the effective section421. Hence, control of motion of the vehicle10for collision avoidance will not be executed. Similarly, referring back toFIG.48, there is a detected obstacle430on the planned trajectory402but outside the effective section422. Hence, control of motion of the vehicle10for collision avoidance will not be executed. In the case illustrated inFIG.4A or4B, when the travel lane is correctly recognized and thereby the planned trajectory is reset along the real travel lane during the approach of the vehicle10to the obstacle, the obstacle will fall outside the planned trajectory. Hence, control of motion of the vehicle10for collision avoidance will not be executed in the end. In the example illustrated inFIG.4Calso, there is an obstacle430outside the effective section423, and thus control of motion of the vehicle10for collision avoidance will not be executed. However, in this case, the planned trajectory403is set along the real travel lane, and the obstacle430is located on the planned trajectory403. For this reason, after the obstacle430becomes included in the effective section423by the approach of the vehicle10to the obstacle430, control of motion of the vehicle10for collision avoidance will be executed.

Additionally, also when receiving from the determining unit35the results of determination that the distance from the vehicle10to the obstacle is not greater than the predetermined distance threshold and that the obstacle lies on the planned trajectory, the vehicle control unit36controls motion of the vehicle10to avoid a collision between the vehicle10and the obstacle.

FIG.5is an operation flowchart of the vehicle control process executed by the processor23. The processor23may execute the vehicle control process in accordance with the following operation flowchart at predetermined intervals.

The lane detection unit31of the processor23detects the lane being traveled by the vehicle, i.e., the travel lane (step S101). The trajectory setting unit32of the processor23sets a planned trajectory along the identified travel lane from the current position of the vehicle10to a predetermined distance away (step S102).

The detection unit33of the processor23detects a lane line between the travel lane and an adjacent lane and an obstacle in an area around the vehicle10from an image obtained by the camera3or a ranging signal by the distance sensor (step S103). Additionally, the effective-section identifying unit34of the processor23identifies a section of the planned trajectory matching the detected lane line as an effective section, based on the lane line between the travel lane and the adjacent lane (step S104).

The determining unit35of the processor23determines whether the obstacle is located in the effective section of the planned trajectory (step S105). When the obstacle is not in the effective section (No in step S105), the determining unit35determines whether the distance from the vehicle10to the obstacle is not greater than a predetermined distance threshold (step S106).

When the obstacle is located in the effective section of the planned trajectory (Yes in step S105) or when the distance from the vehicle10to the obstacle is not greater than the predetermined distance threshold (Yes in step S106), the vehicle control unit36determines whether the obstacle is located on the planned trajectory (step S107).

When the obstacle is located on the planned trajectory (Yes in step S107), the vehicle control unit36of the processor23controls motion of the vehicle10to avoid a collision between the vehicle10and the obstacle (step S108).

When the distance from the vehicle10to the obstacle is greater than the predetermined distance threshold (No in step S106) or when the obstacle is not located on the planned trajectory (No in step S107), the vehicle control unit36controls motion of the vehicle10so that it will travel along the planned trajectory (step S109). After step S108or S109, the processor23terminates the vehicle control process.

As has been described above, the vehicle controller detects a travel lane and sets a planned trajectory along the detected travel lane. Additionally, the vehicle controller detects a lane line dividing the travel lane and an adjacent lane from an image that is generated by the camera mounted on the vehicle and that represents a region around the vehicle. Based on the detected lane line, the vehicle controller further identifies a section of the planned trajectory matching the detected lane line as an effective section. In the case that an obstacle detected by the sensor mounted on the vehicle is located on the planned trajectory and in the effective section, the vehicle controller controls motion of the vehicle to avoid a collision between the obstacle and the vehicle. In this way, the vehicle controller can prevent unnecessary control of motion of the vehicle, even if it has erroneously recognized an adjacent lane as the travel lane and set a planned trajectory along the adjacent lane and thus an obstacle with low risk of collision is located on the erroneously set planned trajectory.

According to a modified example, in the case that the obstacle is not located in the effective section of the planned trajectory and that the distance from the vehicle10to the obstacle is not greater than the predetermined distance threshold, the determining unit35may predict the trajectory of the vehicle10from the current time until a predetermined time ahead by executing a predetermined prediction process on the trajectory of the vehicle10calculated from information indicating motion of the vehicle10obtained in a preceding predetermined period, such as the yaw rate, acceleration, and speed of the vehicle10. In this case, the determining unit35obtains the information indicating motion of the vehicle10from sensors for measuring motion of the vehicle10, such as a yaw rate sensor (not illustrated), an acceleration sensor (not illustrated), and a vehicle speed sensor (not illustrated). Then, the determining unit35may determine whether the obstacle lies on the predicted trajectory of the vehicle10. In this case, when the obstacle lies on the predicted trajectory of the vehicle10, the vehicle control unit36may control motion of the vehicle10to avoid a collision between the vehicle10and the obstacle.

A computer program for achieving the functions of the processor23of the ECU5according to the embodiment or modified example may be provided in a form recorded on a computer-readable and portable medium, such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

As described above, those skilled in the art may make various modifications according to embodiments within the scope of the present invention.