Patent ID: 12198445

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of a drive assist device, a drive assist method, and a program according to the invention will be described with reference to the drawings. A drive assist device according to an embodiment is mounted in an automatic driving vehicle. The drive assist device according to the embodiment is a device that assists vehicle drive control. The drive control mentioned here is, for example, control for causing a vehicle to move to an appropriate lane in accordance with a desired travel direction and control for causing the vehicle to appropriately start and stop in accordance with the lane along which the vehicle is traveling and display states of traffic signals. The drive assist device according to the embodiment assists vehicle drive control by managing information related to lanes and road markings, for example, and providing the information to other devices. For example, the drive assist device according to the embodiment appropriately provide the information related to the lanes, the road markings, and the like to a map positioning unit (MPU) for determining a recommended lane in accordance with a travel route considered by a navigation system. In this manner, the drive assist device according to the embodiment can secure a lane selection by the MPU.

However, the vehicle in which the drive assist device according to the embodiment of the invention is mounted is not limited to an automatic driving vehicle and may be a vehicle that requires manual driving operations by a driver. Note that in the vehicle that requires manual driving operations by a driver, the drive assist device according to the invention appropriately provides the information related to the lanes and the road markings to a human machine interface (HMI) that provides information to a passenger, for example. In this manner, the drive assist device according to the embodiment of the invention can cause the HMI to provide a notification to the passenger in a case where a direction in which the vehicle can travel in the lane where the vehicle is traveling is different from a direction indicated by an indicator, for example.

Overall Configuration

FIG.1is a configuration diagram of a vehicle system1using a drive assist device according to a first embodiment. A vehicle in which the vehicle system1is mounted (hereinafter, referred to as a “vehicle M”) is a two-wheeled, three-wheeled, or four-wheeled vehicle, for example, and a drive source thereof is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using power generated by a power generator coupled to the internal combustion engine or power discharged by a secondary battery or a fuel battery.

The vehicle system1includes, for example, a camera10, a radar device12, a finder14, an object recognition device16, a communication device20, an HMI30, a vehicle sensor40, a navigation device50, an MPU60, a drive operator80, an automatic driving control device100, a traveling drive force output device200, a brake device210, and a steering device220. Note that the automatic driving control device100is an example of the “drive assist device”. These devices and machines are connected to each other via a multiplex communication line such as a controller area network (CAN), a serial communication line, a wireless communication network, or the like. Note that the configuration illustrated inFIG.1is just an example, and some of the configurations may be omitted, or different configurations may further be added.

The camera10is a digital camera using a solid imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), for example. The camera10is attached to an arbitrary location of the vehicle M. In a case in which a front side of the vehicle M is imaged, for example, the camera10is attached to an upper portion of a front wind shield, a back surface of a rear-view mirror, or the like. In a case in which a back side of the vehicle M is imaged, the camera10is attached to an upper portion of a rear wind shield or the like. In a case in which a right side or a left side of the vehicle M is imaged, the camera10is attached to a right side surface, a left side surface, or the like of a vehicle body or a door mirror. The camera10periodically and repeatedly images the surroundings of the vehicle M, for example. The camera10may be a stereo camera.

Also, the camera10can image a pavement when the vehicle M is traveling and stopping. The camera10images the pavement in front of the vehicle M, for example. The camera10captures an image such that a travel lane along which the vehicle M is traveling (hereinafter, referred to as a “driving lane”) and other travel lanes that are present in parallel with the driving lane (hereinafter, referred to as “other lanes”) are included in the captured image, for example. Other lanes descried here include at least other lanes that are adjacent to and in parallel with the driving lane (hereinafter, referred to as “adjacent lanes”).

Also, the camera10can image a traffic signal (a traffic signal for vehicles) when the vehicle M is traveling and stopping. The camera10images the traffic sign by imaging a slightly upper side of a front side of the vehicle M, for example. The camera10captures the image such that a three-light signal and an arrow signal of the traffic signal are included in the captured image, for example.

The radar device12emits radio waves such as millimeter waves to the surroundings of the vehicle M, detects radio waves (reflected waves) reflected by an object and detects at least the position (the distance and the azimuth) of the object. The radar device12is attached to an arbitrary location of the vehicle M. The radar device12may detect the position and the speed of the object by a frequency modulated continuous wave (FM-CW) scheme.

The finder14is a light detection and ranging (LIDAR). The finder14emits light to the surroundings of the vehicle M and measures scattered light. The finder14detects the distance to a target on the basis of a time from light emission to light reception. The emitted light is, for example, pulse-shaped laser light. The finder14is attached to an arbitrary location of the vehicle M.

The object recognition device16performs sensor fusion processing on results of detection performed by some or all of the camera10, the radar device12, and the finder14and recognizes the position, the type, the speed, and the like of an object. The object recognition device16outputs a result of the recognition to the automatic driving control device100. The object recognition device16may output results of the detection of the camera10, the radar device12, and the finder14as they are to the automatic driving control device100. The object recognition device16may be omitted from the vehicle system1.

Also, the object recognition device16recognizes lane lines and road marking, for example, from an image captured by the camera10. The object recognition device16recognizes, for example, a three-light signal and an arrow signal from the image captured by the camera10.

The communication device20communicates with other vehicles that are present in the surroundings of the vehicle M using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), or a dedicated short range communication (DSRC), for example, or communicates with various server devices via a wireless base station.

The HMI30presents various kinds of information to a passenger (a driver, for example) of the vehicle M and receives an operation input by the passenger. The HMI30includes, for example, various display devices, a speaker, a buzzer, a touch panel, a switch, and a key.

The vehicle sensor40includes a vehicle speed sensor that detects a speed of the vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects an angular speed around a vertical axis, an azimuth sensor that detects an orientation or a travel direction of the vehicle M, and the like. Various kinds of information detected by the vehicle sensor40are output to a navigation device50or an automatic driving control device100, for example.

The navigation device50includes, for example, a global navigation satellite system (GNSS) receiver51, a navigation HMI52, and a route determination unit53. The navigation device50holds first map information54in a storage device such as a hard disk drive (HDD) or a flash memory. The GNSS receiver51specifies the position of the vehicle M on the basis of a signal received from a GNSS satellite. The position of the vehicle M may be specified or complemented by an inertial navigation system (INS) using outputs of the vehicle sensor40. The navigation HMI52includes a display device, a speaker, a touch panel, a key, and the like. The navigation HMI52may commonly use a part or entirety of the aforementioned HMI30. The route determination unit53determines a route (hereinafter, referred to as a route on the map) from the position of the vehicle M specified by the GNSS receiver51(or an input arbitrary position) to a destination input by the passenger using the navigation HMI52with reference to the first map information54. The first map information54is information in which road shapes are expressed by links indicating roads and nodes connected by the links, for example. The first map information54may include curvatures of roads, point-of-interest (POI) information, and the like. The route on the map is output to the MPU60. The navigation device50may perform route guidance using the navigation HMI52on the basis of the route on the map. The navigation device50may be realized by functions of a terminal device such as a smartphone, or a tablet owned by the passenger, for example. The navigation device50may transmit a current position and a destination to a navigation server via the communication device20and acquire a route equivalent to the route on the map from the navigation server.

The MPU60includes, for example, a recommended lane determination unit61and holds second map information62in a storage device such as an HDD or a flash memory. The recommended lane determination unit61splits the route on the map provided from the navigation device50into a plurality of blocks (split it into every 100 [m] in the vehicle travel direction, for example) and determines a recommended lane for each block with reference to the second map information62. The recommended lane determination unit61determines what number of lane from the left the vehicle is to travel. In a case in which a junction location is present on the route in the map, the recommended lane determination unit61determines the recommended lane such that the vehicle M can travel along a reasonable route for traveling after the junction.

The second map information62is more precise map information than the first map information54. The second map information62includes, for example, information regarding centers of lanes and information regarding boundaries of the lanes. Also, the second map information62may include road information, traffic restriction information, address information (addresses and postal codes), facility information, phone number information, and the like. The second map information62may be updated as needed by the communication device20communicating with other devices.

The drive operator80includes, for example, an acceleration pedal, a brake pedal, a shift lever, a steering wheel, a deformed steering, a joystick, and other operators. A sensor that detects the amount of operation or whether or not an operation has been performed is attached to the drive operator80, and a result of detection thereof is output to the automatic driving control device100or some or all of the traveling drive force output device200, the brake device210, and the steering device220.

The automatic driving control device100(drive assist device) includes, for example, a first control unit120, a second control unit160, a storage unit170, and a notification control unit180. Each of the first control unit120, the second control unit160, and the notification control unit180is realized by a hardware processor such as a central processing unit (CPU), for example, executing a program (software). Also, some or all of these components may be realized by hardware (a circuit unit; including a circuitry) such as large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a graphics processing unit (GPU) or the like or may be realized by cooperation of software and hardware. The program may be stored in a storage device (a storage device including a non-transitory storage medium) such as an HDD or a flash memory in the automatic driving control device100in advance, or may be stored in a detachable storage medium such as a DVD or a CD-ROM and installed in the HDD or the flash memory in the automatic driving control device100by the storage medium (non-transitory storage medium) being attached to a drive device.

FIG.2is a functional configuration diagram of the first control unit120and the second control unit160. The first control unit120includes, for example, a recognition unit130and an action plan generation unit140. The first control unit120realizes a function based on artificial intelligence (AI) and a function based on a model provided in advance in parallel, for example. For example, a function of “recognizing an intersection” may be realized by executing recognition of the intersection based on deep learning or the like and recognition based on conditions provided in advance (there are a pattern matching available signal, a road marking and the like) being executed in parallel and scoring and comprehensively evaluating both the recognition results. In this manner, reliability of the automatic driving is secured.

The recognition unit130recognizes an environment in the surroundings of the vehicle M. For example, the recognition unit130recognizes states such as positions, speed, acceleration, travel directions, and the like of objects (surrounding vehicles and targets, for example) in the surroundings of the vehicle M on the basis of information input from the camera10, the radar device12, and the finder14via the object recognition device16. The positions of the objects are recognized as positions in an absolute coordinate system including a representative point (a center of gravity, a drive axis center, or the like) of the vehicle M as an origin, for example, and are used for control. The positions of the objects may be represented by representative points, such as centers of gravity, centers, or corners of the objects or may be represented as expressed regions. In a case in which the objects are vehicles, the “states” of the objects may include acceleration or jerks of the objects or “action states” (for example, whether the vehicles are changing lanes or trying to change lanes).

Also, the recognition unit130recognizes the lane on which the vehicle M travels and other lanes, for example. For example, the recognition unit130recognizes a travel lane by comparing patterns of lane lines (alignment of solid lines and broken lines, for example) obtained from the second map information62with patterns of lane lines in the surroundings of the vehicle M recognized from the image captured by the camera10. Note that the recognition unit130may recognize the travel lane by recognizing not only the lane lines but also travel path boundaries (road boundaries) including lane lines, road shoulders, curbstones, median strips, and guard rails. In this recognition, the position of the vehicle M acquired from the navigation device50and a result of processing performed by the INS may be taken into consideration. Also, the recognition unit130recognizes a temporary stop line (hereinafter, referred to as a stop line), a traffic signal, an obstacle, a tollgate, and other road matters.

The recognition unit130recognizes the position and the posture of the vehicle M relative to the travel lane when the recognition unit130recognizes the travel lane. The recognition unit130may recognize separation of a reference point of the vehicle M from the center of the lane and an angle of the travel direction of the vehicle M relative to a line obtained by connecting centers of the lane as the relative position and the posture of the vehicle M relative to the travel lane, for example. Instead, the recognition unit130may recognize the position of the reference point of the vehicle M relative to any one of side end portions (a lane line or a road boundary) of the travel lane as the relative position of the vehicle M relative to the travel lane.

The recognition unit130recognizes information regarding the positions of the surrounding vehicles on the basis of the vehicles in the surroundings of the vehicle M recognized from the image captured by the camera10, the image captured by the camera10, information regarding a traffic jam in the surroundings of the vehicle M acquired by the navigation device50, and position information obtained from the second map information62.

Note that the recognition unit130may acquire, via the communication device20, various kinds of information received from vehicles and the like traveling in the surroundings of the vehicle M through inter-vehicle communication and recognize the surroundings of the vehicle M on the basis of the information. Also, the recognition unit130includes, for example, a lane recognition unit131, a marking recognition unit132, a marking information management unit133(an example of the “information management unit”), a lane change detection unit134(an example of the “detection unit”), a reset determination unit135(an example of the “determination unit”), a signal state recognition unit136, and a lane interpolation unit137(an example of the “interpolation unit”). Details of the function of the recognition unit130including these components will be described later.

The action plan generation unit140generates a target track along which the vehicle M will automatically (without operations of the driver) travel in the future such that the vehicle M can travel along the recommended lane determined by the recommended lane determination unit61in principle and to further address situations in the surroundings of the vehicle M. The target track includes, for example, a speed element. Fr example, the target track is expresses as points (track points), which the vehicle M is to reach, aligned in order. The track points are points that the vehicle M is to reach at every predetermined travel distance (several [m], for example) in terms of a distance along the road, and separately, a target speed and target acceleration for each predetermined sampling time (zero point every several fractions of a [sec], for example) are generated as parts of the target track. Each track point may be a position that the vehicle M is to reach at a sampling clock time for each predetermined sampling time. In this case, the information regarding the target speed and the target acceleration is expressed as intervals of the track points.

The action plan generation unit140may set automatic driving events when the target track is generated. The automatic driving events include, for example, a constant speed traveling event, a low speed following travel event, a lane change event, a junction event, a converging event, and a takeover event. The action plan generation unit140generates a target track in accordance with the activated events.

The second control unit160controls the traveling drive force output device200, the brake device210, and the steering device220such that the vehicle M passes through the target track generated by the action plan generation unit140at the clock time as scheduled.

The second control unit160includes, for example, an acquisition unit162, a speed control unit164, and a steering control unit166. The acquisition unit162acquires information regarding the target track (track points) generated by the action plan generation unit140and causes a memory (not illustrated) to store the information. The speed control unit164controls the traveling drive force output device200or the brake device210on the basis of a speed element that accompanies the target track stored in the memory. The steering control unit166controls the steering device220in accordance with how the target track stored in the memory is curved. Processing of the speed control unit164and the steering control unit166is realized by, for example, a combination of feed forward control and feedback control, for example. In one example, the steering control unit166executes the feed forward control in accordance with a curvature of the road in front of the vehicle M and the feedback control based on separation from the target track.

The notification control unit180notifies the passenger of predetermined information using the HMI30. Note that the notification control unit180and the HMI30are an example of the “notification unit”. The predetermined information is, for example, information recognized by the recognition unit130, a state of the automatic driving (drive assist) executed by the automatic driving control device100, and other information regarding vehicle control. Also, the predetermined information may include information acquired by the navigation device50and information such as TV programs and content (movies, for example) stored in a storage medium such as a DVD. Also, the notification control unit180outputs the information received by the HMI30to the communication device20, the navigation device50, and the first control unit120, for example.

The traveling drive force output device200outputs a travel drive force (torque) for the vehicle to travel to a driving wheel. The traveling drive force output device200includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and the like and an electronic control unit for controlling these components. The ECU may control the aforementioned configurations in accordance with information input from the second control unit160or information input from the drive operator80.

The brake device210includes, for example, a brake caliper, a cylinder that transmits a hydraulic pressure to the brake caliper, an electric motor that generates a hydraulic pressure in a cylinder, and a brake ECU. The brake ECU controls the electric motor in accordance with information input from the second control unit160or information input from the drive operator80and causes a brake torque in accordance with a braking operation to be output to each wheel. The brake device210may include, as backup, a mechanism for transmitting a hydraulic pressure generated by an operation of the brake pedal included in the drive operator80to the cylinder via a master cylinder. Note that the brake device210is not limited to the configuration described above and may be an electronic control-type hydraulic brake device that controls an actuator in accordance with information input from the second control unit160and transmits a hydraulic pressure of the master cylinder to the cylinder.

The steering device220includes, for example, a steering ECU and an electric motor.

The electric motor causes a force to act on a rack-and-pinion mechanism to change the direction of a steered wheel, for example. The steering ECU drives the electric motor and causes the direction of the steered wheel to be changed in accordance with information input from the second control unit160or information input from the drive operator80.

Hereinafter, functions of the recognition unit130according to the embodiment and details of drive assist control based on a result of recognition performed by the recognition unit130will be specifically described. In the following description, a drive assist control pattern for each situation will be described.

First Drive Assist Control Pattern

FIGS.3and4are diagrams for explaining drive control performed by the recognition unit130.FIG.3illustrates a road RD, a lane center line CL, two lane boundary lines BL, a lane outer line OL, a road marking RMa displayed at a point A, and a road marking RMc displayed at a point C. Also,FIG.3illustrates the vehicle M that is about to pass through each of the point A, the point B, and the point C. In other words,FIG.3illustrates a state in which the vehicle M is traveling along the road RD while passing through the point A, the point B, and the point C in this order.

The lane recognition unit131of the automatic driving control device100mounted in the vehicle M recognizes a driving lane, an adjacent lane that is adjacent to the driving lane on the left side (hereinafter, referred to as a “left-hand lane”), and an adjacent lane that is adjacent to the driving lane on the right side (hereinafter, referred to as a “right-hand lane”) on the basis of information output from the object recognition device16. The information output from the object recognition device16mentioned here is information indicating the lane center line CL, the lane boundary lines BL, and the lane outer line OL recognized by the object recognition device16from an image of a pavement captured by the camera10from the vehicle M.

The marking recognition unit132of the automatic driving control device100mounted in the vehicle M recognizes a road marking displayed in each of the driving lane and the adjacent lanes on the basis of information output from the object recognition device16. The marking recognition unit132specifies the direction in which traveling is permitted (hereinafter, referred to as a “permitted travel direction”) of each of the driving lane and the adjacent lanes on the basis of the recognized road markings displayed in the driving lane and the adjacent lanes. The information output from the object recognition device16mentioned here is information indicating the road marking RMa and the road marking RMc recognized by the object recognition device16from the image of the pavement captured by the camera10from the vehicle M. As illustrated inFIG.3, the road marking RMa and the road marking RMc include a road marking in the driving lane and a road marking in the adjacent lanes.

When the vehicle M is about to pass through the point A, the marking recognition unit132acquires information indicating the road marking RMa and recognizes the road marking in the driving lane and the road marking in the adjacent lane (left-hand lane).

The marking recognition unit132specifies that the permitted travel direction of the driving lane is a straight-ahead direction and a right turn direction while the permitted travel direction of the left-hand lane is the straight-ahead direction and a left turn direction on the basis of a result of the recognition. The marking recognition unit132outputs information indicating the result of the specification to the marking information management unit133. The marking information management unit133acquires information output from the marking recognition unit132. The marking information management unit133causes the storage unit170to store the acquired information as marking information171.

FIG.4illustrates, in the form of a table, marking information171stored in the storage unit170at a timing at which the vehicle M passes through each point. As illustrated inFIG.4, according to the marking information171at the point A, the permitted travel directions of the left-hand lane are the left turn direction and the straight-ahead direction while the permitted travel directions of the driving lane are the straight-ahead direction and the right turn direction. Note that since there is no right-hand lane at the point A as illustrated inFIG.3and no road marking on the right-hand lane are recognized, the permitted travel direction of the right-hand lane has not yet been detected in the marking information171at the point A as illustrated inFIG.4. Note that in a case in which the permitted travel direction has not yet been detected, it is determined that the vehicle can travel in all directions.

The point B is a point at which a right-hand lane has been additionally added next to the driving lane on the right side. When the vehicle M is about to pass through the point B, the lane recognition unit131recognizes the right-hand lane. Since the road marking on the right-hand lane has not yet been displayed at the point B as illustrated inFIG.3, the marking information management unit133does not update the marking information171and leaves the right-hand lane as a lane where the vehicle can travel in all directions as illustrated inFIG.4.

Next, when the vehicle M is about to pass through the point C, the marking recognition unit132acquires information indicating road marking RMc and recognizes the road marking on the driving lane and the road markings on the adjacent lanes (the left-hand lane and the right-hand lane). The marking recognition unit132specifies that the permitted travel direction of the driving lane is only the straight-ahead direction, the permitted travel directions of the left-hand lane are the straight-ahead direction and the left turn direction, and the permitted travel directions of the right-hand lane are the straight-ahead direction and the right turn direction on the basis of a result of the recognition. The marking recognition unit132outputs information indicating results of the specification to the marking information management unit133. The marking information management unit133acquires information output from the marking recognition unit132. The marking information management unit133causes the storage unit170to store the acquired information as the marking information171.

As illustrated inFIG.4, the marking information171at the point C is updated from the marking information171at the point B, in which the permitted travel directions of the left-hand lane are the left turn direction and the straight-ahead direction, the permitted travel direction of the driving lane is only the straight-ahead direction, and the permitted travel direction of the right-hand lane is only the right turn direction. In this manner, the marking information management unit133updates the marking information171stored in the storage unit170every time new road markings are recognized with the travel of the vehicle M.

In the first drive assist pattern, the automatic driving control device100performs drive assist in accordance with a change in lanes in a case in which the vehicle M changes lanes.FIG.5is a diagram for explaining processing performed by the recognition unit130in the first drive assist pattern. As illustrated inFIG.5, the vehicle M passes through the point D where the road marking RMd is displayed and then changes lanes to the left-hand lane at the point E.

When the vehicle M is about to pass through the point D, the marking recognition unit132acquires information indicating the road marking RMd and recognizes the road marking on the driving lane and the road marking on the adjacent lane (left-hand lane).

Note that a right-hand lane is not present at the point D, no road markings on a right-hand lane are recognized. The marking recognition unit132specifies that the permitted travel directions of the driving lane are the straight-ahead direction and the right turn direction while the permitted travel directions of the left-hand lane are the straight-ahead direction and the left turn direction on the basis of results of the recognition. The marking recognition unit132outputs information indicating results of the specification to the marking information management unit133. The marking information management unit133acquires the information output from the marking recognition unit132. The marking information management unit133causes the storage unit170to store the acquired information as the marking information171.

FIG.6illustrates, in the form of a table, the marking information171stored in the storage unit170at the timing at which the vehicle M passes through each point. As illustrated inFIG.6, according to the marking information171at the point D, the permitted travel directions of the left-hand lane are the left turn direction and the straight-ahead direction, and the permitted travel directions of the driving lane are the straight-ahead direction and the right turn direction. Note that since the right-hand lane is not present at the point D as illustrated inFIG.5, and no road marking is recognized, the permitted travel direction of the right-hand lane has not yet been detected in the marking information171at the point D as illustrated inFIG.6. Note that in a case in which the permitted travel direction has not yet been detected, it is determined that the vehicle can travel in all the directions as described above.

The point E is a point at which the vehicle M has changed lanes to the left-hand lane. The lane change detection unit134detects a change in lanes of the vehicle M. If the lane change detection unit134detects a change in lanes, then the marking information management unit133replaces data such that the marking information171stored in the storage unit170is slid for each lane. Specifically, the marking information management unit133updates the marking information171stored in the storage unit170such that the permitted travel direction of the lane after the lane change of the vehicle M (the left-hand lane inFIG.5) is changed to the permitted travel direction of the vehicle M as illustrated inFIG.6. Also, the marking information management unit133updates the marking information171stored in the storage unit170such that the permitted travel direction of the lane before the lane change of the vehicle M is changed to the permitted travel direction of the lane on the side opposite to the direction to which the lane change has been performed (the right-hand lane inFIG.5). Note that since the left-hand lane is not present for the driving lane after the change in lanes as illustrated inFIG.5, the permitted travel direction of the left-hand lane has not yet been detected in the marking information171at the point E as illustrated inFIG.6. Note that in a case in which the permitted travel direction has not yet been detected, it is determined that the vehicle can travel in all the directions as described above.

Note that the marking information management unit133may invalidate (or delete) the marking information171indicating the permitted travel direction of the lane that has become two lanes away due to the change in lanes. A configuration in which when the vehicle M has changed lanes from the center lane of a three-lane road to one of adjacent lanes (the left-hand lane, for example; a first adjacent lane), for example, the marking information171indicating the permitted travel direction of the other adjacent lane (the right-hand lane of the lane before the change in lanes, for example; a second adjacent lane) is invalidated (or deleted) may be employed. It is thus possible to reduce a data storage capacity required for the storage unit170and to reduce device cost and the like.

The reset determination unit135invalidates (or deletes) all pieces of marking information171stored in the storage unit170in a case in which a predetermined condition is satisfied. The predetermined condition mentioned here means that at least one of turning right or left of the vehicle M, passing of the vehicle M through an intersection, and travel of the vehicle M by a predetermined distance (a long distance such as 800 [m], for example) after recognition of a road marking, for example, is satisfied. This is because the case in which the aforementioned condition is satisfied is a situation in which the permitted travel direction recognized until then is typically meaningless any more.

In a case in which the vehicle M turns right or left, since the road along which the vehicle M travels is changed, the marking information171stored until then is invalidated. Also, since a road marking indicating a permitted travel direction typically indicates a permitted travel direction of each lane until a next intersection, the marking information171stored until then is invalidated in a case in which the vehicle M passes through the intersection.

Also, in a case in which the vehicle M has traveled by some distance with no other road marking detected after a road marking is recognized, there is a high probability that a road marking displayed in the course has been missed or passing of an intersection has been missed. Therefore, in the case in which the vehicle M has traveled by a predetermined distance with no next road marking recognized after a road marking is recognized, the marking information171stored until then is invalidated.

FIG.7is a diagram illustrating a state in which the vehicle M passes through an intersection. The marking recognition unit132recognizes a road marking at the point F and causes the storage unit170to store the marking information171. Thereafter, the vehicle M passes through an intersection IS at the point G. Also,FIG.8is a diagram illustrating a state in which the vehicle M travels by a predetermined distance after a road marking is recognized. The marking recognition unit132recognizes a road marking at the point H and causes the storage unit170to store the marking information171. Thereafter, the vehicle M arrives at the point I separated from the point H by the predetermined distance. Note that the reset determination unit135measures the distance by which the vehicle M has traveled after the road marking is recognized.

FIG.9illustrates, in the form of a table, the marking information171stored in the storage unit170at a timing at which the vehicle M passes through each point. In this manner, the reset determination unit135invalidates (or deletes) all the pieces of marking information171stored in the storage unit170after passing through the point G or the point I.

Operations of Automatic Driving Control Device

Hereinafter, an example of operations of the automatic driving control device100will be described.FIG.10is a flowchart illustrating operations of the automatic driving control device100according to the embodiment of the invention.

The lane recognition unit131recognizes the driving lane and the adjacent lanes on the basis of information output from the object recognition device16(Step S010). Next, in a case in which the marking recognition unit132specifies each of the permitted travel directions of the driving lane and the adjacent lanes on the basis of the road markings displayed on the recognized driving lane and the adjacent lanes (Step S020), the marking information management unit133causes the storage unit170to store information indicating the specified permitted travel directions as marking information171. Also, the reset determination unit135starts to measure the distance by which the vehicle M has traveled after the road markings are recognized (Step S030).

Next, in a case in which the lane change detection unit134detects a change in lanes of the vehicle M (Step S040), the marking information management unit133replaces data such that the marking information171stored in the storage unit170is slid for each lane as illustrated inFIG.6(Step S050). Next, in a case in which the marking recognition unit132specifies each of the permitted travel directions of the driving lane and the adjacent lanes on the basis of road markings displayed on the recognized driving lane and the adjacent lanes (Step S060), the marking information management unit133updates the marking information171stored in the storage unit170with the information indicating the specified permitted travel directions. Also, the reset determination unit135resets the measurement of the distance by which the vehicle M has traveled after the road markings are recognized (Step S070).

In a case in which the predetermined condition is satisfied (Step S080), the reset determination unit135executes invalidation (or deletion) of all the pieces of marking information171stored in the storage unit170(Step S090). The predetermined condition mentioned here means that any one of turning right or left of the vehicle M, passing of the vehicle M through an intersection, and the travel of the vehicle M by a predetermined distance after recognition of a road marking is satisfied, for example, as described above.

The automatic driving control device100repeats the aforementioned operations in and after Step S040until the vehicle M ends travel (step S100). As described hitherto, the operations of the automatic driving control device100illustrated in the flowchart inFIG.10ends.

As described above, the automatic driving control device100according to the embodiment recognizes a road marking of the driving lane and a road marking of one adjacent lane from the image of the pavement captured from the vehicle M. The automatic driving control device100specifies the permitted travel direction of the driving lane and the permitted travel direction of the one adjacent lane. The automatic driving control device100stores first information indicating the permitted travel direction of the driving lane and second information indicating the permitted travel direction of the one adjacent lane. In a case in which a change in lanes to the one adjacent lane is detected, the automatic driving control device100updates the information such that the second information is the permitted travel direction of the driving lane and the first information is the permitted travel direction of the other adjacent lane. As described above, the automatic driving control device100may further recognize the permitted travel direction of the other adjacent lane and may further specify the permitted travel direction of the other adjacent lane. In this case, the automatic driving control device100further stores, in the storage unit, third information indicating a permitted travel direction of the other adjacent lane. In a case in which a change in lanes to the one adjacent lane is detected, the automatic driving control device100invalidates the third information. As described above, in a case in which it is determined that the condition that any of the right or left turning of the vehicle M, passing through an intersection, and travel by a predetermined distance after the permitted travel direction is recognized is performed is satisfied, the automatic driving control device100may invalidate the aforementioned stored information indicating the permitted travel direction.

With this configuration, the automatic driving control device100can use the second information indicating the permitted travel direction of the adjacent lane that is the lane after the change in lanes as information indicating the permitted travel direction of the driving lane after that time immediately after the change in lanes is performed and can thus perform more appropriate drive assist. For example, the automatic driving control device100can provide, to the MPU60, information indicating the permitted travel direction of each lane after the change in lanes at the timing when the change in lanes is performed.

Second Drive Assist Control Pattern

In a second drive assist pattern, when there is a lane for which the marking recognition unit132cannot recognize a road marking, for example, a case in which another vehicle (hereinafter, referred to as “another vehicle”) is stopped on the road marking, the marking information management unit133does not update the marking information171for the lane.

FIG.11is a diagram for explaining processing performed by the recognition unit130in the second drive assist pattern.FIG.11illustrates a road RD, a lane center line CL, two lane boundary lines BL, a lane outer line OL, a road marking RMj displayed at the point J, and a road marking RMIc displayed at the point K. Also,FIG.11illustrates the vehicle M that is about to pass through the point J and the point K and each of other vehicles m1to m3. In other words,FIG.11illustrates a state in which the vehicle M travels along the road RD while passing through the point J and the point K in this order.

As illustrated inFIG.11, a state in which another vehicle m3travel along the left-hand lane is stopping on the road marking RMk displayed at the point K when the vehicle M is about to pass through the point K after the road marking RMj is recognized when the vehicle M is about to pass through the point J is illustrated. In this manner, the marking recognition unit132of the vehicle M cannot recognize the road marking RMk on the left-hand lane and cannot specify the permitted travel direction of the left-hand lane. In such a case, the marking recognition unit132does not update the marking information171indicating the permitted travel direction of the left-hand lane based on the previous (the last) road marking RMj displayed at the point J and holds the marking information171as it is. In other words, the marking recognition unit132regards the road marking indicating the straight-ahead direction and the left turn direction that are the permitted travel directions of the left-hand lane based on the road marking RMj displayed at the last point J as being similarly displayed at the point K as well for the left-hand lane. Therefore, the marking information171indicating the permitted travel directions of the left-hand lane based on the road marking RMj is not invalidated in response to travel of the vehicle M by the predetermined distance from the point J and can be held until the vehicle M travels by the predetermined distance from the point K.

Note that in a case in which the marking recognition unit132recognizes a road marking of at least one lane, the marking recognition unit132estimates that road markings are also displayed on the other lanes at the same point. This is because road markings of lanes are typically displayed together at one location in many cases.FIG.12illustrates, in the form of a table, the marking information171stored in the storage unit170at a timing at which the vehicle M passes through each point. As illustrated inFIG.12, the information indicating the permitted travel directions of the left-hand lane specified at the point J is taken over as permitted travel directions of the left-hand lane at the next point K where the road marking of the left-hand lane has not been recognized.

As described above, the automatic driving control device100according to the embodiment does not update the information indicating the permitted travel direction of the lane for which the road marking has not been recognized. The automatic driving control device100includes such a configuration, can thus use information indicating the permitted travel direction based on the road marking previously recognized even for the lane for which the road marking has not been recognized, and can thereby perform appropriate drive assist. When there is a lane for which the road marking has not been recognized, for example, the automatic driving control device100can provide, to the MPU60, information indicating the previously recognized permitted travel direction of the lane.

Third Drive Assist Control Pattern

In a third drive assist pattern, the notification control unit180provides a notification indicating an alert to the passenger of the vehicle M in a case in which the permitted travel direction based on the marking information171stored in the storage unit170does not conform to the permitted travel direction indicated by a traffic signal.

FIG.13is a diagram for explaining processing performed by the recognition unit130in the third drive assist pattern.FIG.13illustrates a road RD, a lane center line CL, a lane boundary line BL, a lane outer line OL, a road marking RM1displayed at the point L, a traffic signal S, and the vehicle M.FIG.13illustrates a state in which the vehicle M travels toward an intersection where the traffic signal S is placed after passing through the point L.

As illustrated inFIG.13, the permitted travel directions of the driving lane are changed to the straight-ahead direction and the left turn direction according to the marking information171stored in the storage unit170by the vehicle M passing through the point L. However, the permitted travel direction indicated by the traffic signal S is the right turn direction as illustrated inFIG.13. In a case in which the permitted travel directions based on the marking information171stored in the storage unit170do not conform to the permitted travel direction indicated by the traffic signal S in this manner, the notification control unit180controls the HMI30and provides a notification to the passenger of the vehicle M. Note that as a notification method, any method such as sound notification, a notification based on image display, a notification based on turning-on of an alert lamp, or a notification based on generation of vibration, for example, may be used.

Note that the notification control unit180may be configured not to provide a notification in a case in which the permitted travel directions based on the marking information171stored in the storage unit170and the permitted travel direction indicated by the traffic signal S partially conform to each other. In a case in which the permitted travel directions based on the marking information171stored in the storage unit170are the straight-ahead direction and the left turn direction and the permitted travel directions indicated by the traffic signal S are the straight-ahead direction and the right turn direction, for example, the fact that the straight-ahead direction is the permitted travel direction is common to the both. In this case, the notification control unit180may not provide a notification. This is because there is a probability that the vehicle M plans to travel in the straight-ahead direction and the vehicle M is traveling along a correct lane.

Note that in a case in which the traffic signal S does not include any arrow signal, or in a case in which an arrow signal is not turned on, the notification control unit180may determine whether or not to provide a notification using a display state of a three-light signal. In this case, the notification control unit180may recognize all the directions as permitted travel directions in a case in which a blue or yellow light of the three-light signal of the traffic signal S is turned on and may recognize that not all the directions are the permitted travel direction in a case in which a red light is turned on, for example.

As described above, the automatic driving control device100according to the embodiment recognizes a display state of the traffic signal S and specifies the permitted travel direction. In a case in which the permitted travel direction based on the display state of the traffic signal S and the permitted travel direction of the driving lane stored on the basis of the road marking do not conform to each other, the automatic driving control device100provides a notification of the passenger of the vehicle M. Since the automatic driving control device100can provide a notification to the passenger using the HMI30, for example, in a case in which the vehicle M is traveling along a lane that does not match a desired traveling direction, for example, by including such a configuration, it is possible to allow the passenger to recognize a risk in the surroundings in an early stage and to perform more appropriate drive assist.

Fourth Drive Assist Control Pattern

In a fourth drive assist pattern, in a case in which at least one lane line has not yet been detected, the lane interpolation unit137interpolates the lane line, which has not been detected, with an estimated (virtual) lane line. In this manner, the lane recognition unit131can recognize the lane. Also, the lane interpolation unit137appropriately corrects the estimated lane line in response to new recognition of a road marking in a case in which a state where the lane line has not yet been detected has continued.

FIGS.14to16are diagrams for explaining processing performed by the recognition unit130in the fourth drive assist pattern.FIG.14illustrates a case where a left-side lane line BL of the driving lane along which the vehicle M is traveling has been recognized while a right-side lane line has not been detected. In this case, the lane interpolation unit137determines the position of a right-side lane line VLr that is a virtual lane line. The lane interpolation unit137determines the position of the right-side lane line VLr on the basis of the distance from the position of the vehicle M to the position of the left-side lane line BL, for example. For example, the lane interpolation unit137regards, as the position of the right-side lane line VLr, the position separated on the right side of the travel direction of the vehicle M by the same distance as the distance from the travel line of the vehicle M to the position of the left-side lane line BL. In other words, the lane interpolation unit137regards the vehicle M as being traveling at the center of the driving lane, for example.

Also,FIG.15illustrates a case where both the left-side lane line and the right-side lane line of the driving lane along which the vehicle M is traveling have not yet been detected. In this case, the lane interpolation unit137determines each of the position of the left-side lane line VL1and the right-side lane line VLr that are virtual lane lines. The lane interpolation unit137regards, as the position of the left-side lane line VL1and the position of the right-side lane line VLr, the positions separated on both sides of the travel direction of the vehicle M by a predetermined distance (2 [m], for example) from the travel line of the vehicle M, for example. In other words, the lane interpolation unit137regards that the width of the driving lane as 4 [m] (=2 [m]×2) and regards the vehicle M as being traveling at the center of the driving lane, for example.

FIG.16illustrates processing for correcting estimated lane lines.FIG.16illustrates a road RD, a lane center line CL, two lane boundary lines BL, a lane outer line OL, a road marking RMm displayed at the point M, and a road marking RMn displayed at the point N. Here, it is assumed that a state in which recognition of the lane center line CL, the two lane boundary lines BL, and the lane outer line OL is difficult due to degradation of painting or snowing, for example has been achieved.

Also,FIG.16illustrates each vehicle M that is about to pass through the point M and the point N. In other words,FIG.3illustrates a state in which the vehicle M travels along the road RD while passing through the point M and the point N in this order. Also,FIG.16illustrates each of a left-side lane line VL1-1and a right-side lane line VLr-1estimated by the lane interpolation unit137when the vehicle M is about to pass through the point M and a left-side lane line VL-2and a right-side lane line VLr-2estimated by the lane interpolation unit137when the vehicle M is about to pass through the point N.

As illustrated inFIG.16, it is assumed that the lane recognition unit131has recognized the lane inclined in the right lower direction in the drawing, which is difference from an actual driving lane, as a driving lane from the left-side lane line VL1-1and the right-side lane line VLr-1estimated by the lane interpolation unit137when the vehicle M is about to pass through the point M. In this manner, the marking recognition unit132erroneously recognizes that the road marking RMm is the road marking of the driving lane. However, it is assumed that the lane recognition unit131has recognized a lane that is approximate to the actual driving lane as a driving lane from the left-side lane line VL1-2and the right-side lane line VLr-2estimated by the lane interpolation unit137when the vehicle M is about to pass through the point N next. In this manner, the marking recognition unit132recognizes the road marking RMn as a road marking of the adjacent lane on the right side.

At this time, in a case in which the permitted travel direction indicated by the road marking RMm recognized at the point M and the permitted travel direction indicated by the road marking RMn recognized at the point N conform to each other (in other words, the road marking RMm and the road marking RMn are the same road markings), the marking information management unit133regards the estimation of the left-side lane line VL1-1and the right-side lane line VLr-1at the point M as erroneous estimation and updates the marking information171stored in the storage unit170to invalidate (delete) the information indicating the permitted travel direction of the driving lane.

FIG.17illustrates, in the form of a table, the marking information171stored in the storage unit170at a timing at which the vehicle M passes through each point. As illustrated inFIG.17, the marking information management unit133invalidates (or deletes), at the point N, the information indicating the permitted travel direction of the driving lane stored in the storage unit170at the point M and updates the marking information171to achieve a state in which the information has not yet been detected. Note that in a case in which the permitted travel direction has not yet been detected, it is determined that the vehicle M can travel in all the directions as described above. Also, the marking information management unit133updates the marking information171to update the information indicating the permitted travel direction of the right-hand lane on the basis of the road marking RMn recognized at the point N.

As described above, the automatic driving control device100according to the embodiment specifies the driving lane and the adjacent lanes on the basis of the lane lines recognized from an image, and in a case in which there is a lane line that has not been recognized, the automatic driving control device100interpolates the lane line which has not yet been detected with a virtual lane line estimated with reference to the travel direction of the vehicle M. The automatic driving control device100can estimate the driving lane and the adjacent lanes, use the information indicating the permitted travel directions even in the case in which the lane lines cannot accurately be recognized, and thus perform more appropriate drive assist, by including such a configuration. Even in a case in which the lane lines are not recognized, for example, the automatic driving control device100can estimate the lanes based on the virtual lane line and provide information indicating the permitted travel direction of the estimated lane to the MPU60.

Also, in a case in which the permitted travel direction of the driving lane specified on the basis of the lane line estimated with reference to the travel direction of the vehicle M and the permitted travel direction of the adjacent lanes based on the road marking recognized from a new image conform to each other when control of the vehicle M is performed on the basis of the virtual lane line, the automatic driving control device100invalidates the information indicating the permitted travel direction of the driving lane and updates the information indicating the permitted travel direction of the adjacent lane. With such a configuration, the automatic driving control device100can further enhance estimation accuracy of the lane in response to recognition of road markings with travel of the vehicle M, correct information indicating the permitted travel direction of the driving lane, and thus perform more appropriate drive assist.

Hardware Configuration

FIG.18is a diagram illustrating an example of a hardware configuration of the automatic driving control device100according to the embodiment. As illustrated the drawing, the automatic driving control device100has a configuration in which a communication controller100-1, a CPU100-2, a random access memory (RAM)100-3used as a working memory, a read only memory (ROM)100-4storing a boot program and the like, a storage device100-5such as a flash memory and a hard disk drive (HDD), a drive device100-6, and the like are connected to each other via an internal bus or a dedicated communication line. The communication controller100-1performs communication with components other than the automatic driving control device100. The storage device100-5stores a program100-5aexecuted by the CPU100-2. The program is developed in the RAM100-3by a direct memory access (DMA) controller (not illustrated) or the like and is executed by the CPU100-2. In this manner, some or all of the first control unit120, the second control unit160, and the notification control unit180are realized.

The aforementioned embodiments can be expressed as follows.

A drive assist device including:a storage device configured to store a program; anda hardware processor torecognize a road marking on a driving lane and a road marking on one adjacent lane from an image of a pavement captured from a vehicle and specify a permitted travel direction of the driving lane and a permitted travel direction of the one adjacent lane;detect a change in lanes performed by the vehicle; andcause a storage unit to store first information indicating the permitted travel direction of the driving lane and second information indicating the permitted travel direction of the one adjacent lane, and in a case in which the change in lanes to the one adjacent lane is detected, update the information stored in the storage unit to change the second information as the permitted travel direction of the driving lane and the first information as the permitted travel direction of the other adjacent lane.

Although an implementation of the invention has been described using the embodiment, the invention is not limited to such an embodiment at all, and various modifications and replacements can be made without departing from the gist of the invention.