Method of switching vehicle drive mode from automatic drive mode to manual drive mode depending on accuracy of detecting object

An apparatus includes a memory, and circuitry which, in operation, performs operations including, storing, in the memory, an object occurrence map defining an occurrence area where there is a possibility that an object appears, detecting the object included in a captured image of a scene seen in a running direction of a vehicle, switching a vehicle drive mode, based on a result of the detection of the object and the object occurrence map, from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

BACKGROUND

1. Technical Field

The present disclosure relates to a drive control apparatus, a drive control method, and a drive control program for controlling driving a vehicle in one of following modes, an automatic drive mode in which the vehicle is driven automatically, and a manual drive mode in which the vehicle is driven manually by a driver.

2. Description of the Related Art

In recent years, actives to develop techniques of automatically driving vehicles have been made. In a circumstance in which it is difficult to drive, in an automatic drive mode, a vehicle having an automatic driving capability, it may be desirable to drive the vehicle manually by a driver instead of driving the vehicle in the automatic drive mode.

For example, in a technique disclosed in U.S. Pat. No. 8,352,110, a control computer determines whether it is possible to safely control a vehicle, and in a case where the control computer determines that it is possible to safely control the vehicle, the vehicle is automatically driven, but otherwise the vehicle is manually driven.

SUMMARY

However, a further improvement is necessary in the technique described above.

One non-limiting and exemplary embodiment provides an apparatus including a first memory, and circuitry that, in operation, performs operations including, storing, in the first memory, an object occurrence map defining an occurrence area where there is a possibility that an object appears, detecting the object included in a captured image of a scene seen in a running direction of a vehicle, switching a vehicle drive mode, based on a result of the detection of the object and the object occurrence map, from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

In the present disclosure, depending on the accuracy of detecting an object, it is possible to switch the vehicle drive mode from the automatic drive mode in which the vehicle is driven automatically to the manual drive mode in which the vehicle is driven manually by a driver.

DETAILED DESCRIPTION

In the technique disclosed in U.S. Pat. No. 8,352,110, as described above, a control computer determines whether it is possible to safely control a vehicle, and in a case where the control computer determines that it is possible to safely control the vehicle, the vehicle is automatically driven but otherwise the vehicle is manually driven. In the technique disclosed in U.S. Pat. No. 8,352,110, for example, the control computer determines that it is not possible to safely control the vehicle at a point for which there is no history indicating that the point has been driven in the past. In a case where a plurality of obstacles approaching the vehicle are detected, the control computer determines that it is not possible to safely control the vehicle.

In U.S. Pat. No. 8,352,110, conditions in terms of the state of the external circumstance where it is not possible to perform safe control are defined based on knowledge acquired by humans. However, there is a difference in criterion for determination between humans and a system in recognizing an obstacle in the external circumstance. Therefore, there is a possibility that an external circumstance easily recognizable by humans may be difficult for the recognition system to correctly recognize. Therefore, in the automatic driving, it is necessary to take into account behavior and performance of the recognition system in recognizing the state of the external circumstance.

In view of the above, according to an aspect, the present disclosure provides an apparatus including a first memory, and circuitry that, in operation, performs operations including including storing, in the first memory, an object occurrence map defining an occurrence area where there is a possibility that an object appears, detecting the object included in a captured image of a scene seen in a running direction of a vehicle, switching a vehicle drive mode, based on a result of the detection of the object and the object occurrence map, from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

In this aspect, the object occurrence map defining the occurrence area, where there is a possibility that an object appears, is stored in the first memory. An object in the captured image of the scene seen in the running direction of the vehicle is detected. Based on the result of the detection of the object and the object occurrence map, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode. The driving of the vehicle is controlled in the switched manual drive mode.

Thus, the determination as to whether the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode is performed based on the result of the detection of the object and the object occurrence map defining the occurrence area where there is a possibility that an object may appear. That is, depending on the object detection accuracy, it is possible to switch the vehicle drive mode from the automatic drive mode in which the vehicle is automatically driven to the manual drive mode in which the vehicle is driven manually by a driver.

In the apparatus, in another aspect, the object occurrence map may be managed in relation to map information.

In this aspect, the object occurrence map is managed in relation to map information and thus it is possible to define the occurrence area on a map where there is a possibility that an object appears.

In the apparatus, in another aspect, the operations may further include acquiring a running circumstance around the vehicle, and, depending on the acquired running circumstance, generating a viewpoint-converted object occurrence map from the object occurrence map so as to be represented from the same viewpoint as that of the captured image, wherein the switching may include superimposing the viewpoint-converted object occurrence map on the captured image, and in a case where a location of the object detected in the captured image is outside the occurrence area in the viewpoint-converted object occurrence map, switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

In this aspect, the running circumstance around the vehicle is acquired. Depending on the acquired running circumstance, the viewpoint-converted object occurrence map is generated from the object occurrence map so as to be represented as seen from the same viewpoint as that of the captured image. The viewpoint-converted object occurrence map is superimposed on the captured image. In a case where the location of the object detected in the captured image is outside the occurrence area in the viewpoint-converted object occurrence map, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

Therefore, in the case where the location of the object detected in the captured image is outside the occurrence area in the viewpoint-converted object occurrence map, it is determined that the detection of the object is not reliable, and thus the vehicle drive mode is switched from the automatic drive mode to the manual drive mode and the vehicle is driven by a driver.

In the apparatus, in another aspect, the map may include a three-dimensional map represented in a three-dimensional coordinate system, the occurrence area may include an occurrence space in which there is a possibility that the object appears on the three-dimensional map, and the generating may include converting the object occurrence map in the three-dimensional coordinate system to the viewpoint-converted object occurrence map in a two-dimensional coordinate system as seen from the same viewpoint as that of the captured image.

In this aspect, the map includes the three-dimensional map represented in the three-dimensional coordinate system, and the occurrence area includes the occurrence space in which there is a possibility that the object appears on the three-dimensional map. The object occurrence map in the three-dimensional coordinate system is converted to the viewpoint-converted object occurrence map in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image.

Thus, the object occurrence map defines the occurrence space on the three-dimensional map where there is a possibility that an object appears, and it is possible to more accurately define the range of the occurrence space where there is a possibility that the object, which makes it possible to enhance the reliability of switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

In the apparatus, in another aspect, in the storing, a plurality of object occurrence maps may be stored that are respectively represented in the two-dimensional coordinate system as seen from a same viewpoint as that of the captured image, and that respectively define occurrence areas depending on shapes of a plurality of roads, and the operations may further include acquiring a shape of a road included in the captured image of the scene seen in the travelling direction of the vehicle, and determining one object occurrence map corresponding to the acquired shape of the road from the plurality of object occurrence maps.

In this aspect, the first memory stores a plurality of object occurrence maps which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of a plurality of roads. The shape of the road is acquired from the captured image of the scene seen in the running direction of the vehicle. One object occurrence map corresponding to the acquired shape of the road is determined from the plurality of object occurrence maps.

By storing in the memory, not map data, but a plurality of object occurrence maps which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of a plurality of roads in the above-described manner, it is possible to reduce the amount of data stored in the memory.

In the apparatus in another aspect, the map may include a three-dimensional map represented in a three-dimensional coordinate system, and the occurrence area may include an occurrence space in which there is a possibility that the object appears on the three-dimensional map, in the storing, a plurality of object occurrence maps may be stored that are respectively represented in the three-dimensional coordinate system, and that respectively define occurrence spaces depending on shapes of a plurality of roads, and the operations may further include acquiring a shape of a road included in the captured image of the scene seen in the travelling direction of the vehicle, determining one object occurrence map corresponding to the acquired shape of the road from the plurality of object occurrence maps, and in a case where the one object occurrence map is determined, converting the one object occurrence map in the three-dimensional coordinate system to an object occurrence map in the same two-dimensional coordinate system as that of the captured image.

In this aspect, the map includes the three-dimensional map represented in the three-dimensional coordinate system, and the occurrence area includes the occurrence space in which there is a possibility that an object appears on the three-dimensional map. The first memory stores the plurality of object occurrence maps which are respectively represented in the three-dimensional coordinate system and which respectively define occurrence spaces depending on shapes of a plurality of roads. The shape of a road is acquired from the captured image of the scene seen in the running direction of the vehicle. One object occurrence map corresponding to the acquired shape of the road is determined from the plurality of object occurrence maps. In a case where the one object occurrence map is determined, the one object occurrence map in the three-dimensional coordinate system is converted to the object occurrence map in the same two-dimensional coordinate system as that of the captured image.

Thus, the plurality of object occurrence maps each define an occurrence space on the three-dimensional map where there is a possibility that an object appears, and thus the range where there is a possibility that an object appears is defined more accurately as the occurrence space, which makes it possible to enhance the reliability of switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

In the apparatus, in another aspect, the apparatus may further include a second memory, and in the storing, a plurality of object occurrence maps may be stored that are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image, and that respectively define occurrence areas depending on shapes of a plurality of roads, the operations may further include determining, depending on a location of the object detected in the captured image, one object occurrence map from the plurality of object occurrence maps, storing, in the second memory, an identification number corresponding to the determined one object occurrence map, and wherein in the switching, in a case where the identification number stored in the second memory is switched a predetermined number of times during a predetermined period of time, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

In this aspect, in the storing, the first memory stores plurality of object occurrence maps which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of a plurality of roads. Depending on the location of the object detected in the captured image, one object occurrence map is determined from the plurality of object occurrence maps, and the identification number corresponding to the determined one object occurrence map is stored in the second memory. When the identification number stored in the second memory is switched the predetermined number of times during the predetermined period of time, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

Therefore, when a change in the object occurrence map occurs for a short period, it can be determined that objects are not accurately detected. In such a situation, by switching the vehicle drive mode from the automatic drive mode to the manual drive mode, it becomes possible to safely drive the vehicle by a driver.

In the apparatus, in another aspect, the object may be a person, and the occurrence area may be an area where the person passes.

In this aspect, it is possible to switch, depending on the person detection accuracy, the vehicle drive mode from the automatic drive mode in which the vehicle is driven automatically to the manual drive mode in which the vehicle is driven manually by a driver.

In another aspect, the present disclosure provides a method including detecting an object included in a captured image of a scene seen in a running direction of a vehicle, and based on a result of the detection of the object and an object occurrence map, defining an occurrence area where there is a possibility that the object appears, switching a vehicle drive mode from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

In this aspect, an object is detected from the captured image of the scene seen in the running direction of the vehicle. Based on a result of the detection of the object and the object occurrence map defining the occurrence area where there is a possibility that the object appears, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode. In the switched manual drive mode, the driving of the vehicle is controlled.

Thus, the determination as to whether the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode is performed based on the result of the detection of the object and the object occurrence map defining the occurrence area where there is a possibility that an object may appear. That is, depending on the object detection accuracy, it is possible to switch the vehicle drive mode from the automatic drive mode in which the vehicle is automatically driven to the manual drive mode in which the vehicle is driven manually by a driver.

In another aspect, the present disclosure provides a non-transitory computer-readable storage medium storing a program, the program causing a computer to execute a method when the program is executed by the computer, the method including detecting an object included in a captured image of a scene seen in a running direction of a vehicle, and based on a result of the detection of the object and an object occurrence map, defining an occurrence area where there is a possibility that the object appears, switching a vehicle drive mode from an automatic drive mode in which the vehicle is automatically driven to a manual drive mode in which the vehicle is driven manually by a driver, and controlling driving of the vehicle in the switched manual drive mode.

In this aspect, the object occurrence map defining the occurrence area where there is a possibility that an object appears is stored in the first memory. An object is detected from the captured image of the scene seen in the running direction of the vehicle. Based on the result of the detection of the object and the object occurrence map, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode. In the switched manual drive mode, the driving of the vehicle is controlled.

Thus, the determination as to whether the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode is performed based on the result of the detection of the object and the object occurrence map defining the occurrence area where there is a possibility that an object may appear. That is, depending on the object detection accuracy, it is possible to switch the vehicle drive mode from the automatic drive mode in which the vehicle is automatically driven to the manual drive mode in which the vehicle is driven manually by a driver.

Referring to accompanying drawings, embodiments of the present disclosure are described below. Note that the embodiments described below are provided by way of example, but the technical scope of the present disclosure is not limited to those embodiments.

First Embodiment

FIG. 1is a block diagram illustrating a configuration of a vehicle control system according to a first embodiment. The vehicle control system shown inFIG. 1includes a drive control apparatus1and a camera2. The drive control apparatus1controls driving of a vehicle in an automatic drive mode in which the vehicle is automatically driven or a manual drive mode in which the vehicle is driven manually by a driver. Although it is assumed by way of example that the vehicle is a car in the following description, the vehicle in the present disclosure is not limited to the car, but the vehicle may be a vehicle of a type other than the car, such as a motorcycle, a truck, a bus, a train, an airplane, or the like.

The camera2is installed near a rearview mirror of the vehicle to take an image of a scene in front of the vehicle. The camera2may take not only the image of the scene in front of the vehicle but an image of a scene behind the vehicle or other scenes.

The drive control apparatus1is installed in the vehicle. The drive control apparatus1includes a memory3and a processor4.

The memory3is a computer-readable storage medium such as a hard disk drive, a read only memory (ROM), a random access memory (RAM), an optical disk, a semiconductor memory, or the like. The memory3stores a drive control program executed by the processor4. The memory3includes an occurrence likelihood map storage13.

The occurrence likelihood map storage13is an example of the first memory for storing an occurrence likelihood map (an object occurrence map) defining an occurrence area where there is a possibility that an object appears. The occurrence likelihood map is managed in relation to map information. In the first embodiment, the occurrence likelihood map defines an occurrence area on a two-dimensional map where there is a possibility that an object appears. The object may be, for example, a person or a vehicle, and the occurrence likelihood map indicates individually an occurrence area where a person passes and an occurrence area where a vehicle passes.

The occurrence likelihood map is stored in advance in the occurrence likelihood map storage13. However, alternatively, the occurrence likelihood map may be acquired from a server via a communication unit (not shown) and stored in the occurrence likelihood map storage13.

Still alternatively, the occurrence area in the occurrence likelihood map may be calculated based on statistic data of detections of persons or vehicles in the past. The occurrence area of the occurrence likelihood map may be defined on the two-dimensional map by an operator, for example, according to a particular rule such as the Road Traffic Law such that an occurrence area is defined in an area where a person or a vehicle is allowed to pass.

The processor4is, for example, a central processing unit (CPU) and executes the drive control program stored in the memory3. The processor4includes an object detector11, a running circumstance acquirer12, an occurrence likelihood map generator14, a drive mode switcher15and a vehicle controller16.

The object detector11detects an object included in a captured image, taken by the camera2, of a scene seen in a running direction of a vehicle. More specifically, the object detector11detects a person or a vehicle included in the captured image, taken by the camera2, of the scene seen in the running direction of the vehicle. To detect the person or the vehicle included in the captured image, for example, the object detector11inputs the captured image to a machine-learned neural network thereby detecting the person or the vehicle. Note that objects detected by the object detector11are not limited to persons and vehicles.

The running circumstance acquirer12acquires a running circumstance around the vehicle. From the captured image, taken by the camera2, of the scene seen in the running direction of the vehicle, the running circumstance acquirer12recognizes a traffic lane line drawn on a road and acquires a shape of the road on which the vehicle is running. Furthermore, from the captured image, the running circumstance acquirer12recognizes a traffic sign and a traffic signal and acquires traffic sign information and traffic signal information. Furthermore, the running circumstance acquirer12acquires a travelling direction of the vehicle, a speed of the vehicle, and a current location of the vehicle. The running circumstance acquirer12may acquire the location and the orientation of the vehicle on a three-dimensional map by using a sensor (for example, Laser Imaging Detection and Ranging (LIDAR)) other than a camera. Note that it is also possible to acquire the current location from a not-shown Global Positioning System (GPS). It is possible to determine the travelling direction by acquiring locations of two points from the GPS and calculating the travelling direction from the locations of the two points.

The occurrence likelihood map generator14is an example of a generator and generates an occurrence likelihood map image as seen from the same viewpoint as that of a captured image depending on the running circumstance acquired by the running circumstance acquirer12. In the present embodiment, the occurrence likelihood map generator14generates the occurrence likelihood map image as described above. However, the form of the occurrence likelihood map is not limited to the occurrence likelihood map image. For example, the occurrence likelihood map may be represented in the form of a two-dimensional matrix corresponding to a viewpoint. The occurrence likelihood map generator14is capable of identifying the location of the vehicle on the occurrence likelihood map and the direction in which the vehicle is running based on the current location of the vehicle and the travelling direction of the vehicle acquired by the running circumstance acquirer12. Furthermore, the occurrence likelihood map generator14identifies the position of the viewpoint and the shooting angle of the camera2from the location of the traffic lane acquired by the running circumstance acquirer12. The running circumstance acquirer12may acquire the location of the viewpoint and the shooting angle of the camera2from pre-stored data. The occurrence likelihood map generator14then generates the occurrence likelihood map image as seen in the traveling direction from the current location of the vehicle on the occurrence likelihood map and as seen from the same viewpoint as that of the captured image.

The drive mode switcher15is an example of a switcher and switches the vehicle drive mode from the automatic drive mode to the manual drive mode based on the object detection result and the occurrence likelihood map. The drive mode switcher15superimposes the occurrence likelihood map image on the captured image. On this superimposed image, if the location of an object in the captured image is outside the occurrence area in the occurrence likelihood map image, then the drive mode switcher15switches the drive mode from the automatic drive mode to the manual drive mode.

The vehicle controller16controls the driving of the vehicle in the automatic drive mode in which the vehicle is driven automatically or the manual drive mode in which the vehicle is driven manually by a driver. In the case where the drive mode is the manual drive mode as a result of the switching by the drive mode switcher15, the vehicle controller16controls the driving of the vehicle in the manual drive mode.

Next, an operation of the drive control apparatus according to the first embodiment is described below.

FIG. 2is a flow chart illustrating the operation of the drive control apparatus according to the first embodiment.

First, in step S1, the object detector11detects a person included in a captured image, taken by the camera2, of a scene seen in a running direction of a vehicle. Note that in the following description, it is assumed by way of example that a person is detected as an object. To detect the person included in the captured image, the object detector11performs image recognition. Note that the object detector11may detect not only a pedestrian but also a person riding a bicycle.

Next, in step S2, the running circumstance acquirer12acquires a running circumstance around the vehicle. More specifically, as described above, the running circumstance acquirer12recognizes traffic lanes drawn on a road on the captured image taken by the camera2, and acquires a shape of the road on which the vehicle is running. Furthermore, the running circumstance acquirer12acquires the travelling direction of the vehicle and the current location of the vehicle.

Next, in step S3, the occurrence likelihood map generator14generates an occurrence likelihood map image as seen from the same viewpoint as that of the captured image depending on the running circumstance acquired by the running circumstance acquirer12.

Next, in step S4, the drive mode switcher15superimposes the occurrence likelihood map image on the captured image, and determines whether the location of the person included in the captured image is outside the occurrence area in the occurrence likelihood map image. In a case where it is determined that the location of the detected person is outside the occurrence area (YES in step S4), in step S5, the drive mode switcher15switches the drive mode from the automatic drive mode to the manual drive mode.

On the other hand, in a case where it is determined that the location of the detected person is not outside the occurrence area (NO in step S4), then in step S6, the drive mode switcher15maintains the current automatic drive mode.

Note that in a case where when the drive mode is in the manual drive mode, it is determined that the location of the detected person is not outside the occurrence area, the drive mode switcher15may switch the vehicle drive mode from the manual drive mode to the automatic drive mode.

In the case where the vehicle drive mode is switched from the automatic drive mode to the manual drive mode, the drive mode switcher15may inform a driver that the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode. In this case, the drive mode switcher15may provide the notification by a voice or an image. Furthermore, after the drive mode switcher15notifies the driver that the drive mode is to be switched from the automatic drive mode to the manual drive mode, the drive mode switcher15may switch the vehicle drive mode from the automatic drive mode to the manual drive mode.

Although in the example described above with reference toFIG. 2, the determination as to whether the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode is performed every one frame, the determination as to whether the vehicle drive mode is to be switched from the automatic drive mode to the manual drive mode may be performed based on a result for a plurality of frames.

Referring toFIG. 3, a process of switching the vehicle drive mode from the automatic drive mode to the manual drive mode is described below.

The drive mode switcher15superimposes the occurrence likelihood map image102on the captured image101. The occurrence likelihood map image102includes an occurrence area103where there is a higher probability that a person appears than in other areas. In the example shown inFIG. 3, the occurrence area103is defined on a pedestrian sidewalk outside a vehicle road. Although in the example shown inFIG. 3, the occurrence area103has an elliptic shape, the shape of the occurrence area in the present disclosure is not limited to ellipse, and any shape may be employed as the shape of the occurrence area.

InFIG. 3, the object detector11detects a person, but also detects utility poles erroneously as persons. In this specific case, the person is detected at a detection position111within the occurrence area103, but utility poles are detected at detection positions112and113outside the occurrence area103. Therefore, the drive mode switcher15determines that the detection positions112and113of the persons in the captured image101are outside the occurrence area103in the occurrence likelihood map image102, and the drive mode switcher15switches the vehicle drive mode from the automatic drive mode to the manual drive mode.

In the example shown inFIG. 3, there are two objects which are detected as persons by the object detector11and whose detection positions are determined as being outside the occurrence area103. However, when at least one person is detected outside the occurrence area103, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

As described above, in the case where a person is detected outside the occurrence area103which is an area where there is a possibility that a person appears, there is a high probability that the detection of the person by the object detector11is not correct, and thus it is determined that it is difficult to continue the driving in the automatic drive mode, and the vehicle drive mode is switched from the automatic drive mode to the manual drive mode. Thus, depending on the reliability of the detection of the object, the vehicle drive mode is switched from the automatic drive mode in which the vehicle is driven automatically to the manual drive mode in which the vehicle is driven manually by a driver, that is, when the reliability of the detection is low, the automatic drive mode is cancelled and the vehicle is driven by the driver.

In the example described above, the occurrence likelihood map defines an occurrence area on a two-dimensional map where there is a possibility that an object appears. Alternatively, the occurrence likelihood map may define an occurrence space on a three-dimensional map where there is a possibility that an object appears. In this case, the occurrence likelihood map generator14converts an occurrence likelihood map in the three-dimensional coordinate system to an occurrence likelihood map image in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image depending on the running circumstance acquired by the running circumstance acquirer12.

In this case, the occurrence likelihood map defines the occurrence space on the three-dimensional map where there is a possibility that an object appears, and thus the range where there is a possibility that an object appears is defined more accurately as the occurrence space, which makes it possible to enhance the reliability of switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

FIG. 4is a diagram illustrating examples of captured images and occurrence likelihood map images according to a modification to the first embodiment.

As illustrated inFIG. 4, the occurrence likelihood map storage13may store a plurality of occurrence likelihood map images which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of roads. For example, in an occurrence likelihood map image102a, occurrence areas103are defined on both sides of a vehicle road. In an occurrence likelihood map image102b, an occurrence area103is defined on a left-hand side of a vehicle road. In an occurrence likelihood map image102c, an occurrence area103is defined at a T-shape intersection.

In the modification to the first embodiment, the drive control apparatus1includes an occurrence likelihood map determiner instead of the occurrence likelihood map generator14.

The running circumstance acquirer12recognizes traffic lanes drawn on a road on the captured image taken by the camera2, and acquires the shape of a road included in a captured image of a scene seen in a running direction of a vehicle. The occurrence likelihood map determiner determines one occurrence likelihood map image corresponding to the shape of the road acquired by the running circumstance acquirer12from a plurality of occurrence likelihood map images stored in the occurrence likelihood map storage13. That is, inFIG. 4, for the captured image101a, the occurrence likelihood map image102athat matches the road shape of the captured image101ais selected, the occurrence likelihood map image102bthat matches the road shape of the captured image101bis selected for the captured image101b, and the occurrence likelihood map image102cthat matches the road shape of the captured image101cis selected for the captured image101c.

By storing, in the memory3, not map data but a plurality of occurrence likelihood map images which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of a plurality of roads in the above-described manner, it is possible to reduce the amount of data stored in the memory3.

In the first embodiment, when it is determined that the location of the detected person in the captured image101is outside the occurrence area103in the occurrence likelihood map image102, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode. However, in the present disclosure, the condition for switching the vehicle drive mode is not limited to the example described above. For example, in a case where two objects of the same type which are not supposed to be detected at the same time are detected within an occurrence area, the vehicle drive mode may be switched from the automatic drive mode to the manual drive mode.

FIG. 5is a schematic diagram illustrating an example in which two objects of the same type which are not supposed to be detected at the same time are detected within an occurrence area in the present embodiment1.

InFIG. 5, an occurrence likelihood map image102is superimposed on a captured image101, and vehicles are detected by the object detector11. The occurrence likelihood map image102includes an occurrence area104adefined on a vehicle road131in a travelling direction of the vehicles, an occurrence area104bdefined on a vehicle road132extending leftward from an intersection of the vehicle road131along which the vehicles are running, and an occurrence area104cdefined on a vehicle road133extending rightward from the intersection of the vehicle road131along which the vehicles are running. The occurrence areas104a,104b, and104care areas where there is a possibility that a vehicle may appear.

The detection positions121and122of the respective vehicles are correctly detected by the object detector11. However, the detection position123is detected erroneously such that not a vehicle but a part of the vehicle road132is detected erroneously as a vehicle.

The running circumstance acquirer12recognizes a color of a traffic signal135from a captured image101taken by the camera2. As a result, the running circumstance acquirer12acquires traffic signal information indicating that the color of the traffic signal135is green in this example. The detection positions121,122, and123detected by the object detector11are all within the occurrence area104aor104b, and thus, in a normal situation, the drive mode switcher15maintains the current automatic drive mode without switching to the manual drive mode from the automatic drive mode.

However, the automatic drive mode acquired by the running circumstance acquirer12indicates that the traffic signal is in green, and thus no vehicle is supposed to enter the intersection from the vehicle road132. In view of this, when the drive mode switcher15determines that the detection positions121,122, and123of the vehicles in the captured image101are within either one of the two occurrence areas104aand104bin the occurrence likelihood map image102, and the drive mode switcher15further determines whether the traffic signal information indicates that the traffic signal is in green or not, and if it is determined that the traffic signal is in green, the drive mode switcher15switches the vehicle drive mode from the automatic drive mode to the manual drive mode.

As described above, in the case where two vehicles of the same type that are not supposed to be detected at the same time are detected in occurrence areas, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

In the present embodiment1, also in a case where two objects of different types which are not supposed to be detected at the same time are detected in occurrence areas, the vehicle drive mode may be switched from the automatic drive mode to the manual drive mode.

FIG. 6is a schematic diagram illustrating an example in which two objects of different types which are not supposed to be detected at the same time are detected within occurrence areas in the present embodiment1.

InFIG. 6, an occurrence likelihood map image102is superimposed on a captured image101, and a person and a vehicle, which are two objects of different types, are detected by the object detector11. The occurrence likelihood map image102includes an occurrence area104adefined on a vehicle road131in a travelling direction of the vehicles, an occurrence area104bdefined on a vehicle road132extending leftward from an intersection of the vehicle road131along which the vehicles are running, and an occurrence area104cdefined on a vehicle road133extending rightward from the intersection of the vehicle road131along which the vehicles are running. The occurrence areas104a,104b, and104care areas where there is a possibility that a vehicle may appear.

The occurrence likelihood map image102includes occurrence areas103defined on both sides of each of vehicle roads131,132, and133. The occurrence areas103are areas where there is a possibility that a person may appear.

The detection positions121and122of the respective vehicles are correctly detected by the object detector11. However, the detection position111is detected erroneously such that not a person but a part of the vehicle road131is detected erroneously as a person.

The occurrence area104ais an area where there is a possibility that a vehicle may appear but there is no possibility that a person may appear. Therefore, it is not supposed to detect a vehicle and a person at the same time in the occurrence area104a. In view of the above, when the drive mode switcher15determines that the detection positions121and122of the vehicles and the detection position111of the person in the captured image101are within the occurrence area104ain the occurrence likelihood map image102, then the drive mode switcher15switches the vehicle drive mode from the automatic drive mode to the manual drive mode.

As described above, in the case where two vehicles of different types that are not supposed to be detected at the same time are detected in an occurrence area, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

Second Embodiment

In the first embodiment described above, the occurrence likelihood map storage13stores an occurrence likelihood map defining an occurrence area on a two-dimensional map where there is a possibility that an object appears. In a second embodiment described below, the occurrence likelihood map storage13stores an occurrence likelihood map defining an occurrence space on a three-dimensional map where there is a possibility that an object appears.

FIG. 7is a block diagram illustrating a configuration of a vehicle control system according to a first embodiment. The vehicle control system shown inFIG. 7includes a drive control apparatus1and a camera2. The drive control apparatus1includes a memory3and a processor4. The memory3includes an occurrence likelihood map storage17.

The occurrence likelihood map storage17is an example of a map storage and stores a plurality of occurrence likelihood maps (models) defining occurrence areas for a plurality of roads having different shapes represented in a three-dimensional coordinate system. In the present embodiment2, the plurality of occurrence likelihood maps each define an occurrence space on a three-dimensional map where there is a possibility that an object appears. The object may be, for example, a person or a vehicle, and the occurrence likelihood maps each indicates individually an occurrence area where a person passes and an occurrence area where a vehicle passes. Note that the occurrence space is not a two-dimensional area but a three-dimensional space.

The plurality of occurrence likelihood maps are stored in advance in the occurrence likelihood map storage17. However, alternatively, the occurrence likelihood maps may be acquired from a server via a communication unit (not shown) and stored in the occurrence likelihood map storage17.

The occurrence spaces of the plurality of occurrence likelihood maps may be calculated based on statistic data of detections of persons or vehicles in the past. The occurrence spaces of the plurality of occurrence likelihood maps may be defined on the three-dimensional map by an operator, according to a particular rule such as the Road Traffic Law such that an occurrence space is defined in a space where a person or a vehicle is allowed to pass.

The processor4is, for example, a CPU, and executes a drive control program stored in the memory3. The processor4includes an object detector11, a running circumstance acquirer12, an occurrence likelihood map determiner22, a drive mode switcher15a vehicle controller16, and a coordinate system converter18. In the second embodiment, similar elements to those in the first embodiment are denoted by similar reference numerals or symbols, and a further description thereof is omitted.

The occurrence likelihood map determiner22determines one occurrence likelihood map corresponding to the shape of the road acquired by the running circumstance acquirer12from a plurality of occurrence likelihood maps stored in the occurrence likelihood map storage17.

When one occurrence likelihood map is determined by the occurrence likelihood map determiner22, the coordinate system converter18converts the occurrence likelihood map in the three-dimensional coordinate system to an occurrence likelihood map image in the same two-dimensional coordinate system as that in which the captured image is represented.

Next, an operation of the drive control apparatus according to the second embodiment is described below.

FIG. 8is a flow chart illustrating the operation of the drive control apparatus according to a second embodiment.

The process in step S11to step S12is similar to the process in step S1to step S2shown inFIG. 2, and thus a further description thereof is omitted.

In step S13, the occurrence likelihood map determiner22determines one occurrence likelihood map corresponding to the shape of the road acquired by the running circumstance acquirer12from a plurality of occurrence likelihood maps stored in the occurrence likelihood map storage17.

Next, in step S14, the coordinate system converter18converts the one occurrence likelihood map in the three-dimensional coordinate system determined by the occurrence likelihood map determiner22to an occurrence likelihood map image in the same two-dimensional coordinate system as that in which the captured image is represented. That is, the one occurrence likelihood map determined by the occurrence likelihood map determiner22is represented in the three-dimensional coordinate system, and the coordinate system converter18converts the one occurrence likelihood map represented in the three-dimensional coordinate system to an occurrence likelihood map image represented in the two-dimensional viewpoint-based coordinate system with reference to the viewpoint of the camera2.

The process in step S15to step S17is similar to the process in step S4to step S6shown inFIG. 2, and thus a further description thereof is omitted.

As described above, the occurrence space in which there is a possibility that an object appears is represented not in the two-dimensional coordinate system but in the three-dimensional coordinate system, and thus it is possible to more precisely define the range of the occurrence space, which makes it possible to enhance the reliability of switching the vehicle drive mode from the automatic drive mode to the manual drive mode.

Third Embodiment

In the second embodiment described above, the occurrence likelihood map image is determined depending on the running circumstance. In a third embodiment described below, an occurrence likelihood map image is determined depending on a location where an object is detected. When the occurrence likelihood map image is switched a predetermined number of times during a predetermined period of time, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

FIG. 9is a block diagram illustrating a configuration of a vehicle control system according to the third embodiment. The vehicle control system shown inFIG. 9includes a drive control apparatus1and a camera2. The drive control apparatus1includes a memory3and a processor4. The memory3includes an occurrence likelihood map storage13and an identification number storage19. In the third embodiment, similar elements to those in the first embodiment are denoted by similar reference numerals or symbols, and a further description thereof is omitted.

The occurrence likelihood map storage13stores a plurality of occurrence likelihood map images which are respectively represented in the two-dimensional coordinate system as seen from the same viewpoint as that of the captured image and which respectively define occurrence areas depending on shapes of roads. The occurrence likelihood map images are respectively assigned identification numbers.

The identification number storage19stores an identification number identifying an occurrence likelihood map image determined by the optimum occurrence likelihood map determiner20.

The processor4is, for example, a CPU, and executes a drive control program stored in the memory3. The processor4includes an object detector11, a running circumstance acquirer12, a vehicle controller16, an optimum occurrence likelihood map determiner20, and a drive mode switcher21.

The optimum occurrence likelihood map determiner20determines, depending on the location of the object detected in the captured image, one occurrence likelihood map image from the plurality of occurrence likelihood map images, and the optimum occurrence likelihood map determiner20stores, in the identification number storage19, the identification number corresponding to the determined one occurrence likelihood map image.

When the identification number stored in the identification number storage19is switched a predetermined number of times during a predetermined period of time, the drive mode switcher21switches the vehicle drive mode from the automatic drive mode to the manual drive mode.

Next, an operation of the drive control apparatus according to the third embodiment is described below.

FIG. 10is a flow chart illustrating the operation of the drive control apparatus according to the third embodiment.

The process in step S21to step S22is similar to the process in step S1to step S2shown inFIG. 2, and thus a further description thereof is omitted.

In step S23, the optimum occurrence likelihood map determiner20determines, depending on the location of the object detected in the captured image, one occurrence likelihood map image from the plurality of occurrence likelihood map images. The optimum occurrence likelihood map determiner20determines, from the plurality of occurrence likelihood map images, an optimum occurrence likelihood map image having an occurrence area matching a location where an object is detected.

Next, in step S24, the optimum occurrence likelihood map determiner20stores, in the identification number storage19, the identification number of the determined occurrence likelihood map image.

Next, in step S25, the drive mode switcher21determines whether counting is being performed as to the elapsed time since the identification number of the optimum occurrence likelihood map image is stored for the first time in the identification number storage19. In a case where it determined that the elapsed time is not being counted (NO in step S25), then in step S26, the drive mode switcher21starts counting the elapsed time since the identification number of the optimum occurrence likelihood map image is stored for the first time in the identification number storage19. After the time counting is started, the processing flow returns to step S21.

In a case where it is determined that the elapsed time is being counted (YES in step S25) then in step S27, the drive mode switcher21determines whether the elapsed time has reached a predetermined value. In a case where it is determined that the elapsed time has not reached the predetermined value (NO in step S27), the processing flow returns to step S21.

On the other hand, in a case where it is determined that the elapsed time has reached the predetermined value (YES in step S27), then in step S28, the drive mode switcher21determines whether the identification number stored in the identification number storage19has been switched a predetermined number of times during a predetermined period of time. In the identification number storage19, identification numbers of optimum occurrence likelihood map images determined during the predetermined period of time by the optimum occurrence likelihood map determiner20are stored cumulatively. Therefore, by counting the number of identification numbers stored in the identification number storage19when the predetermined period of time has expired, it is possible to determine how many times the identification number has been changed.

In a case where it is determined that the identification number has been changed the predetermined number of times (YES in step S28), then in step S29, the drive mode switcher21switches the vehicle drive mode from the automatic drive mode to the manual drive mode.

On the other hand, in a case where it is determined that the identification number has not been changed the predetermined number of times (NO in step S28), then in step S30the drive mode switcher21maintains the current automatic drive mode.

FIG. 11is a schematic diagram illustrating the process of switching the vehicle drive mode from the automatic drive mode to the manual drive mode according to the third embodiment. InFIG. 11, a horizontal axis represents time t.

As illustrated inFIG. 11, at time t0, a person included in the captured image is detected, and an occurrence likelihood map image is determined depending on the location where the object is detected in the captured image. The identification number (ID:3in the present example) of the determined optimum occurrence likelihood map image is stored in the identification number storage19, and the counting of the elapsed time since time t0is started.

Next, the determination is performed as to whether the elapsed time has reached a predetermined value T. The predetermined value of the period is, for example, 1 second. In a case where the elapsed time has not reached the predetermined value T, and the process of detecting a person in the captured image is performed again. On the other hand, in a case where it is determined that the elapsed time has reached the predetermined value T, then a further determination is performed whether the identification number has been changed the predetermined number of times. The predetermined number is, for example, two. That is, in a case where the occurrence likelihood map image is changed twice during one second, the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

In example shown inFIG. 11, an identification number (ID:3) is first stored, an identification number (ID:102) is then stored at time t1, and finally, at time t2, an identification number (ID:401) is stored. Therefore, the occurrence likelihood map image has been changed twice before the elapsed time has reached the predetermined value T. Thus, in this example, it is determined that the identification number has been changed the predetermined number of times during the predetermined period of time T, and the vehicle drive mode is switched from the automatic drive mode to the manual drive mode.

When a change in the occurrence likelihood map image occurs for a short period, it can be determined that objects are not accurately detected. In such a situation, by switching the vehicle drive mode from the automatic drive mode to the manual drive mode, it becomes possible to safely drive the vehicle by a driver in the manual drive mode instead of driving the vehicle in the automatic drive mode in the state in which objects are not detected accurately.

In the present disclosure, all or part of units, apparatuses, and elements and all or part of functional blocks illustrated in the figures may be implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), an LSI (Large Scale Integration). The LSI or the IC may be integrated on a single chip or may be realized by a combination of a plurality of chips. For example, functional blocks other than storage elements may be integrated on a signal chip. The integrated circuits called the LSI or the IC herein may be called differently depending on the integration scale, and integrated circuits called a system LSI, a VLSI (Very Large Scale Integration), or a ULSI (Ultra Large Scale Integration) may also be used in the present disclosure. Furthermore, a field programmable gate array (FPGA) capable of being programmed after the LIS is produced, and a reconfigurable logic device capable of being reconfigured in terms of internal connections or capable of being set up in terms of internal circuits segments may also be used for the same purpose.

Furthermore, functions or operations of all or part of units, apparatuses, and elements may be executed by software. In this case, the software may be stored in a non-transitory storage medium. The non-transitory storage medium may be one of or a combination of a ROM, an optical disk, a hard disk drive, or the like. When the software is executed by a processor, a function corresponding to the software is executed by the processor and a peripheral apparatus. The system or the apparatus may include one of or a combination of a non-transitory storage medium, a processor, a hardware device such as an interface, and the like.

The drive control apparatus, the drive control method, and the drive control program according to the present disclosure are capable of, depending on the object detection accuracy, switching the vehicle drive mode from the automatic drive mode in which the vehicle is automatically driven to the manual drive mode in which the vehicle is driven manually by a driver, and are useful in controlling the driving of the vehicle in the automatic drive mode or the manual drive mode.