Patent ID: 12194987

DESCRIPTION OF EMBODIMENT

Exemplary embodiments of the present invention will be described below, with reference to the drawings. A dangerous scene prediction device according to the present invention predicts a dangerous scene that may occur during vehicle driving using a learning model. Each learning model used in the present invention is a model for predicting whether each dangerous scene expected during vehicle driving occurs or the degree of occurrence of each dangerous scene, and is learned beforehand based on observed data. Examples of dangerous scenes predicted include interruption prediction, left-turn collision prediction, run-out crossing prediction, lateral intrusion prediction, and run-out crossing prediction.

Exemplary Embodiment 1

FIG.1is a block diagram depicting an example of the structure of Exemplary Embodiment 1 of a dangerous scene prediction device according to the present invention. A dangerous scene prediction device100in this exemplary embodiment includes a scene determination unit10, a learning model selection/synthesis unit20, a dangerous scene prediction unit30, and a storage unit90.

The storage unit90stores a plurality of learning models used by the below-described dangerous scene prediction unit30. The storage unit90may also store various information necessary for the operation of the dangerous scene prediction device100. The storage unit90is implemented, for example, by a magnetic disk or the like.

The scene determination unit10receives input of information acquired by various sensors for detecting the state of the vehicle. Examples of the sensors for detecting the state of the vehicle include a GPS (global positioning system) receiver that acquires position information, millimeter wave radar that detects millimeter wave information, LiDAR (laser imaging detection and ranging) that acquires the distance to an object, and a camera that captures a surrounding image.

The scene determination unit10determines the scene of the vehicle based on information obtained during vehicle driving, such as the input information of the sensors. Herein, the scene denotes the external environment surrounding the vehicle. Examples of the scene include scenes where the vehicle is running (highway, urban intersection, parking lot entrance/exit, urban straight road, etc.). The scene determined in this exemplary embodiment is not limited to being expressed so as to be interpretable by humans as mentioned above, and may be expressed, for example, by a feature vector representing the external environment. In the following, the operation of the dangerous scene prediction device in this exemplary embodiment will be described using specific scenes as an example, for ease of explanation.

The scene determination unit10determines the scene by any method. The scene determination unit10may determine the scene based on a rule, or determine the scene using a discriminant model for discriminating the likelihood of each scene. For example, the scene determination unit10may determine the current running location (e.g. highway, shopping street, etc.) based on map information given beforehand and position information acquired by GPS. The scene determination unit10may hold a scene discriminant model generated by machine learning beforehand based on acquirable sensor information, and determine the scene based on the received sensor information.

The learning model selection/synthesis unit20selects a learning model used for dangerous scene prediction from two or more learning models, depending on the determined scene. For example, a learning model to be selected for each scene may be defined beforehand according to the feature of the learning model, and the learning model selection/synthesis unit20may select a learning model corresponding to the determined scene. The learning model used in this embodiment may be in any form, and is, for example, a neural network.

FIG.2is an explanatory diagram depicting an example of learning models defined for different scenes. In the example depicted inFIG.2, “highway”, “urban intersection” or “parking lot entrance/exit”, and “urban straight road” are assumed as scenes. On the “highway”, an interruption in front of the vehicle is assumed as a dangerous scene. Hence, in the example depicted inFIG.2, the scene “highway” is associated with a learning model for performing interruption prediction. At the “urban intersection” or “parking lot entrance/exit”, a plurality of dangerous scenes (left-turn collision, run-out crossing, and lateral intrusion) are assumed. Hence, in the example depicted inFIG.2, the scene “urban intersection” or “parking lot entrance/exit” is associated with a model synthesizing learning models for performing left-turn collision prediction, run-out crossing prediction, and lateral intrusion prediction. The same applies to the scene “urban straight road”.

The learning model selection/synthesis unit20can select the learning model corresponding to the scene determined by the scene determination unit10, based on the correspondence relationship between the scene and the learning model defined as depicted inFIG.2. In the case where the determination result by the scene determination unit10includes the probability of the scene, the learning model selection/synthesis unit20may select the learning model corresponding to the scene, and associate the selected learning model with the probability of the scene.

The dangerous scene prediction unit30predicts a dangerous scene occurring during vehicle driving, using the selected learning model. Specifically, the dangerous scene prediction unit30predicts the dangerous scene occurring during vehicle driving based on object position information acquired from the foregoing sensor such as LiDAR or millimeter wave information, and outputs a dangerous scene prediction result. The dangerous scene prediction result is information indicating whether each dangerous scene occurs or the degree of occurrence of each dangerous scene, and may be binary information of 0 or 1. The dangerous scene prediction result may be probability information indicating the likelihood of the danger. Based on the dangerous scene prediction result, for example, whether the vehicle is to issue a warning against a predicted interruption or run-out is decided.

For example, in the case where the learning model is associated with the probability of the scene, the dangerous scene prediction unit30may adjust whether the dangerous scene occurs or the degree of occurrence depending on the probability.

FIG.3is an explanatory diagram depicting an example of a process of selecting a learning model and outputting a dangerous scene prediction result. The example depicted inFIG.3relates to a process of selecting a learning model defined in the example depicted inFIG.2and outputting a dangerous scene prediction result. Models M1to M3depicted inFIG.3are, for example, a neural network.

For example, in the case where the scene is determined as “urban intersection”, the learning model selection/synthesis unit20selects the model M2synthesizing a learning model m21for performing left-turn collision prediction, a learning model m22for performing run-out crossing prediction, and a learning model m23for performing lateral intrusion prediction. The dangerous scene prediction unit30then predicts each dangerous scene using the model M2, and outputs the occurrence probability (e.g. 0 to 1) of each dangerous scene of left-turn collision, run-out crossing, and lateral intrusion.

For example, in the case where the scene is determined as “highway”, the learning model selection/synthesis unit20selects the model M1including a learning model m11for performing interruption prediction. In the case where the scene is determined as “urban straight road”, the learning model selection/synthesis unit20selects the model M3including a learning model m31for performing run-out crossing prediction. The dangerous scene prediction result is the same as in the case of the model M2as mentioned above.

The scene determination unit10, the learning model selection/synthesis unit20, and the dangerous scene prediction unit30are implemented by a CPU of a computer operating according to a program (dangerous scene prediction program). For example, the program may be stored in the storage unit90in the dangerous scene prediction device100, with the CPU reading the program and, according to the program, operating as the scene determination unit10, the learning model selection/synthesis unit20, and the dangerous scene prediction unit30. The scene determination unit10, the learning model selection/synthesis unit20, and the dangerous scene prediction unit30may each be implemented by dedicated hardware.

The operation of the dangerous scene prediction device100in this exemplary embodiment will be described below.FIG.4is a flowchart depicting an example of the operation of the dangerous scene prediction device100in this exemplary embodiment. The scene determination unit10determines the scene of the vehicle, for example based on object position information acquired from sensor information obtained during vehicle driving (step S11). The learning model selection/synthesis unit20selects a learning model used for dangerous scene prediction from two or more learning models, depending on the determined scene (step S12). The dangerous scene prediction unit30predicts a dangerous scene occurring during vehicle driving, using the selected learning model (step S13).

As described above, in this exemplary embodiment, the learning model selection/synthesis unit20selects a learning model used for dangerous scene prediction from two or more learning models, depending on a scene determined based on information obtained during vehicle driving. The dangerous scene prediction unit30predicts a dangerous scene occurring during vehicle driving, using the selected learning model. Thus, the prediction accuracy for dangerous scenes expected during driving can be improved while reducing the computation load.

That is, in this exemplary embodiment, the learning model selection/synthesis unit20selects a learning model necessary for dangerous scene prediction suitable for the scene, and the dangerous scene prediction unit30predicts a dangerous scene using only the learning model. Therefore, highly accurate dangerous scene prediction can be achieved while reducing the computation amount.

Exemplary Embodiment 2

Exemplary Embodiment 2 of a dangerous scene prediction device according to the present invention will be described below. Exemplary Embodiment 1 describes a method of predicting a dangerous scene using detected object position information. Exemplary Embodiment 2 describes a method of predicting a dangerous scene using a result of recognizing a detected object.

FIG.5is a block diagram depicting an example of the structure of Exemplary Embodiment 2 of a dangerous scene prediction device according to the present invention. A dangerous scene prediction device200in this exemplary embodiment includes the scene determination unit10, the learning model selection/synthesis unit20, a dangerous scene prediction unit31, a recognition object/posture decision unit40, a learning model selection unit50, an object recognition unit60, an imaging device70, and a storage unit91. The scene determination unit10and the learning model selection/synthesis unit20are the same as those in Exemplary Embodiment 1.

The scene determination unit10, the learning model selection/synthesis unit20, and the dangerous scene prediction unit31may be implemented by a device different from the recognition object/posture decision unit40, the learning model selection unit50, and the object recognition unit60.

The storage unit91stores a plurality of learning models used by the dangerous scene prediction unit31, as with the storage unit90in Exemplary Embodiment 1. The storage unit91in this exemplary embodiment also stores a plurality of learning models (hereafter referred to as “object recognition models”) used for object recognition by the below-described the object recognition unit60. The storage unit91may store various information necessary for the operation of the dangerous scene prediction device200. The storage unit91is implemented, for example, by a magnetic disk or the like.

The recognition object/posture decision unit40decides information (hereafter referred to as “recognition object information”) about an object subjected to recognition, based on information of a learning model selected by the learning model selection/synthesis unit20. Specifically, the recognition object/posture decision unit40decides the recognition object information necessary in the case of performing dangerous scene prediction using the learning model selected by the learning model selection/synthesis unit20. The recognition object information may include not only information indicating the recognition object itself but also information indicating the part of the object subjected to recognition (hereafter referred to as “posture of object”).

The recognition object information is defined beforehand for each learning model for performing dangerous scene prediction, as an important recognition object when performing dangerous scene prediction.FIG.6is an explanatory diagram depicting an example of the relationship between learning models and recognition object information. In the example depicted inFIG.6, each prediction model for performing dangerous scene prediction is associated with an important recognition object and posture. For example, in the case of predicting vehicle interruption as a dangerous scene, information of the rear of a vehicle (passenger car, truck, and bus) can be regarded as an important recognition object and posture, as depicted in (1) inFIG.6. Hence, a model (interruption prediction model) for performing vehicle interruption prediction is associated with “rear of vehicle (passenger car, truck, and bus)” as an important recognition object and posture. The same applies to other prediction models.

The learning model selection unit50selects an object recognition model used for object recognition from two or more object recognition models, depending on the scene determined by the scene determination unit10and the recognition object information decided by the recognition object/posture decision unit40. For example, an object recognition model to be selected for each scene and recognition object information may be defined beforehand, and the learning model selection unit50may select an object recognition model corresponding to the determined scene. The object recognition model used in this embodiment may be in any form, and is, for example, a neural network.

FIG.7is an explanatory diagram depicting an example of object recognition models defined for different scenes and recognition object information. In the example depicted inFIG.7, “highway”, “urban intersection” or “parking lot entrance/exit”, and “urban straight road” are assumed as scenes. For example, suppose an object recognition model 1 has high accuracy in object recognition for the rear of a vehicle. In such a case, “highway” and the object recognition model 1 are associated with each other, and the learning model selection unit50selects the object recognition model corresponding to the scene determined by the scene determination unit10and the recognition object information. The same applies to other scenes.

The imaging device70captures an image outside the vehicle during vehicle driving. The timing at which the imaging device70captures an image may be any timing during running or stopping. The imaging device70may capture an image at predetermined time intervals, or capture an image according to an instruction from the driver or the like or a control device. The imaging device70is, for example, a vehicle-mounted camera that captures a landscape outside the vehicle.

The object recognition unit60recognizes an object in the image captured by the imaging device70, using the selected object recognition model. The method whereby the object recognition unit60recognizes the object using the learning model is widely known, and accordingly its detailed description is omitted.

The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using not only the object position information in Exemplary Embodiment 1 but also the object recognition result by the object recognition unit60in the selected learning model. The dangerous scene prediction unit31may predict the dangerous scene, without using the object recognition result (i.e. using the object recognition result as auxiliary information).

The scene determination unit10, the learning model selection/synthesis unit20, the dangerous scene prediction unit31, the recognition object/posture decision unit40, the learning model selection unit50, and the object recognition unit60are implemented by a CPU of a computer operating according to a program (dangerous scene prediction program). The scene determination unit10, the learning model selection/synthesis unit20, the dangerous scene prediction unit31, the recognition object/posture decision unit40, the learning model selection unit50, the object recognition unit60may each be implemented by dedicated hardware.

The operation of the dangerous scene prediction device200in this exemplary embodiment will be described below.FIG.8is a flowchart depicting an example of the operation of the dangerous scene prediction device200in this exemplary embodiment. The method whereby the scene determination unit10determines the scene of the vehicle and the learning model used for dangerous scene prediction is selected is the same as the process from steps S11to S12inFIG.4.

FIG.9is an explanatory diagram depicting an example of the relationship between scene determination results and selected learning models. For example, in the case where the scene determination unit10determines the scene as “highway”, the learning model selection/synthesis unit20selects the learning model (1) depicted inFIG.6. For example, in the case where the scene determination unit10determines the scene as “urban intersection” or “parking lot entrance/exit”, the learning model selection/synthesis unit20selects the learning model (2), the learning model (3) and the learning model (4) depicted inFIG.6. That is, the learning model selection/synthesis unit20selects and synthesizes the plurality of learning models. For example, in the case where the scene determination unit10determines the scene as “urban straight road”, the learning model selection/synthesis unit20selects the learning model (3) depicted inFIG.6.

The recognition object/posture decision unit40decides recognition object information based on information of the learning model selected by the learning model selection/synthesis unit20(step S21inFIG.8). The learning model selection unit50selects an object recognition model depending on the determined scene and the recognition object information (step S22). The object recognition unit60recognizes an object in an image captured during vehicle driving, using the selected object recognition model (step S23). The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using the object recognition result by the object recognition unit60in the selected learning model (step S24).

As described above, in this exemplary embodiment, the recognition object/posture decision unit40decides recognition object information based on information of a selected learning model. The learning model selection unit50selects an object recognition model depending on the determined scene and the recognition object information. The object recognition unit60recognizes an object in an image captured during vehicle driving, using the selected object recognition model. The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using the object recognition result in the selected learning model.

Thus, in addition to the effects in Exemplary Embodiment 1, the object recognition model can be selected based on the recognition object and posture necessary for the dangerous scene prediction model, so that the dangerous scene prediction accuracy can be improved.

Exemplary Embodiment 3

Exemplary Embodiment 3 of a dangerous scene prediction device according to the present invention will be described below. This exemplary embodiment describes a method of selecting a learning model to be used, based on the computation amount of the dangerous scene prediction process and the object recognition process.

FIG.10is a block diagram depicting an example of the structure of Exemplary Embodiment 3 of a dangerous scene prediction device according to the present invention. A dangerous scene prediction device300in this exemplary embodiment includes the scene determination unit10, a learning model selection/synthesis unit21, the dangerous scene prediction unit31, a model adjustment unit51, an object recognition unit61, the imaging device70, and a storage unit92. The scene determination unit10, the dangerous scene prediction unit31, and the imaging device70are the same as those in Exemplary Embodiment 2.

The storage unit92stores a plurality of learning models used by the dangerous scene prediction unit31, as with the storage unit90in Exemplary Embodiment 1. The storage unit92in this exemplary embodiment also stores a plurality of neural networks (hereafter referred to as “neural NW”) used for object recognition by the below-described the object recognition unit61, and learning models corresponding to the neural NWs. The storage unit92may store various information necessary for the operation of the dangerous scene prediction device300. The storage unit92is implemented, for example, by a magnetic disk or the like.

Each neural NW used in this exemplary embodiment is a model for performing inference (type determination) of object recognition by machine learning, and performs inference (type determination) using weights and biases representing the strength of connection between nodes indicated in the corresponding learning model. Since the object recognition accuracy and the computation amount are different between different neural NWs, a plurality of types of neural NWs and corresponding learning models are prepared beforehand depending on the computation amount in this exemplary embodiment.

The model adjustment unit51decides a neural NW used for object recognition by the below-described object recognition unit61and a learning model used for dangerous scene prediction by the dangerous scene prediction unit31, depending on the scene determined by the scene determination unit10. Here, the model adjustment unit51decides the neural NW and the learning model so that the computation amount required for the object recognition process and the dangerous scene prediction process will be less than or equal to the computation amount that is allowed (hereafter referred to as “allowable computation amount”).

In the case of object recognition, the change of the computation amount is greater when changing the neural NW than when changing the learning model while fixing the neural NW. Accordingly, in this exemplary embodiment, the model adjustment unit51controls not the learning model but the neural NW.

First, the model adjustment unit51decides the process is to be prioritized. Specifically, the model adjustment unit51decides the relative priority of the object recognition process and the dangerous scene prediction process depending on the determined scene. The relative priority may be set beforehand depending on the scene.

For example, in the case of the scene “highway”, it is considered that the danger prediction performance is demanded more than the object recognition performance. Hence, for the scene “highway”, the priority of the dangerous scene prediction process is set higher than the priority of the object recognition process. For example, in the case of the scene “straight road on shopping street”, there are many small objects such as pedestrians, and thus it is considered that the object recognition performance is demanded more than the danger prediction performance. Hence, for the scene “straight road on shopping street”, the priority of the object recognition process is set higher than the priority of the dangerous scene prediction process. The model adjustment unit51decides the process to be prioritized, based on such setting.

The model adjustment unit51then decides a neural NW and a learning model so that the computation amount required for the two processes will be less than or equal to the allowable computation amount, based on the decided priority. In this exemplary embodiment, the computation amount when performing object recognition using each neural NW and the computation amount when performing dangerous scene prediction using each learning model are measured and stored in, for example, the storage unit92beforehand. The model adjustment unit51decides the neural NW and the learning model so that the computation amount required for the two processes will be less than or equal to the allowable computation amount, based on the computation amount measured beforehand. For example, the computation amount is defined as an index value corresponding to the number of calculation steps such as multiplication and addition in digital signal processing.

To describe the process of deciding the neural NW and the learning model based on the computation amount, computation amount levels are defined here. Each computation amount level is an index value defined according to the computation amount. In this example, the minimum level is 1, and the maximum level is 4.FIG.11is an explanatory diagram depicting an example of a process of deciding a neural NW and a learning model according to the allowable computation amount.

In the example depicted inFIG.11, a process corresponding to a scene in which the priority of the dangerous scene prediction process is set higher than the priority of the object recognition process is referred to as “dangerous scene prediction priority mode”, and a process corresponding to a scene in which the priority of the object recognition process is set higher than the priority of the dangerous scene prediction process is referred to as “object recognition priority mode”. A process intermediate between “dangerous scene prediction priority mode” and “object recognition priority mode” is referred to as “intermediate mode”.

In the example depicted inFIG.11, a graph is assumed in which the horizontal axis represents the computation amount when using a learning model generated as a dangerous scene prediction synthesis model and the vertical axis represents the computation amount when using a neural NW, where 1 cell of the grid in each mode represents 1 computation amount. In the example depicted inFIG.11, the allowable computation amount is 4 cells.

For example, in the “dangerous scene prediction priority mode”, the priority of the dangerous scene prediction process is higher than the priority of the object recognition process, and accordingly the model adjustment unit51decides the computation amount level of the learning model to be high and the computation amount level of the neural NW to be low. For example, in the case where the computation amount level of the learning model is decided to be 4, the computation amount level of the neural NW is decided to be 1 based on the allowable computation amount (4 cells).

In the “object recognition priority mode”, the priority of the object recognition process is higher than the priority of the dangerous scene prediction process, and accordingly the model adjustment unit51decides the computation amount level of the learning model to be low and the computation amount level of the neural NW to be high. For example, in the case where the computation amount level of the neural NW is decided to be 4, the computation amount level of the learning model is decided to be 1 based on the allowable computation amount (4 cells).

In the “intermediate mode”, the priority of the object recognition process and the priority of the dangerous scene prediction process are equal, and accordingly the model adjustment unit51decides the computation amount level of the learning model and the computation amount level of the neural NW to be equal. For example, in the case where the computation amount level of the neural NW is decided to be 2, the computation amount level of the learning model is decided to be 2 based on the allowable computation amount (4 cells).

The object recognition unit61recognizes an object in an image captured during vehicle driving, using the selected neural NW. Since the method of recognizing an image using a neural NW is widely known, its detailed description is omitted.

The learning model selection/synthesis unit21selects a learning model used for dangerous scene prediction, based on the selected learning model. The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using the object recognition result by the object recognition unit61in the selected learning model.

The scene determination unit10, the learning model selection/synthesis unit21, the dangerous scene prediction unit31, the model adjustment unit51, and the object recognition unit61are implemented by a CPU of a computer operating according to a program (dangerous scene prediction program). The scene determination unit10, the learning model selection/synthesis unit21, the dangerous scene prediction unit31, the model adjustment unit51, the object recognition unit61may each be implemented by dedicated hardware.

The operation of the dangerous scene prediction device300in this exemplary embodiment will be described below.FIG.12is a flowchart depicting an example of the operation of the dangerous scene prediction device300in this exemplary embodiment. The flowchart inFIG.12depicts a process in the case where a learning model for performing dangerous scene prediction is prioritized. The process by which the scene determination unit10determines the scene of the vehicle is the same as the process in step S11inFIG.4.

The model adjustment unit51first decides a learning model for performing dangerous scene prediction (step S31). The model adjustment unit51decides an object recognition neural NW, based on the computation amount (computation amount level) required for the dangerous scene prediction process using the decided learning model and the allowable computation amount set beforehand (step S32).

The object recognition unit61recognizes an object in an image captured during vehicle driving, using the selected neural NW (step S33). The learning model selection/synthesis unit21selects a learning model used for dangerous scene prediction based on the selected learning model (step S34). The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using the object recognition result by the object recognition unit61in the selected learning model (step S35).

FIG.13is a flowchart depicting another example of the operation of the dangerous scene prediction device300in this exemplary embodiment. The flowchart inFIG.13depicts a process in the case where a neural NW used for object recognition is prioritized. The process by which the scene determination unit10determines the scene of the vehicle is the same as the process in step S11inFIG.12.

The model adjustment unit51first decides a neural NW used for object recognition (step S36). The model adjustment unit51decides a learning model for performing dangerous scene prediction, based on the computation amount (computation amount level) required for the object recognition process using the decided neural NW and the allowable computation amount set beforehand (step S37). The subsequent process until a dangerous scene is predicted is the same as the process in steps S33to S35inFIG.12.

As described above, in this exemplary embodiment, the model adjustment unit51decides a neural NW and a learning model so that the computation amount required for the object recognition process and the dangerous scene prediction process will be less than or equal to the allowable computation amount, depending on the determined scene. The object recognition unit61recognizes an object in an image captured during vehicle driving, using the selected neural NW. The learning model selection/synthesis unit21selects a learning model used for dangerous scene prediction, based on the selected learning model. The dangerous scene prediction unit31predicts a dangerous scene occurring during vehicle driving, using the object recognition result by the object recognition unit61in the selected learning model.

Thus, in addition to the effects in Exemplary Embodiment 1, dangerous scene prediction can be performed with high accuracy within limited resources by taking into account the computation amount of the neural NW for performing object recognition and the learning model for performing dangerous scene prediction.

Exemplary Embodiment 4

Exemplary Embodiment 4 of a dangerous scene prediction device according to the present invention will be described below. Exemplary Embodiment 3 describes the case where the allowable computation amount is set beforehand. This exemplary embodiment describes a method of deciding the allowable computation amount based on the scene determination result.

FIG.14is a block diagram depicting an example of the structure of Exemplary Embodiment 4 of a dangerous scene prediction device according to the present invention. A dangerous scene prediction device400in this exemplary embodiment includes the scene determination unit10, the learning model selection/synthesis unit21, the dangerous scene prediction unit31, the model adjustment unit51, an allowable computation amount decision unit52, the object recognition unit61, the imaging device70, and the storage unit92. That is, the dangerous scene prediction device400in this exemplary embodiment differs from the dangerous scene prediction device300in Exemplary Embodiment 3 in that the allowable computation amount decision unit52is further included. The other structure is the same as that in Exemplary Embodiment 3.

The allowable computation amount decision unit52decides an allowable computation amount depending on the scene determined by the scene determination unit10. Specifically, the allowable computation amount decision unit52decides to increase the allowable computation amount in the case where the allowable time of dangerous scene prediction (hereafter referred to as “danger prediction time”) for the determined scene is long, and decrease the allowable computation amount in the case where the danger prediction time for the scene is short. For example, the allowable computation amount is set beforehand depending on the scene, and the allowable computation amount decision unit52specifies the allowable computation amount corresponding to the determined scene.

FIG.15is an explanatory diagram depicting an example of the danger prediction time. For example, in the case where the dangerous scene is “collision”, the danger prediction time corresponds to the time from the issuance of a warning of danger prediction to the collision.

The model adjustment unit51decides a neural NW and a learning model so that the computation amount required for each process will be less than or equal to the allowable computation amount decided by the allowable computation amount decision unit52. The method by which the model adjustment unit51decides the model based on the computation amount is the same as that in Exemplary Embodiment 3.

FIG.16is an explanatory diagram depicting another example of a process of deciding a neural NW and a learning model according to the allowable computation amount.FIG.16depicts the case where the process is performed in two types of modes (“low-speed computation/high accuracy mode” and “high-speed computation/low accuracy mode”) depending on the length of the danger prediction time. The “low-speed computation/high accuracy mode” is an example when the danger prediction time is long (e.g. from 3 sec to less than 10 sec), and “high-speed computation/low accuracy mode” is an example when the danger prediction time is short (e.g. within 3 sec).

The graph depicted inFIG.16is a graph in which the horizontal axis represents the computation amount when using a learning model generated as a dangerous scene prediction synthesis model and the vertical axis represents the computation amount when using a neural NW, where 1 cell of the grid represents 1 computation amount, as in the graph depicted inFIG.11. More computation can be performed for a scene with a long danger prediction time. Accordingly, in the example depicted inFIG.16, the allowable computation amount for the “low-speed computation/high accuracy mode” is set to 16 cells. For a scene with a short danger prediction time, on the other hand, the computation time is more limited. Accordingly, in the example depicted inFIG.16, the allowable computation amount for the “high-speed computation/low accuracy mode” is set to 1 cell.

In this case, in the “low-speed computation/high accuracy mode”, the computation amount level of each of the neural NW and the learning model can be set to 4. In the “high-speed computation/low accuracy mode”, the computation amount level of each of the neural NW and the learning model can be set to only 1.

The scene determination unit10, the learning model selection/synthesis unit21, the dangerous scene prediction unit31, the model adjustment unit51, the allowable computation amount decision unit52, and the object recognition unit61are implemented by a CPU of a computer operating according to a program (dangerous scene prediction program). The scene determination unit10, the learning model selection/synthesis unit21, the dangerous scene prediction unit31, the model adjustment unit51, the allowable computation amount decision unit52, and the object recognition unit61may each be implemented by dedicated hardware.

The operation of the dangerous scene prediction device400in this exemplary embodiment will be described below.FIG.17is a flowchart depicting an example of the operation of the dangerous scene prediction device400in this exemplary embodiment. The flowchart inFIG.17depicts a process in the case where a learning model for performing dangerous scene prediction is prioritized. The process by which the scene determination unit10determines the scene of the vehicle is the same as the process in step S11inFIG.12.

The allowable computation amount decision unit52decides an allowable computation amount depending on a determined scene (step S41). The model adjustment unit51then decides a learning model for performing dangerous scene prediction (step S31), and decides an object recognition neural NW based on the computation amount (computation amount level) required for the dangerous scene prediction process using the decided learning model and the decided allowable computation amount (step S32). The subsequent process until a dangerous scene is predicted is the same as the process in steps S33to S35inFIG.12.

FIG.18is a flowchart depicting another example of the operation of the dangerous scene prediction device400in this exemplary embodiment. The flowchart inFIG.18depicts a process in the case where a neural NW used for object recognition is prioritized. The process by which the scene determination unit10determines the scene of the vehicle is the same as the process in step S11inFIG.17.

The allowable computation amount decision unit52decides an allowable computation amount depending on a determined scene (step S41). The model adjustment unit51then decides a neural NW used for object recognition (step S36), and decides a learning model for performing dangerous scene prediction based on the computation amount (computation amount level) required for the object recognition process using the decided neural NW and the decided allowable computation amount (step S37). The subsequent process until a dangerous scene is predicted is the same as the process in steps S33to S35inFIG.17.

As described above, in this exemplary embodiment, the allowable computation amount decision unit52decides an allowable computation amount depending on the allowable time of dangerous scene prediction for the determined scene. The model adjustment unit51decides a neural NW and a learning model so that the computation amount required for the process in object recognition and the process of dangerous scene prediction will be less than or equal to the decided allowable computation amount. Thus, in addition to the effects in Exemplary Embodiment 3, the processing speed and the recognition and prediction accuracy in object recognition and dangerous scene prediction can be optimized depending on the scene determination result.

A specific example according to the present invention will be described below.FIG.19is a schematic block diagram depicting the structure of a computer according to at least one of the foregoing exemplary embodiments. A computer1000includes a processor1001, a main storage device1002, an auxiliary storage device1003, an interface1004, an output device1005, and a sensor1006. The computer1000is, for example, a vehicle-mounted computer mounted in a vehicle, and controls the vehicle according to the dangerous scene prediction result. The computer1000can therefore be regarded as a vehicle control device. In this case, the processor1001operates as a vehicle control unit.

The foregoing dangerous scene prediction device is implemented in the computer1000, and receives input of various information detected by the sensor1006(e.g. GPS receiver, millimeter wave radar, LiDAR, camera, etc.). The operation of each processing unit described above is stored in the auxiliary storage device1003in the form of a program (dangerous scene prediction program). The processor1001reads the program from the auxiliary storage device1003, expands the program in the main storage device1002, and executes the above-described process according to the program.

In at least one exemplary embodiment, the auxiliary storage device1003is an example of a non-transitory tangible medium. Examples of the non-transitory tangible medium include a magnetic disk, magneto-optical disk, CD-ROM (compact disc read-only memory), DVD-ROM (read-only memory), and semiconductor memory connected via the interface1004. In the case where the program is distributed to the computer1000through a communication line, the computer1000to which the program has been distributed may expand the program in the main storage device1002and execute the above-described process.

The program may realize part of the above-described functions. The program may be a differential file (differential program) that realizes the above-described functions in combination with another program already stored in the auxiliary storage device1003.

The output device1005notifies the driver whether there is a danger, based on the processing result by the computer, i.e. the dangerous scene prediction result by the dangerous scene prediction device. For example, the output device1005notifies the driver of the occurrence of a danger, by sound or by display on a display device (not depicted). As a result of providing such information, the driver can be notified of the danger.

The output device1005may control the vehicle by notifying a control device (not depicted) of the vehicle of a predicted dangerous scene. By outputting such information, it is possible to automatically control the vehicle depending on the dangerous scene.

An overview of the present invention will be described below.FIG.20is a block diagram depicting an overview of a dangerous scene prediction device according to the present invention. The dangerous scene prediction device according to the present invention is a dangerous scene prediction device80(e.g. dangerous scene prediction device100to400) for predicting a dangerous scene occurring during driving of a vehicle, the dangerous scene prediction device including: a learning model selection/synthesis unit81(e.g. learning model selection/synthesis unit20) for selecting, from two or more learning models, a learning model used for predicting the dangerous scene, depending on a scene determined based on information (e.g. position information, millimeter wave information, LiDAR information, camera information) obtained during the driving of the vehicle; and a dangerous scene prediction unit82(e.g. dangerous scene prediction unit30) for predicting the dangerous scene occurring during the driving of the vehicle, using the selected learning model.

With such a structure, the prediction accuracy for dangerous scenes expected during driving can be improved while reducing the computation load.

The dangerous scene prediction device80(e.g. dangerous scene prediction device200) may include: a recognition object information decision unit (e.g. recognition object/posture decision unit40) for deciding recognition object information that is information about an object subjected to recognition, based on information of the learning model selected by the learning model selection/synthesis unit81; a learning model selection unit (e.g. learning model selection unit50) for selecting, from two or more learning models, an object recognition model that is a learning model used for object recognition, depending on the scene determined based on the information obtained during the driving of the vehicle and the recognition object information; and an object recognition unit (e.g. object recognition unit60) for recognizing an object in an image captured during the driving of the vehicle, using the selected object recognition model. The dangerous scene prediction unit82may predict the dangerous scene occurring during the driving of the vehicle, using an object recognition result by the object recognition unit in the selected learning model. With such a structure, the object recognition model can be selected based on the recognition object and posture necessary for the dangerous scene prediction model, so that the dangerous scene prediction accuracy can be improved.

The dangerous scene prediction device80(e.g. dangerous scene prediction device300) may include: a model adjustment unit (e.g. model adjustment unit51) for deciding a neural network used for object recognition and a learning model used for prediction of the dangerous scene, depending on the determined scene; and an object recognition unit (e.g. object recognition unit61) for recognizing an object in an image captured during the driving of the vehicle, using the selected neural network. The model adjustment unit may decide the neural network and the learning model so that a computation amount required for a process in the object recognition and a process of the prediction of the dangerous scene will be less than or equal to an allowable computation amount that is a computation amount allowed. The learning model selection/synthesis unit81may select the learning model used for predicting the dangerous scene, based on the selected learning model, and the dangerous scene prediction unit82may predict the dangerous scene occurring during the driving of the vehicle, using an object recognition result by the object recognition unit in the selected learning model. By taking into account the computation amount of the neural NW for performing object recognition and the learning model for performing dangerous scene prediction in this way, dangerous scene prediction can be performed with high accuracy within limited resources.

Specifically, the model adjustment unit may decide, depending on the determined scene, a relative priority of the process in the object recognition and the process of the prediction of the dangerous scene, and decides, based on the decided priority, the neural network and the learning model so that the computation amount required for the process in the object recognition and the process of the prediction of the dangerous scene will be less than or equal to the allowable computation amount.

The dangerous scene prediction device80(e.g. dangerous scene prediction device400) may include an allowable computation amount decision unit (e.g. allowable computation amount decision unit52) for deciding the allowable computation amount, depending on an allowable time of the prediction of the dangerous scene for the determined scene. The model adjustment unit may decide the neural network and the learning model so that the computation amount required for the process in the object recognition and the process of the prediction of the dangerous scene will be less than or equal to the allowable computation amount. With such a structure, the processing speed and the recognition and prediction accuracy in object recognition and dangerous scene prediction can be optimized depending on the scene determination result.

The dangerous scene prediction device80may include a scene determination unit for determining the scene of the vehicle, based on the information obtained during the driving of the vehicle.

Although the present invention has been described with reference to the exemplary embodiments and examples, the present invention is not limited to the foregoing exemplary embodiments and examples. Various changes understandable by those skilled in the art can be made to the structures and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2018-212890 filed on Nov. 13, 2018, the disclosure of which is incorporated herein in its entirety.

REFERENCE SIGNS LIST

10scene determination unit20,21learning model selection/synthesis unit30,31dangerous scene prediction unit40recognition object/posture decision unit50learning model selection unit51model adjustment unit52allowable computation amount decision unit60,61object recognition unit70imaging device90,91,92storage unit100,200,300,400dangerous scene prediction device