Driving assistance method, driving assistance system, and server

A driving assistance method of assisting driving of a vehicle, includes, by a computer, acquiring biological information of a driver who is driving the vehicle, acquiring environment information of the driver, acquiring operation information of the driver, generating biological index data from the biological information, generating integrated information by aligning and combining time series of the biological index data, the environment information, and the operation information, calculating accident risk information after a predetermined time, by inputting the integrated information to an accident risk prediction model set in advance, and calculating factor information of the accident risk information by inputting the accident risk information, the integrated information, and accident risk judge information set in advance to a factor calculation model set in advance.

CLAIM OF PRIORITY

The present application claims priority from Japanese patent application JP 2022-049414 filed on Mar. 25, 2022, the content of which is hereby incorporated by reference into this application.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present invention claims priority under 35 U.S.C. § 119 to Japanese Patent Application No. 2022-049414 filed on Mar. 25, 2022, the entire content of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a driving assistance method, a driving assistance system, and a server that predict a risk of a traffic accident and assist driving of transportation.

2. Description of the Related Art

In recent years, quantitative evaluation of a biological state has been performed in order to prevent an occurrence of an accident caused by health of a driver in a transportation operation such as a truck or a bus. For example, in JP 2021-37033 A, evaluation of an autonomic nervous function based on measurement of heartbeat interval data by heart rate sensors of various forms that easily perform measurement among biological states has been known.

JP 2021-37033 A discloses estimation of a psychological state of a driver from biological data or the like, generation of psychological data regarding driving of the driver, and estimation of suitability and unsuitability of the state of the driver.

JP 2021-196625 A discloses estimation of an autonomic nervous function index of a driver from heartbeat interval data of biological data and prediction of an accident risk after a predetermined time based on the autonomic nervous function index.

SUMMARY OF THE INVENTION

In order to predict the risk of an accident of a driver on operation, it is necessary to predict an accident risk in the near future in real time. The biological state of the driver on operation changes sequentially, and the factors of the accident risk include environment information, operation information (business information), and the like in addition to biological information. The environment information such as a traveling state and the operation information such as the work contents are different for each driver.

However, in the conventional example, there is a problem that it is difficult to predict the accident risk of each individual driver with high accuracy by accurately reflecting the environment information and the operation information to the biological information. In addition, in the conventional example, a warning is output when the accident risk increases, but it is not possible to obtain the understanding of the driver only by simply presenting the warning.

Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to predict an accident risk with high accuracy and to present information that can be understood by a driver.

According to the present invention, there is provided a driving assistance method in which a computer including a processor and a memory assists driving of a vehicle. The driving assistance method includes, by the computer, acquiring biological information of a driver who is driving the vehicle, acquiring environment information of the driver, acquiring operation information of the driver, generating biological index data from the biological information, generating integrated information by aligning and combining time series of the biological index data, the environment information, and the operation information, calculating accident risk information after a predetermined time, by inputting the integrated information to an accident risk prediction model set in advance, and calculating factor information of the accident risk information by inputting the accident risk information, the integrated information, and accident risk judge information set in advance to a factor calculation model set in advance.

Thus, according to the present invention, it is possible to present information that does not make the driver feel uncomfortable, by not only predicting accident risk information after a predetermined time, but also outputting factor information of an accident risk.

Details of at least one embodiment of the subject matter disclosed herein are set forth in the accompanying drawings and the following description. Other features, aspects, and effects of the disclosed subject matter will be apparent from the following disclosure, drawings, and claims.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

System Configuration

FIG.1is a block diagram illustrating an embodiment of the present invention and illustrating an example of a configuration of a driving assistance system. According to the present embodiment, a driving assistance system includes a driving assistance server1. The driving assistance server1collects driving information of one or more vehicles8, biological information of a driver of the vehicle8, environment information of the vehicle8or the driver, and operation information of the driver via a network19, and predicts a risk of a traffic accident of the driver (referred to as an accident risk below). When a prediction value of an accident risk exceeds a threshold value, the driving assistance server1notifies the driver. Note that the operation information can include business information, transportation information, delivering information, and the like.

A driving information collection device10that acquires the driving information of the vehicle8, a biological information collection device60that acquires the biological information of the driver, an environment information collection device70that acquires information regarding an environment of the driver who drives the vehicle8, an operation information collection device80that acquires information regarding operation of the driver, and a prediction result display terminal90that outputs a notification from the driving assistance server1are connected to the network19and can communicate with the driving assistance server1.

The driving information collection device10collects types of information from a global navigation satellite system (GNSS)11that detects position information of the vehicle8, an inter-vehicle distance sensor12that detects a distance from a preceding vehicle, a speed sensor13that detects a speed of the vehicle8, an acceleration sensor14that detects a movement of the vehicle8, and a camera15that picks up an image of the surroundings of the vehicle8. The driving information collection device10transmits the collected types of information to the driving assistance server1.

The driving information collection device10is not limited to the above sensors, and a distance measurement sensor that detects an object and a distance around the vehicle8, a steering angle sensor that detects a driving operation, or the like can be used. In addition, the driving information collection device10can be provided with a driver ID reading device (not illustrated) that reads a medium recording the identifier of the driver in order to identify the driver.

The biological information collection device60includes sensors, that is, a heart rate meter61that detects heart rate data, a body thermometer62that detects the body temperature of the driver, and a sphygmomanometer63that detects the blood pressure of the driver. As the biological information collection device60, a sensing device attached to the inside of the vehicle8, such as a steering wheel, a seat, or a seat belt, an image recognition system that picks up an image of the expression or behavior of the driver and analyzes the image, or the like can be used in addition to a wearable device that can be worn by the driver.

The sensors of the biological information collection device60are not limited to the above description, and a sensor that detects the amount of perspiration, body temperature, blinking, eye movement, brain waves, or the like can be adopted. In addition, the biological information collection device60can set an identifier for identifying the driver and add the identifier to various types of sensing data.

The environment information collection device70includes an air thermometer71and a barometer72. The environment information collection device70may be attached to the vehicle8or may be worn by the driver similarly to the biological information collection device60.

The operation information collection device80includes an operation content input unit81provided for inputting the operation content of the driver. The driver can input, from the operation content input unit81, information such as the type of operation, the start and end of operation, a break, and the content of delivery operation. The operation information collection device80may be mounted on the vehicle8, may be a portable terminal possessed by a driver, or may be a terminal that is installed in an office or the like and can be remotely operated.

The prediction result display terminal90includes a factor label input unit91and an output unit92. The factor label input unit91receives a factor label selected or input by the driver as to a factor that has caused the warning of an accident risk, and transmits the factor label to the driving assistance server1. The output unit92includes a display device or a speaker, and outputs a warning of an accident risk or a notice of calling attention, which has been transmitted from the driving assistance server1. The factor label input unit91can receive an input of a voice in addition to text data.

The prediction result display terminal90may be a portable terminal possessed by the driver, a car navigation device mounted on the vehicle8, or a computer installed in an office or the like.

The driving assistance server1is a computer including a processor2, a memory3, an auxiliary storage device4, a communication interface5, an input device6, and an output device7. The memory3loads, as programs, the respective functional units of a time-series biological-environment-operation information generation unit (or integrated information generation unit)31, an accident risk prediction model training unit32, an accident risk prediction unit33, an accident risk factor calculation unit34, and a prediction result presentation unit35. Each of the programs is executed by the processor2. Details of each functional unit will be described later.

The processor2executes processing in accordance with a program of each functional unit to run as the functional unit that provides a predetermined function. For example, the processor2executes an accident risk prediction program to function as the accident risk prediction unit33. The same applies to other programs. Further, the processor2also runs as a functional unit that provides each function in a plurality of pieces of processing executed by the respective programs. The computer and a computer system are a device and a system including such functional units.

The auxiliary storage device (storage device)4stores data used by each functional unit described above. The auxiliary storage device4stores biological information41, environment information42, operation information43, driving information44, accident risk information45, accident risk factor information46, time-series biological-environment-operation information47, an accident risk prediction model48, an accident risk factor calculation model49, accident risk prediction model training data50, a presentation content dictionary53, and an accident risk factor label54.

The accident risk prediction model training data50includes past time-series biological-environment-operation information51and past accident risk judge information52. Details of the data will be described later.

The input device6includes a mouse, a keyboard, a touch panel, or the like. The output device7includes a display, a speaker, and the like. The communication interface5is connected to the network19and communicates with the vehicle8, the biological information collection device60, the environment information collection device70, the operation information collection device80, and the prediction result display terminal90.

Software Configuration

The time-series biological-environment-operation information generation unit31acquires driving information from the vehicle8and stores the acquired driving information, acquires biological information from the biological information collection device60and stores the acquired biological information as the biological information41in the auxiliary storage device4, acquires environment information from the environment information collection device70and stores the acquired environment information as the environment information42in the auxiliary storage device4, and acquires operation information from the operation information collection device80and stores the acquired operation information as the operation information43in the auxiliary storage device4.

The time-series biological-environment-operation information generation unit31calculates an R wave interval (RRI=R-R Interval) of the heart rate data from data (referred to as heart rate data below) of the heart rate meter61in the biological information41, and calculates an autonomic nervous function (ANF) index from the RRI data (or the heartbeat interval data). Then, the time-series biological-environment-operation information generation unit31stores the calculated R wave interval and the calculated index as the biological information41in the auxiliary storage device4.

Furthermore, as will be described later, the time-series biological-environment-operation information generation unit31combines the granularity (measurement interval) of the biological information41, the granularity (measurement interval) of the environment information42, the granularity (collection interval) of the operation information43, and the granularity (measurement interval) of the driving information44with the granularity (calculation interval or analysis window width) of the autonomic nervous function index to generate the time-series biological-environment-operation information47.

The accident risk prediction model training unit32performs learning of the accident risk prediction model48being a machine learning model by using the accident risk prediction model training data50collected in advance, and generates or updates the accident risk prediction model48.

Using the learned accident risk prediction model48, the accident risk prediction unit33receives, as an input, the time-series biological-environment-operation information47collected from the vehicle8and the driver, and outputs accident risk information45indicating an accident occurrence probability of the driver after a predetermined time.

Using the accident risk factor calculation model49of a preset machine learning model, the accident risk factor calculation unit34receives, as an input, the time-series biological-environment-operation information47and the accident risk information45, and calculates a highly relevant factor as the accident risk factor information46.

When the calculated accident risk information45satisfies a predetermined condition (when the accident occurrence probability exceeds a preset threshold value Th1), the prediction result presentation unit35acquires the accident risk factor information46corresponding to the accident risk information45, and acquires a message (or warning) corresponding to the accident risk factor information46from the presentation content dictionary53set in advance. Then, the prediction result presentation unit35transmits the message to the prediction result display terminal90.

As will be described later, the prediction result presentation unit35receives a label input by the driver with the factor label input unit91in response to the transmitted warning, and stores the received label in the accident risk factor label54.

In the present embodiment, a case where the biological information measured from the driver of the vehicle8is targeted is exemplified, but the target is not limited to the driver who operates the vehicle8. For example, a person who operates a moving object such as an airplane or a train may be targeted.

Outline of Processing

FIG.2is a flowchart illustrating an outline of processing performed by the driving assistance system. First, the accident risk prediction model training unit32trains and updates an accident risk prediction model48by using accident risk prediction model training data50set in advance.

In the driving assistance system in the present embodiment, a machine learning model for estimating an accident risk from past driving information and past biological information is generated in advance as the accident risk prediction model48, and the machine learning model is trained in a manner that the accident risk prediction model is learned by inputting the accident risk prediction model training data50to the accident risk prediction model48(S1).

Generation or training of the accident risk prediction model48may be performed in a similar manner to that in JP 2021-196625 A. For example, a definition model for estimating an accident risk from a traveling state of the vehicle8is generated by using past traveling data and past risk occurrence data as an input. Then, a probability of an occurrence of the accident risk is generated as accident risk estimation data, by inputting the traveling data collected in the past to the definition model. Then, the machine learning model of calculating biological index data of the driver from the past biological information, receiving, as an input, the accident risk estimation data and the biological index data, and outputting an accident risk (probability) after a predetermined time based on the biological index data of the traveling vehicle8can be generated as the accident risk prediction model.

As the biological index data in the present embodiment, for example, power spectral density (to be described later) calculated from the heart rate data of the driver, an autonomic nervous index (to be described later) based on an NN interval (a difference between intervals of R waves and R waves) calculated from a time-domain analysis, or the like can be used. In addition, a result of the analysis and calculation from the autonomic nervous index or the like may be used.

The accident risk prediction model training data50for training the accident risk prediction model48includes past time-series biological-environment-operation information51and past accident risk judge information52. The past time-series biological-environment-operation information51is data obtained by aligning and combining biological information of the driver collected in the past, and environment information and operation information (driving information) when the biological information has been collected, with the same time-series granularity as described later.

The accident risk prediction model training unit32may add the newly collected time-series biological-environment-operation information47to the past time-series biological-environment-operation information51. In addition, when the accident risk factor label54is updated, the accident risk prediction model training unit32can perform feedback of the updated accident risk factor label54to the accident risk judge information52.

As will be described later, the accident risk judge information52is data obtained by collecting incidents such as an accident or a near miss occurring in the past, and has the same time series as the time series of the past time-series biological-environment-operation information51.

After training the accident risk prediction model48with the past accident risk prediction model training data50, the time-series biological-environment-operation information generation unit31acquires biological information41, environment information42, operation information43, and driving information44(S2).

The time-series biological-environment-operation information generation unit31performs predetermined pre-processing on the biological information41of the driver of the vehicle8, the environment information42, the operation information43, and the driving information44corresponding to the time series of the biological information41to generate the time-series biological-environment-operation information47(S3).

The time-series biological-environment-operation information generation unit31excludes or interpolates a missing section of the heart rate data (RRI data), as the pre-processing of the biological information41. When the length of the missing section exceeds a predetermined threshold value Thf, the time-series biological-environment-operation information generation unit31can exclude the heart rate data in this section. When the length of the missing section is equal to or smaller than the predetermined threshold value Thf, the time-series biological-environment-operation information generation unit31can perform the interpolation processing. Then, the time-series biological-environment-operation information generation unit31calculates an autonomic nervous function index (ANF information) from the pre-processed heart rate data as described later.

The time-series biological-environment-operation information generation unit31also performs missing section exclusion or interpolation processing on the environment information42, the operation information43, and the driving information44, similarly to the biological information41, to generate the respective types of pre-processed information.

Then, as will be described later, the time-series biological-environment-operation information generation unit31combines the pre-processed environment information42, the pre-processed operation information43, and the pre-processed driving information44corresponding to the time series of the ANF information to generate the time-series biological-environment-operation information47.

Then, the accident risk prediction unit33calculates accident risk information45by inputting the generated time-series biological-environment-operation information47and the past accident risk judge information52to the trained accident risk prediction model48(S4).

Then, the accident risk factor calculation unit34calculates accident risk factor information46by inputting the accident risk information45calculated by the accident risk prediction unit33and the time-series biological-environment-operation information47to an accident risk factor calculation model49set in advance (S5). The accident risk factor information46is an estimation result of a factor of the accident risk information45predicted by the accident risk prediction model48.

Then, when the prediction result (probability) calculated by the accident risk prediction unit33exceeds a predetermined threshold value Th1, the prediction result presentation unit35acquires a message set in the presentation content dictionary53based on the accident risk factor information46, and transmits a warning message including the accident risk information45and the accident risk factor information46to the prediction result display terminal90of the corresponding driver (S6). When the prediction result (the probability of the accident risk information45) calculated by the accident risk prediction unit33exceeds the predetermined threshold value Th1, the prediction result presentation unit35can also notify the administrator, the user, and the like of the driving assistance server1of the message.

In the notification when the accident risk information45exceeds the threshold value Th1, it is possible to notify the driver of a warning that is easily understood by the driver, by including the base for issuing the warning in the accident risk information45in the message in addition to the content of the accident risk information45.

The prediction result presentation unit35can transmit, to the prediction result display terminal90, a message for urging the driver to input a factor recognized by the driver with respect to the accident risk factor information46transmitted to the prediction result display terminal90. In addition, the prediction result presentation unit35can receive the input (factor label) of the driver from the factor label input unit91of the prediction result display terminal90(S7).

By inputting a factor recognized by the driver for the warning of which the notification has been performed by the driving assistance server1, and accumulating the input factor as a factor label in the accident risk factor information46, it is possible to make the warning output by the driving assistance server1be information that does not make the driver feel uncomfortable. The factor label from the factor label input unit91may be input after the end of driving or after the end of operation. The factor label from the factor label input unit91can be input by a driving administrator or the like instead of the driver.

When receiving the input of the factor label from the prediction result display terminal90, the prediction result presentation unit35sets and updates the received factor label in the accident risk factor information46. The accident risk prediction model training unit32can feed back the contents of the accident risk factor information46with the updated factor label to the accident risk judge information52to reflect the contents into the accident risk prediction model48and the accident risk factor calculation model49.

When training the accident risk prediction model48or the accident risk factor calculation model49, the accident risk prediction model training unit32can feed back the accident risk actually encountered by the driver to the accident risk prediction model48by using the accident risk judge information52with the updated factor label. The accident risk prediction model training unit32trains the accident risk prediction model48and the accident risk factor calculation model49at a predetermined timing (for example, monthly).

FIG.20is a diagram illustrating an example in which a warning from the driving assistance server1is output by a voice of the prediction result display terminal90. In the example illustrated inFIG.20, the prediction result display terminal90outputs, by voice, a warning message901indicating that the probability of an incident (near miss inFIG.20) occurring within 30 minutes is 80%, and a factor message902indicating that the number of consecutive clock-in days is six and the current weather is rainy as factors of issuing the warning.

The driver can understand that the cause of the call attention is the factor message902by the warning message901, and can call attention without feeling uncomfortable with respect to the warning of which the notification has been received during driving. In addition, a proposal leading to risk reduction may be added together with the warning.

FIG.21is a diagram illustrating an example of a screen for inputting a factor label displayed on the prediction result display terminal90. A screen910is an input screen of the factor label output to the display device of the prediction result display terminal90. The screen910is output by the factor label input unit91of the prediction result display terminal90.

The screen910includes a graph911of the accident risk information45including the time and the accident occurrence probability, the biological information41(progress of physical condition/fatigue)916, regions (number of consecutive times of clock-in912, weather913, delay status914) for displaying the progress of the operation information43and the environment information42, and an input unit915of a factor label.

In the example illustrated inFIG.21, the name of an accident risk factor at a time point Tx when the accident occurrence probability exceeds the predetermined threshold value Th1is urged to be input. The driver inputs a factor label through an input device (not illustrated) of the prediction result display terminal90. The factor label input unit91transmits the input factor label to the driving assistance server1.

Regarding the factor label, not only the text may be input to the input unit915, but also a plurality of factor labels may be displayed as buttons940as illustrated inFIG.22, and the factor label may be selected from the buttons940.

The screen910inFIG.22includes not only the elements inFIG.21, but also a video window930showing a driving state before the accident risk information45exceeds the threshold value Th1, in addition to the button940. In the video window930, it is possible to reproduce a video obtained by tracing back to a predetermined time point from a time point when the accident risk information45exceeds the threshold value Th1(the occurrence time point of the accident risk information45) among the videos captured by the camera15of the vehicle8. The video captured by the camera15is included in the driving information44and accumulated in the auxiliary storage device4.

The video of the driver tracing back to a predetermined time from the occurrence time point of the accident risk information45may be provided as the accident risk factor information46.

The driving assistance server1in the present embodiment can present the background of the accident risk information45from the biological information41(ANF information) and the driving information44by integrating the granularities of the measurement intervals of the biological information41, the environment information42, the operation information43, and the driving information44in accordance with the calculation interval of the biological information41(ANF information) to generate the time-series biological-environment-operation information47.

That is, when the accident occurrence probability of the accident risk information45increases, the driving assistance server1causes the accident risk factor calculation model49to predict the operation information43and the environment information42, which are factors of the increase, and performs a notification of the factor of the warning when performing a notification of the warning. Thus, it is possible to output the warning that can be understood by the driver.

When the driver is suddenly notified only that the probability of the occurrence of the accident risk has increased while driving the vehicle8, it is difficult for the driver to immediately understand why the warning has been issued. Therefore, when issuing the warning, the driving assistance server1in the present embodiment can notify the driver of information that is easily understood by the driver, by presenting the operation information43and the environment information42, which are the background of the warning.

Details of Data

Next, details of data used in the driving assistance server1will be described.

FIG.3is a diagram illustrating an example of the biological information41measured by the biological information collection device60. The biological information41includes a user ID411, the date and time412, heartbeat interval data413, blood pressure414, a body temperature415, and a medical inquiry result416in one record.

The user ID411stores an identifier of the driver. In the present embodiment, it is assumed that an identifier set in advance in the biological information collection device60is used. The date and time412stores the date and time when the biological information collection device60has measured the data.

The heartbeat interval data413stores a heartbeat interval (RRI data) measured by the heart rate meter61. The blood pressure414stores the blood pressure measured by the sphygmomanometer63. The body temperature415stores the body temperature measured by the body thermometer62. The medical inquiry result416stores an inquiry result at the start of work or the like. “Not measured” is stored in an item for which data has not been measured.

FIG.4is a diagram illustrating an example of the environment information42measured by the environment information collection device70. The environment information42includes an area ID421, the date422, a day of the week423, a time section424, weather425, an air temperature426, and an atmospheric pressure427in one record.

The area ID421stores an identifier of an area (such as a prefecture) where data has been acquired. The date422stores the date when the data has been acquired. The day of the week423stores the day of the week on which the data is acquired. The time section424stores a start point and an end point of the time at which the data has been acquired.

The weather425stores the weather acquired for each area. The air temperature426stores the air temperature measured by the air thermometer71. The atmospheric pressure427stores the atmospheric pressure measured by the barometer72.

FIG.5is a diagram illustrating an example of the attendance data431in the operation information43received by the operation information collection device80. The attendance data431includes a user ID4311, the clock-in date and time4312, the previous clock-out date and time4313, the number of consecutive clock-in days4314, and a break time4315in one record.

The user ID4311stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The clock-in date and time4312and the previous clock-out date and time4313store dates and times of clock-in and clock-out. The number of consecutive clock-in days4314stores the number of consecutive clock-in days. The break time4315stores a break time acquired by the driver.

FIG.6is a diagram illustrating an example of delivery data in the operation information43received by the operation information collection device80. The delivery data432includes a user ID4321, the date and time4322, an area ID4323, a carried object4324, a delivery route4325, and delayed-or-not4326in one record.

The user ID4321stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The date and time4322stores the date and time when the delivery has been started. The carried object4324stores the type of article to be delivered. The delivery route4325stores a delivery route. The delayed-or-not4326stores whether or not a delay has been reported in delivery.

The driving information44including position information or a traveling route of the vehicle8may be included in the delivery data432and handled as the operation information43.

FIG.12is a diagram illustrating an example of the time-series biological-environment-operation information47generated by the time-series biological-environment-operation information generation unit31. The past time-series biological-environment-operation information51has a similar configuration.

The time-series biological-environment-operation information47includes a user ID471, the date and time472, an autonomic nervous LF/HF473, a body temperature474, an air temperature475, the number of consecutive clock-in days476, a delay477, and a traveling state478, in one record.

The user ID471stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The date and time472stores the date and time when the biological information collection device60has measured the heart rate data as the starting point of an analysis window of the autonomic nervous LF/HF473forming the biological information41.

As will be described later, the autonomic nervous LF/HF473is a ratio between a low-frequency (LF) component and a high-frequency (HF) component of the power spectral density of the interval (RRI) of the R waves in the heart rate data. The autonomic nervous LF/HF473is stored as a value indicating the balance of the autonomic nerve (sympathetic nerve and parasympathetic nerve). The low-frequency component indicates an activity index of the sympathetic nerve, and the high-frequency component indicates an activity index of the parasympathetic nerve.

The body temperature474stores the body temperature measured by the biological information collection device60. The air temperature475stores the air temperature in the environment information42. The number of consecutive clock-in days476stores a value of the number of consecutive clock-in days4314of the attendance data431in the operation information43.

The delay477stores a value of the delayed-or-not4326of the delivery data432in the operation information43. The traveling state478stores a traveling state based on the speed and the position in the driving information44. In the present embodiment, the type of road is stored when the vehicle is traveling, and “stopped” is stored when the vehicle is stopped.

When the driving assistance server1predicts an accident risk based on the biological information41, the environment information42, and the operation information43of the driver regardless of the traveling state of the vehicle8, the driving assistance server1does not need to combine the value of the driving information44with the time-series biological-environment-operation information47.

FIG.14is a diagram illustrating an example of the accident risk judge information52. In the accident risk judge information52, pieces of information on accidents or incidents occurred in the past are accumulated. The accident risk judge information52includes a user ID521, the detection date and time522, a vehicle speed523, an acceleration524, an inter-vehicle distance525, camera information526, an incident presence or absence527, and an incident factor528, in one record.

The user ID521stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The detection date and time522stores the date and time when an accident or an incident has occurred. The vehicle speed523, the acceleration524, and the inter-vehicle distance525store detection values of the vehicle speed, the acceleration, and the inter-vehicle distance when the accident or the incident has occurred, respectively. The camera information526indicates image information when the accident or the incident has occurred.

The incident presence or absence527stores the presence or absence of an incident (or an accident). When there is an incident (or an accident), “1” is stored. When there is no incident, “0” is stored. The incident factor528stores a label of a factor causing the occurrence of the incident (or the accident).

The incident (or the accident) may be automatically detected from the driving information44of the inter-vehicle distance sensor12, the speed sensor13, the acceleration sensor14, and the like mounted on the vehicle8by a program (not illustrated), a machine learning model, or the like, with information at which a time point at which the possibility of the incident (or accident) is high, such as sudden braking.

For the detected incident (or accident), for example, the administrator or the like of the driving assistance server1refers to the camera information before and after the detection date and time522to set the incident presence or absence527. Then, the administrator or the like of the driving assistance server1determines the incident factor528, and inputs a label by text or the like. A machine learning model set in advance may perform setting of the incident presence or absence527and determination and setting of the incident factor528.

Regarding the determination of the incident (or the accident), as illustrated inFIG.23, the occurrence of the incident is detected in a deceleration state in which the inter-vehicle distance D1corresponding to the vehicle speed S1is less than a predetermined threshold value or the acceleration A1is less than a threshold value. The driving assistance server1acquires a video captured by the camera15from the driving information44before and after a time point at which the occurrence of the incident has been detected, and displays the video on the output device7. The administrator of the driving assistance system determines the factor of the incident (or the accident) from the videos before and after the incident from the output device7, and inputs the factor label from the input device6.

FIG.16is a diagram illustrating an example of the accident risk information45. The accident risk information45stores a prediction result calculated by the accident risk prediction unit33. The accident risk information45includes a user ID451, a measurement time point452, time-series biological-environment-operation information453, a prediction target time section454, and an accident occurrence probability455, in one record.

The user ID451stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The measurement time point452stores the date and time when the prediction has been made. The time-series biological-environment-operation information453stores a pointer for specifying the time-series biological-environment-operation information47used in prediction.

The prediction target time section454stores a start point and an end point of time in which the accident risk is predicted. The end point is after a predetermined time predicted by the accident risk prediction model48. The accident occurrence probability455stores a value representing the probability of the occurrence of an accident or an incident in percentage.

FIG.18is a diagram illustrating an example of the accident risk factor information46. The accident risk factor information46stores the factor calculated by the accident risk factor calculation unit34.

The accident risk factor information46includes a user ID461, an accident risk462, a first accident risk factor463, a second accident risk factor464, and a factor label465, in one record.

The user ID461stores the identifier of the driver. In the present embodiment, the identifier of the biological information41is used. The accident risk stores a pointer of the corresponding accident risk information45. The first accident risk factor463stores an element acting as the largest factor of the accident or the incident output by the accident risk factor calculation model49. The second accident risk factor464stores an element acting as the second largest factor output by the accident risk factor calculation model49. The factor label465stores the factor label received from the prediction result notification device9.

The accident risk factor calculation model49extracts items and values acting as factors of the accident occurrence probability455from the environment information42and the operation information43of the driver for which the accident risk information45has been calculated. Then, the accident risk factor calculation model49outputs the first accident risk factor463and the second accident risk factor464.

In addition, the first accident risk factor463and the second accident risk factor464can be distinguished from each other, for example, such that an item having the largest probability of being a factor of the accident occurrence probability455is set as the first accident risk factor463, and the next item is set as the second accident risk factor464.

The items of the accident risk factor calculation unit34are not limited to the illustrated items, and may be items included in the biological information41, the environment information42, the operation information43, and the driving information44. Items to be integrated as the time-series biological-environment-operation information47may be set in advance.

Although not illustrated, the presentation content dictionary53includes a template of a message of which the driver is notified. A sentence example corresponding to the magnitude of the accident occurrence probability455and the contents of the first accident risk factor463and the second accident risk factor464is set in advance.

Although not illustrated, the accident risk factor label54may be any information in which the factor label is associated with the first accident risk factor463or the second accident risk factor464.

Details of Processing

Details of the processing illustrated inFIG.2will be described below.FIG.7is a flowchart illustrating an example of a process performed by the time-series biological-environment-operation information generation unit31. This process is a process performed in Step S3inFIG.2.

The time-series biological-environment-operation information generation unit31first extracts heartbeat interval data413from the biological information41acquired in Step S2inFIG.2(S11). The time-series biological-environment-operation information generation unit31may also extract data of the blood pressure414and the body temperature415corresponding to the heartbeat interval data413in the biological information41.

Then, the time-series biological-environment-operation information generation unit31performs pre-processing such as exclusion or interpolation of a missing section, on each type of the extracted heartbeat interval data413, and the environment information42, the operation information43, and the driving information44acquired in Step S2, to generate pre-processed data450(S12). The pre-processed data450includes pre-processed heartbeat interval data413A, pre-processed environment information42A, pre-processed operation information43A, and pre-processed driving information44A.

Regarding the determination of the missing section, the biological information41to the driving information44have different data measurement (or acquisition) intervals. Therefore, as the threshold value Thf for determining the missing section, different values for the respective types of the heartbeat interval data413, the environment information42, the operation information43, and the driving information44can be set.

Then, as will be described later, the time-series biological-environment-operation information generation unit31calculates an autonomic nervous function LF/HF as an autonomic nerve function index from the pre-processed heartbeat interval data413A at a predetermined analysis window (time width), and accumulates the autonomic nervous function LF/HF in the pre-processed data450as ANF information473(S13).

The ANF information473is calculated as follows. The time-series biological-environment-operation information generation unit31calculates heartbeat interval data (RRI data) of an analysis window (predetermined period) ΔTw from the pre-processed heartbeat interval data413A illustrated inFIG.8, as heart rate variation time-series data, and further calculates variation from the heart rate variation time-series data.FIG.9is a graph illustrating an example of variation of the heartbeat interval data (heart rate variation) calculated by the time-series biological-environment-operation information generation unit31. The RRI in the heartbeat interval data is not constant and varies depending on the activity of the autonomic nerve or the like.

The time-series biological-environment-operation information generation unit31performs a frequency spectrum analysis on the heart rate variation time-series data to calculate power spectral density (PSD). A known method may be applied to the calculation of the power spectral density.

Then, the time-series biological-environment-operation information generation unit31calculates the intensity LF of the low-frequency component and the intensity HF of the high-frequency component in the power spectral density.FIG.10is a graph illustrating an example of the frequency domain of the power spectral density of the heart rate variation.

As illustrated inFIG.10, the time-series biological-environment-operation information generation unit31calculates, as autonomic nerve total power, a value obtained by summing (LF+HF) intensity (integral value) LF in a low-frequency component region (0.05 Hz to 0.15 Hz) and intensity (integral value) HF in a high-frequency component region (0.15 Hz to 0.40 Hz) of the power spectrum.

In addition, the time-series biological-environment-operation information generation unit31calculates, as the ANF information473, a ratio (autonomic nervous LF/HF) between the intensity LF of the low-frequency component and the intensity HF of the high-frequency component of the power spectrum.

With the above process, the driving assistance server1calculates the heart rate variation time-series data for each analysis window ΔTw from the heartbeat interval data of the biological information41, and further calculates the ratio between the intensity of the low-frequency component and the intensity of the high-frequency component, as the ANF information473.

The high-frequency component in the ANF information473appears in the heart rate variation when the parasympathetic nerve is activated (tensioned). The low-frequency component appears in the heart rate variation both when the sympathetic nerve is activated (tensioned) and when the parasympathetic nerve is activated (tensioned).

Since it is known that the driver is in a stress state when the sympathetic nerve is activated and is in a relaxed state when the parasympathetic nerve is activated, it is possible to determine whether the driver is in the stress state or the relaxed state, from the intensity LF of the low-frequency component and the intensity HF of the high-frequency component.

Then, the time-series biological-environment-operation information generation unit31aligns the time widths of the ANF information473, the pre-processed environment information42A, the pre-processed operation information43A, and the pre-processed driving information44A, and generates the time-series biological-environment-operation information47obtained by combining the ANF information473, the pre-processed environment information42A, the pre-processed operation information43A, and the pre-processed driving information44A (S14).

The ANF information473is generated for each predetermined analysis window ΔTw for which the pre-processed heartbeat interval data413A is acquired. The time interval of the analysis window ΔTw is, for example, 1 minute. On the other hand, the pre-processed environment information42A is acquired at a time interval such as every 1 hour. Data of the pre-processed driving information44A is collected at a measurement interval (for example, one second interval) of the sensor of the vehicle8. The pre-processed operation information43A is irregularly recorded in accordance with the break of the operation of the driver.

As illustrated inFIG.11, the measurement (or acquisition) timings of the pre-processed environment information42A to the pre-processed driving information44A are different from the calculation interval (analysis window ΔTw) of the ANF information473. In addition, the granularity of the measurement interval (acquisition interval) is also different.

Therefore, the time-series biological-environment-operation information generation unit31shapes types of data of the pre-processed environment information42A to the pre-processed driving information44A in accordance with the calculation interval of the ANF information473which is the biological information41of a monitoring target for issuing the warning. Then, the time-series biological-environment-operation information generation unit31combines the pieces of shaped data to generate the time-series biological-environment-operation information47of one record.

In the case of the pre-processed driving information44A having a time interval shorter than the calculation interval (analysis window ΔTw) of the ANF information473, the time-series biological-environment-operation information generation unit31calculates a representative value such as an average value within the time interval of the analysis window ΔTw, as a value corresponding to the ANF information473.

On the other hand, in the case of the pre-processed environment information42A or the pre-processed operation information43A having a time interval that is longer than the calculation interval (analysis window ΔTw) of the ANF information473, the time-series biological-environment-operation information generation unit31acquires data immediately after (or immediately before) the analysis window ΔTw, as data corresponding to the ANF information473.

As described above, the time-series biological-environment-operation information generation unit31combines the types of data of the pre-processed environment information42A, the pre-processed operation information43A, and the pre-processed driving information44A into one record after matching with the interval of the analysis window ΔTw of the ANF information473, to generate the time-series biological-environment-operation information47.

As a result, the time-series biological-environment-operation information47can be generated without deviation in the time-series direction as illustrated inFIG.12, in a manner that the pre-processed environment information42A, the pre-processed operation information43A, and the pre-processed driving information44A corresponding to the time series of the ANF information473are combined based on the time interval of the ANF information473calculated for each analysis window ΔTw for acquiring the heartbeat interval data413.

The data accumulated in the time-series biological-environment-operation information47is reflected in the past accident risk prediction model training data50at a predetermined timing.

FIG.13is a flowchart illustrating an example of a process performed by the accident risk prediction model training unit32. This process is a process performed in Step S1inFIG.2.

The accident risk prediction model training unit32receives a designated period of data to be used for training, extracts data within the designated period from the past time-series biological-environment-operation information51in the accident risk prediction model training data50, and generates the extracted data as model input data500(S21).

The designated period can be received from the input device6or an external computer, and is input by the user or the administrator of the driving assistance system. In addition, the designated period desirably has a time width of 2 minutes to several hours.

Then, the accident risk prediction model training unit32acquires accident risk judge information52in which past information in which the administrator or the computer has determined an actual accident or an incident leading to an accident has been set in advance. Then, the accident risk prediction model training unit32extract information on the presence or absence of an incident occurrence from the past time-series biological-environment-operation information51within the designated period, and uses the extracted information as a teacher label55(S22).

In this process, the future in which an incident is predicted from the past time-series biological-environment-operation information51means a period within a designated period or a period similar to the designated period. For example, when the designated period is 30 minutes, if the time point of the occurrence of the incident in the accident risk judge information52is an incident within 30 minutes from the start time point of the time-series biological-environment-operation information51, a label of the occurrence of the incident is given to this time point, and is used as the teacher label55. That is, the incident factor528corresponding to the incident presence or absence527registered in the accident risk judge information52is associated with the past time-series biological-environment-operation information51in a section tracing back any time width (30 minutes or the like) from the corresponding time point, as the teacher label of the accident risk prediction model48.

Then, the accident risk prediction model48of outputting an accident risk in the future (after a predetermined time: for example, after 30 minutes) is trained by using the model input data500extracted in Step S21and the teacher label55generated in Step S22(S23).

FIG.24illustrates an example of training the accident risk prediction model48. While the presence or absence of the incident corresponding to the teacher label55is a binary value of “0” or “1”, the value output by the accident risk prediction model48is a continuous value falling within a range of 0 to 1, for example. When the accident risk prediction model48is applied, a process, for example, using the predicted continuous value as it is, or setting and converting a threshold value is performed to calculate an accident occurrence probability of 0 to 100%.

In the above training, it is possible to improve the prediction accuracy of the accident risk prediction model48by adding new time-series biological-environment-operation information47to the past time-series biological-environment-operation information51and using the accident risk judge information52with the added accident risk factor label54.

FIG.15is a flowchart illustrating an example of a process performed by the accident risk prediction unit33. This process is a process performed in Step S4inFIG.2. The accident risk prediction unit33acquires the time-series biological-environment-operation information47, and inputs the time-series biological-environment-operation information47to the trained accident risk prediction model48to cause the accident risk prediction model48to predict an accident risk of the driver after a predetermined time. The accident risk prediction unit33stores the accident occurrence probability output by the accident risk prediction model48as the accident risk information45in the auxiliary storage device4(S31).

The accident risk prediction unit33stores the user ID471of the time-series biological-environment-operation information47in the user ID451of the accident risk information45, and similarly stores the date and time472of the time-series biological-environment-operation information47in the measurement time point452. The accident risk prediction unit33stores the pointer for specifying a record of the time-series biological-environment-operation information47in the time-series biological-environment-operation information453, and stores a range of the prediction time point output by the accident risk prediction model48in the prediction target time section454. The accident risk prediction unit33stores the accident occurrence probability output by the accident risk prediction model48in the accident occurrence probability455.

The data as a processing target, which is input to the accident risk prediction model48by the accident risk prediction unit33, is unprocessed data in the time-series biological-environment-operation information47.

With the above process, the data of the time-series biological-environment-operation information47is input to the trained accident risk prediction model48, and the accident risk information45after a predetermined time is output for each driver.

The accident risk prediction unit33may omit generation of the accident risk information45when the accident occurrence probability output by the accident risk prediction model48is equal to or less than a predetermined threshold value Th2(for example, 5%).

FIG.17is a flowchart illustrating an example of a process performed by the accident risk factor calculation unit34. This process is a process performed in Step S5inFIG.2.

The accident risk factor calculation unit34acquires the accident risk information45output by the accident risk prediction unit33, the accident risk judge information52obtained by collecting past cases, and the time-series biological-environment-operation information47input to the accident risk prediction unit33. The accident risk factor calculation unit34inputs the acquired types of information to the accident risk factor calculation model49set in advance, to generate accident risk factor information46(S41).

The accident risk factor calculation unit34stores the user ID451of the accident risk information45in the user ID461of the accident risk factor information46, and stores the pointer for specifying a record of the accident risk information45in the accident risk462. The accident risk factor calculation unit34stores the first and second accident risk factors output by the accident risk prediction model48in the first accident risk factor463and the second accident risk factor464, respectively, and stores the incident factor528of the accident risk judge information52in the factor label465.

With the above process, for the driver for which the accident risk information45has been generated, factors predicted by the accident risk factor calculation model49from the incident factors528of the current time-series biological-environment-operation information47and the past accident risk judge information52are generated as the first accident risk factor463, the second accident risk factor464, and the factor label465.

The accident risk factor calculation unit34sets “none” in the factor label465of the accident risk factor information46for data in which the incident factor528of the accident risk judge information52is blank. In addition, the first accident risk factor463indicates a main factor by which the accident risk has occurred, and the second accident risk factor464indicates a background factor by which the accident risk has increased.

The accident risk factor information46inFIG.18illustrates an example in which the first accident risk factor463is estimated based on the operation information43as a main factor that “the number of consecutive clock-in days exceeds six days”, and the second accident risk factor464is estimated based on the environment information42as a factor in the background that it rains.

FIG.19is a flowchart illustrating an example of a process performed by the prediction result presentation unit35. This process is performed in Steps S6and S7inFIG.2.

The prediction result presentation unit35acquires the prediction target time section454and the accident occurrence probability455from the accident risk information45, and performs processes as follows when the accident occurrence probability455exceeds the predetermined threshold value Th1.

The prediction result presentation unit35acquires the first accident risk factor463, the second accident risk factor464, and the factor label465from the accident risk factor information46corresponding to the accident risk information45. The prediction result presentation unit35searches the presentation content dictionary53for the accident occurrence probability455, and the first accident risk factor463or the second accident risk factor464to acquire a template of a sentence. The prediction result presentation unit35inserts the accident occurrence probability455, the prediction target time section454, and the first accident risk factor463or the second accident risk factor464into the acquired template to generate a warning or a call attention message. Then, the prediction result presentation unit35transmits the generated warning or call attention message to the prediction result display terminal90used by the driver or the administrator (S51).

In addition, the prediction result presentation unit35transmits information for receiving the factor label in addition to the message. When receiving the factor label input by the driver, the administrator, or the like with the prediction result display terminal90, the prediction result presentation unit35updates (or adds) the factor label465of the accident risk factor information46(S52).

When the accident risk factor information46is updated, the prediction result presentation unit35can feed back the updated information to the accident risk judge information52and add the content of the factor label465to the incident factor528. As a result, it is possible to reflect the incident factor528set by the driver or the like when the accident risk prediction model48is trained. Then, the driving assistance server1can generate and transmit a message that does not make the driver feel uncomfortable.

As described above, the driving assistance server1in the present embodiment combines the types of information having different measurement intervals and acquisition intervals, for example, the environment information42, the operation information43, and the driving information44, with each other in accordance with the time interval at which the biological information41is calculated, to generate the time-series biological-environment-operation information47and stores the generated time-series biological-environment-operation information47in time series. Then, the driving assistance server1predicts an accident occurrence probability455(accident risk information45) after a predetermined time (future) by using the time-series biological-environment-operation information47in accordance with the calculation interval of the biological information41, and transmits a warning or a call attention message when the accident occurrence probability455exceeds the threshold value Th1.

As a result, by integrating the environment information42, the operation information43, and the driving information44that affect the accident occurrence probability455into information in accordance with the calculation interval of the biological information41, it is possible to improve the prediction accuracy of an accident or an incident.

In addition, the driving assistance server1calculates the accident risk factor information46by inputting the accident risk information45of the prediction result, the past accident risk judge information52, and the new time-series biological-environment-operation information51to the accident risk factor calculation model49, and includes the cause of the accident risk in the message. Thus, it is possible to present the reason for warning or call attention to the driver. By adding the factor of an accident risk in addition to a warning or call attention, it is possible to perform a notification of a message without discomfort.

In addition, by inputting, to the accident risk factor calculation model49, the time-series biological-environment-operation information47obtained by integrating data of the environment information42, the operation information43, and the driving information44such that the time-series of the granularities of the environment information42, the operation information43, and the driving information44match with the calculation interval of the biological information41, it is possible to calculate the accident risk factor information46having no time series deviation.

In addition, the driving assistance server1can receive a label for the accident risk factor from the prediction result display terminal90with respect to the accident risk information45transmitted to the prediction result display terminal90, and feedback the received label to the accident risk prediction model48and the accident risk factor calculation model49. As a result, it is possible to issue a warning that does not make the driver feel uncomfortable.

In addition, the driving assistance server1provides a video of the driving state before the occurrence time point of the accident risk information45as the accident risk factor information46, so that it is possible to notify the driver without feeling uncomfortable.

CONCLUSION

As described above, the above-described embodiment can have the following configurations.

(1) A driving assistance method in which a computer (driving assistance server1) including a processor (2) and a memory (3) assists driving of a vehicle (8), the driving assistance method including: by the computer (1), acquiring biological information (41) of a driver who is driving the vehicle (8) (biological information collection device60); acquiring environment information (42) of the driver (environment information collection device70); acquiring operation information (43) of the driver (operation information collection device80); generating biological index data from the biological information (41) (S31); generating integrated information (time-series biological-environment-operation information47) by aligning and combining time series of the biological index data (ANF information473), the environment information (42), and the operation information (time-series biological-environment-operation information generation unit31); calculating accident risk information (45) after a predetermined time, by inputting the integrated information (31) to an accident risk prediction model (48) set in advance (S4); and calculating factor information (accident risk factor information46) of the accident risk information (45) by inputting the accident risk information (45), the integrated information (31), and accident risk judge information (52) set in advance to a factor calculation model (accident risk factor calculation model49) set in advance (accident risk factor calculation unit34).

With the above configuration, the driving assistance server1not only predicts accident risk information45after a predetermined time, but also outputs accident risk factor information46, so that it is possible to present information that does not make the driver feel uncomfortable.

(2) The driving assistance method described in (1), further including outputting, by the computer, the accident risk information (45) and the factor information (46) when the accident risk information (45) satisfies a predetermined condition (exceeding a threshold value Th1) (S6).

With the above configuration, the driving assistance server1outputs the accident risk information45and the accident risk factor information46to the prediction result display terminal90, when the accident risk information45after the predetermined time satisfies the predetermined condition. In addition to the fact that the accident risk to the driver has increased, it is possible to present the factors of the accident risk.

(3) The driving assistance method described in (2), further including updating, by the computer, the factor information (46) with a factor label when the computer receives the factor label for the output factor information (46) (S7).

With the above configuration, by updating the accident risk judge information52with the factor label set by the driver or an administrator, it is possible to reflect an accident risk actually encountered by the driver.

(4) The driving assistance method described in (3), further including, by the computer, reflecting the factor label used to update the factor information (46) to the accident risk judge information (52), and performing learning of the factor calculation model (49) with the accident risk judge information (52) in which the factor label has been reflected (S48).

With the above configuration, when the accident risk prediction model48is trained again, it is possible to feed back the accident risk actually encountered by the driver to the accident risk prediction model48by using the accident risk judge information52with the updated factor label.

(5) In the driving assistance method described in (1), in which, in the generating of the integrated information, when the biological information (41), the environment information (42), and the operation information are integrated, values of the environment information (42) and the operation information are acquired by using, as a reference, a calculation interval of the biological information (41).

With the above configuration, the environment information42and the operation information43are acquired by matching the time-series biological-environment-operation information47with the calculation interval of the information (for example, the ANF information473) calculated from the biological information41, so that the time-series biological-environment-operation information47is information with a uniform granularity in time series. As a result, it is possible to improve the calculation accuracy of the accident risk prediction model48and the accident risk factor calculation model49using the time-series biological-environment-operation information47.

The present invention is not limited to the above embodiment, and various modification examples may be provided. For example, the above embodiments are described in detail in order to explain the present invention in an easy-to-understand manner, and the above embodiments are not necessarily limited to a case including all the described configurations. Further, some components in one embodiment can be replaced with the components in another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, for some of the components in the embodiments, any of addition, deletion, or replacement of other components can be applied singly or in combination.

Some or all of the configurations, functions, functional units, processing means, and the like may be realized in hardware by being designed with an integrated circuit, for example. Further, the above-described respective components, functions, and the like may be realized by software by the processor interpreting and executing a program for realizing the respective functions. Information such as a program, a table, and a file, that realizes each function can be stored in a memory, a recording device such as a hard disk and a solid state drive (SSD), or a recording medium such as an IC card, an SD card, and a DVD.

Control lines and information lines considered necessary for the descriptions are illustrated, and not all the control lines and the information lines in the product are necessarily shown. In practice, it may be considered that almost all components are connected to each other.