System and method for fusing information related to a driver of a vehicle

A method, apparatus and computer program product of determining a state of a vehicle driver, the method comprising: receiving an image of the driver captured by a hyper spectral camera capable of imaging body features invisible to a human; receiving telemetry information from a car telemetry system; analyzing the image to receive at least one indicator to a clinical parameter of the driver; and fusing the at least one indicator with the telemetry information to obtain an assessment to a stress level of the driver.

TECHNICAL FIELD

The present disclosure relates to assessing the stress level of a driver and to utilizing this assessment for improving safety of the driver and the environment.

BACKGROUND

A driver of a vehicle carries enormous responsibility to the wellbeing of himself, passengers in the car, other people in the environment, and property, including the driver's vehicle and other vehicles and objects in the environment.

The driver's behavior also has effect on the behavior of other drivers. Aggressive behavior, including for example denying the right of way from other drivers, over speeding, tailgating, crossing lanes or other behaviors is not only dangerous in itself, but may also cause other drivers to react aggressively too, due to confusion, as vengeance or since they become aggravated too, while patient and courteous behavior may improve the atmosphere and the driving manners of other drivers in the vicinity.

The driving style of a driver is largely affected by the basic character of the driver, the driver's physical health which in itself may comprise a multiplicity of factors, the driver's mental health and mental status, including for example the current stress level experienced by the driver.

BRIEF SUMMARY

One exemplary embodiment of the disclosed subject matter is a computer-implemented method of determining a state of a vehicle driver, to be performed by a device comprising a processor and a memory device, the method comprising: receiving an image of the driver captured by a hyper spectral camera capable of imaging body features invisible to a human; receiving telemetry information from a car telemetry system; analyzing the image to receive an indicator to a clinical parameter of the driver; and fusing the indicator with the telemetry information to obtain an assessment to a stress level of the driver. The method can further comprise determining and taking an action in response to the stress level exceeding a predetermined threshold. Within the method, the action is optionally affecting a behavior of a car system. Within the method, the car system is optionally an autonomous driving system. Within the method, the behavior optionally relates to changing parameters in accordance with a weather condition and driver behavior. Within the method, the action is optionally collecting information to be provided to an insurer or making an offer to the driver. The method can further comprise collecting information related to a multiplicity of drivers in a road, and issuing an alert related to a road, subject to a multiplicity of drivers having stress levels exceeding a threshold. The method can further comprise: receiving data from an additional source; and analyzing the data to obtain an additional indicator; and fusing the additional indicator with the telemetry information and the indicator to obtain the assessment. Within the method, the additional source is optionally an image capture device or a voice capture device, and further comprising analyzing an image captured by the image capture device or voice captured by the voice capture device. Within the method, the additional source is at least one item selected from the group consisting of: a calendar of the driver, a global positioning system, a weather forecast source, a road condition report and a wearable device. The method can further comprise determining in accordance with the calendar and with a Global Positioning System whether the driver is late for a meeting. The method can further comprise adapting parameters or thresholds from a behavior of the driver over time; and using the parameters or thresholds as learned in obtaining a stress level assessment.

Another exemplary embodiment of the disclosed subject matter is an apparatus for fusing information related to a driver of a vehicle, the apparatus comprising: a hyper spectral camera capable of imaging body features invisible to a human; a car telemetry system providing telemetry information; a processor adapted to perform the steps of: receiving an image of the driver captured by the hyper spectral camera; receiving telemetry information from the car telemetry system; analyzing the image to receive an indicator to a clinical parameter of the driver; and fusing the indicator with the telemetry information to obtain an assessment to a stress level of the driver. Within the apparatus, the processor is optionally further adapted to determine and take an action in response to the stress level exceeding a predetermined threshold. Within the apparatus, the action is optionally affecting an autonomous driving system or an alert system. Within the apparatus, the action is optionally collecting information to be provided to an insurer or making an offer to the driver. Within the apparatus, the processor is optionally further adapted to: receive data from an additional source; analyze the data to obtain an additional indicator; and fuse the additional indicator with the telemetry information and the indicator to obtain the assessment. Within the apparatus, the additional source is optionally selected form the group consisting of: an image capture device, a voice capture device, a calendar of the driver, a global positioning system, a weather forecast source, a road condition report and a wearable device. Within the apparatus, the processor is optionally further adapted to: adapt parameters or thresholds from a behavior of the driver over time; and use the parameters or thresholds as learned in obtaining a stress level assessment.

Yet another exemplary embodiment of the disclosed subject matter is a computer program product comprising a non-transitory computer readable storage medium retaining program instructions configured to cause a processor to perform actions, which program instructions implement: receiving an image of the driver captured by the hyper spectral camera; receiving telemetry information from the car telemetry system; analyzing the image to receive an indicator to a clinical parameter of the driver; and fusing the indicator with the telemetry information to obtain an assessment to a stress level of the driver.

DETAILED DESCRIPTION

Multi-spectral capturing is a technology that is gaining popularity and becoming more and more affordable. A multi spectral camera is a camera that captures images at two or more frequency bands across the electromagnetic spectrum. The wavelengths are separated by filters or other components that are sensitive to a particular wavelength.

Thus, a multi spectral camera provides images having features at two or more separate wavelength bands.

Hyperspectral imaging (HSI), like other spectral imaging techniques, is used for collecting and processing information in wavelengths across the electromagnetic spectrum. For example, a hyper spectral camera divides the light range to small sub ranges, such as 300 sub-ranges of 2-nanometer each. HSI is aimed at obtaining a multiplicity of ranges within the spectrum in an image of a scene, usually for detecting objects, identifying materials or states thereof, processes, or the like. In the disclosure below multi spectral cameras and hyper spectral cameras are used interchangeably as long as the device captures at least very-near IR (VNIR), mid-wave IR (MWIR), all or part of the wavelength range of 400 to 1,000 nm and thermal Infrared range.

Whereas the human eye sees color of visible light in mostly three bands (red, green and blue), spectral imaging divides the spectrum into a larger number, covering a wide range of bands, some of which may be beyond the visible spectrum. In hyperspectral imaging, the recorded spectra may have fine wavelength resolution and may cover a wide range of wavelengths. Hyperspectral imaging may measure contiguous spectral bands, as opposed to measuring spaced spectral bands.

Hyperspectral sensors and respective processing systems have applications in a multiplicity of areas. Of special interest are applications in which body parts such as the face of a person are captured by a hyperspectral sensor. Various parameters such as tissue oxygen level, and in particular blood oxygenation level, although invisible to the human eye, are noticeable by HSI.

One technical problem dealt with by the disclosed subject matter is the need to assess the mental state of a driver, and in particular the driver's stress level.

Another technical problem relates to tracking the stress level of the driver, whether during a ride or over a time period such as a day, a month, a year, or the like, and taking an action in accordance with the driver stress level when an acceptable level is exceeded. The action may be aimed at relaxing a driver experiencing high stress level, rewarding relaxed drivers, or the like.

One technical solution comprises the assessment of the stress level of a driver from hyper spectral images of the driver, for example by evaluating physical parameters such as blood oxygenation. This assessment is combined with a second assessment based on telemetry data retrieved from the car systems and providing information about the driving, such as the speed the driver is driving at, lane crossing, acceleration and braking, compliance with road signs and road conditions, or the like. Additional data can be considered in assessing the stress level, for example information obtained by capturing and analyzing the driver's voice using speech to text and natural language processing (NLP), by capturing the driver's face and body and, or analyzing the driver's gestures, accessing and analyzing text or voice messages the driver received the like. Further information can be obtained from external sources, such as a calendar of the driver with or without location data, such that it may be assessed whether the driver is late for a meeting, a Global Positioning System (GPS), weather reports or road conditions report, a wearable device providing indications about the clinical parameters of the driver, a thermometer, or the like.

Another technical solution comprises taking actions in accordance with the stress level. Thus, the stress level should be assessed in real-time, near-real-real-time or at such time intervals that enable actions to be timely taken. Short-range actions may include allowing or preventing control of the car from the user; changing the parameters of alerts given to the driver by the system such as alert level, alert frequency, alert volume or the like; presenting suggestions to the driver, for example to stop for a rest, to get some food or drink wherein the suggestion may be displayed on a dedicated display, on the user's mobile device, or the like; sending a message to a predetermined person, such as a friend of the driver asking them to call the driver and relax him, or the like.

Longer duration actions may include collecting information over time, and using it for determining the premium of the driver for the car insurance, such that a driver that is often stressed may pay a higher premium than a driver who is generally calm.

Yet another technical solution comprises personalizing the parameters for assessing the stress level, such that the stress level is assessed in accordance with the particular driver. For example, a base line blood oxygenation level, a base pulse or other parameters may be determined, such that measured values can be interpreted correctly in respect of the driver.

Yet another technical solution comprises learning and updating the parameters for assessing the stress level over time, such that the stress level is assessed correctly for the particular driver over time.

Yet another technical solution comprises receiving information from a multiplicity of systems as above by a server, and analyzing behaviors of a multiplicity of drivers driving on the same road, at the same time or at different times. The analysis may enable the identification of problematic roads, such that measures can be taken, for example providing rest areas, setting reduced speed limit, or the like.

One technical effect of the disclosure relates to monitoring the mental state of a driver and particularly the stress level of the driver, and taking one or more actions to increase safety for the driver and the people and objects in the vicinity thereof. The actions may include immediate actions, as well as actions intended to reward calm drivers and encourage stressed drivers to relax.

Referring now toFIG. 1, showing a schematic block diagram of an apparatus for assessing a stress level of a driver and taking related actions, in accordance with some embodiments of the disclosure.

The apparatus, generally referenced100, may be installed in a vehicle such as a car, a bus, or the like. Apparatus100may comprise a hyperspectral camera104. Hyperspectral camera104may be installed such that it can capture the face or other exposed body parts of the driver. Hyperspectral camera104may be sensitive to very-near IR (VNIR), mid-wave IR (MWIR) and all or parts of the wavelength range of 400 to 1,000 nm and the thermal range.

Apparatus100may comprise a vehicle telemetry interface108for receiving indications from one or more systems of the car, such as the brakes system, the engine, the cruise controller, or the like. Vehicle telemetry interface108may operate by apparatus100being connected to a car communication system, such as CANBUS, and communicating with the vehicle systems via such channel.

Apparatus100may comprise one or more image capture devices112, for capturing one or more images or video frames of the driver, of the environment for example the inside of the car, the road including road signs, other vehicles, objects such as other vehicles, pedestrians, road signs, or the like, in the vicinity of the vehicle. Image capture devices112may include any visible light camera, such as dashboard camera, which may capture still or visible images.

Apparatus100may comprise one or more voice capture devices116, for capturing the voice of the driver, passengers in the car and optionally external noises.

Apparatus100may comprise one or more physical measurement components120for taking measurements of physical environmental parameters, such as temperature, humidity, barometric pressure, visibility, or others, or clinical parameters of the driver, such as pulse, blood pressure, temperature or the like.

Apparatus100may comprise computing device124, which may comprise one or more processors128. Any of processors128may be a Central Processing Unit (CPU), a microprocessor, an electronic circuit, an Integrated Circuit (IC) or the like. Alternatively, computing device124can be implemented as firmware written for or ported to a specific processor such as digital signal processor (DSP) or microcontrollers, or can be implemented as hardware or configurable hardware such as field programmable gate array (FPGA) or application specific integrated circuit (ASIC). Processor128may be utilized to perform computations required by apparatus100or any of it subcomponents.

Computing device124may comprise one or more storage devices132for storing executable components, and which may also contain data during execution of one or more components. Storage device132may be persistent or volatile. For example, storage device132can be a Flash disk, a Random Access Memory (RAM), a memory chip, an optical storage device such as a CD, a DVD, or a laser disk; a magnetic storage device such as a tape, a hard disk, storage area network (SAN), a network attached storage (NAS), or others; a semiconductor storage device such as Flash device, memory stick, or the like. In some exemplary embodiments, storage device132may retain data structures and program code operative to cause any of processors128to perform acts associated with any of the steps shown inFIG. 2below.

The components detailed below may be implemented as one or more sets of interrelated computer instructions, executed for example by any of processors804or by another processor. The components may be arranged as one or more executable files, dynamic libraries, static libraries, methods, functions, services, or the like, programmed in any programming language and under any computing environment.

In some exemplary embodiments of the disclosed subject matter, storage device132may comprise hyperspectral image analysis engine136, for analyzing one or more images captured by hyperspectral image104. Due to the wavelengths captured by hyperspectral image104, including 400-1000 nanometers and the thermal range, hyperspectral image analysis engine136may assess clinical parameters such as but not limited to pigmentations of oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb).

Storage device132may comprise telemetric analysis engine144for analyzing the data received by vehicle telemetric interface108, and determining one or more factors related to stress level by analyzing demonstrated behaviors, such as over speeding, high acceleration and deceleration, lane crossing, or the like.

Analyzing the stress level using a hyper spectral imaging device is disclosed, for example, in “Hyperspectral Imaging for Safety and Security” by Valerie C. Coffey, published in Optics and Photonics News Vol. 26, Issue 10, pp. 26-33 (2015), or in “Detection of Psychological Stress Using a Hyperspectral Imaging Technique” by Tong Chen, Peter Yuen, Mark Richardson, Guangyuan Liu, and Zhishun She, published in IEEE Transactions ON Affective Computing, Vol. 5, No. 4, October-December 2014.

Storage device132may comprise image analysis engine148for determining one or more parameters from captured images, such as the facial and hand gestures by the driver, outside conditions, or the like.

Storage device132may comprise voice analysis engine152, which may comprise for example speech recognition for recognizing speech by the driver or by the passengers.

Storage device132may comprise calendar interface156, for obtaining information about when and where the driver or another passenger onboard has to be.

Storage device132may comprise additional source interface160, for example interface to a positioning system for determining a current location, interface to the media system of the vehicle or to other information sources.

Data from the above sources may be input into stress level analysis engine164, which may combine the data into an estimation of the stress level of the driver.

For example, if a sum, a product, an average or another combination of a normalized oxygen saturation level obtained by physical attribute estimation component140and normalized stress level as obtained from telemetric analysis engine144is below a first threshold, it may be assessed that the driver is relaxed, if the combination exceeds the first threshold and is below a second threshold, then it may be assumed that the driver is somewhat stressed, while if the combination exceeds the second threshold, the driver may be assumed to be stressed.

The combination may also take into account data from the other sources. For example, if it is known from the calendar that the driver has to arrive at a certain time to a certain place, and the GPS indicates that he is running late, the combined stress level may be increased. If image analysis engine148indicates excess hand gestures by the driver, or if voice analysis engine152indicates loud voices or strong language, the stress level estimation may also be increased.

Once an estimation of the stress level is obtained, it may be provided to action selection engine168, which may determine one or more actions to be taken in accordance with the estimation. For example, if the driver is extremely stressed, an automatic driving system may take over and not let the driver control the car; an alert system may issue a severe alert to the driver; a message may be sent to a predetermined number to report the stress level of the driver; the stress level may be stored and the accumulated stored reports may be used for determining the insurance premiums, or the like.

Action activation engine172may be used for carrying the selected action, such as sending a command to a car system via the communication channel such as the CANBUS, calling or messaging a number, storing a report, or the like.

Storage device132may comprise learning engine176for learning the parameters of the specific driver, for example stress threshold level, basic saturation level, normal speech volume or the like. Learning may be performed by transmitting the measurements and stress assessments to a server and normalizing the results with other drivers, normalizing clinical parameters of the driver over time, or the like. Learning can also be influenced by feedback, for example the effect of the taken actions on the stress level, e.g., were the actions effective in reducing the stress level. The initial values may be set as general defaults, retrieved from a look up table if some driver characteristics such as gender and age are known, or the like. Receiving information related to a multiplicity of drivers on the same road can also be used for analyzing whether the road is problematic and requires additional surveillance, additional rest areas, or the like, subject for example to an average stress level exceeding a normal level.

The apparatus may be a unit comprising components as detailed above. However, the components may be located in various places over the vehicle, for example, some image and/or voice capture devices may be located at different locations within the car.

Referring now toFIG. 2, showing a schematic flowchart of a method for assessing a stress level of a driver, in accordance with some embodiments of the disclosure.

On step200, system setup and calibration may take place. System setup and calibration step200can include, for example placing the various capture devices in places where the required image or voice can be captured, for example hyper spectral camera104can be located and calibrated such that it captures at least the driver's face and neck.

On step204, one or more images captured by the hyperspectral camera may be received, and on step208the images may be analyzed in order to obtain a physical attribute estimate, such as an estimate to the stress level of the driver.

On step212, one or more readings may be received from the car telemetry systems, such as speed, acceleration, lane passing, honking, or the like, and on step216the stress level may be assessed form the readings. For example, frequent lane passing, frequent honking, a predetermined number of accelerations or decelerations having intensity that exceeds a threshold over a duration, or other factors, may indicate stress.

On step220, information from other sensors or systems may be received, such as data, readings or capturing. For example, images may be received form one or more image capture devices, audio can be received form a voice capture device, calendar information may be obtained via an interface, location can be obtained from a GPS. or the like. On step224, the information may be analyzed for stress level indications. For example, high volume speech or excessive hand gestures, being late to a meeting, as deduced from a calendar event and a GPS location, or other events or combinations may indicate high stress levels.

On step228, an overall assessment of the driver stress level may be obtained by combining at least the stress level as determined on step208based on the images taken by the hyper spectral camera and the stress level as obtained on step216from the car telemetry. If additional assessments are available, they can also be combined to obtain a possibly more exact assessment. The assessment can also be affected by combination of factors such as driver stress level, and weather as received from a weather report system, or road condition received from a suitable source, and may include actions as detailed above. In other examples the assessment can use a combination of the driver's location and scheduled meeting to realize whether the driver is about to be late and may thus be stressed, analyzing the driver's voice, words, gestures, or the like; analyzing received text or voice messages or the like.

On step232it may be determined whether the assessed stress level is below or above a predetermined threshold. It will be appreciated that the threshold can be different for different people, or even change over time for the same driver. Such changing can be periodical for example a different threshold may be applied on different parts of the day, while the change may also be more due to the learning mechanism.

If the stress level does not exceed the threshold, data about the current measurements, received images, and obtained stress levels may be stored in a storage device such as storage device132.

If the stress level exceeds the threshold, then on step236one or more actions to be taken may be determined, such as but not limited to affecting the behavior of car system, such as taking decisions including driving decisions, changing parameters, collecting data, changing control or car alert systems, providing offers to the driver, notifying a predetermined person or system, storing the information, sending a report, or the like.

On step238the determined one or more actions can be taken. For example, on step240a message may be sent to one or more of the car systems, such as changing the alert level and timing, limiting the speed, enabling or disabling automatic or manual driving, or the like. The message to the car system or to a controller may be sent via a vehicle communication channel, such as CANBUS.

On step244one or more offers may be provided to the driver, wither visually on a display device or orally via a speaker, such as offers to stop for food, drink, rest, or the like.

On step248the relevant information may be stored as detailed on association with step234above, and/or reported to a third party. It will be appreciated that such reporting can but is not required to be immediate. For example, a report about excessive stress level can be reported to the authorities, while data collected over time may be provided to an insurer upon specific request or when the insurance is to be renewed.

On step252, a predetermined destination may be notified of the stress level, such as a predetermined person who can call the driver.

It will be appreciated that the listed actions are exemplary only, and other actions or combinations may be selected and taken, such as switching on or off the media system, or changing the broadcasted materials.

On step256, various parameters and thresholds may be adapted, on an immediate or on long term basis, in order to provide better stress level estimation for the particular driver at the particular time. Learning may utilize the effect of previously taken actions in reducing the stress level, the changing of stress level over the day or another time period, the effect pf long drives, or the like.

In some embodiments, too low stress levels may also be identified and alerted, since they may indicate drowsiness or other lack of attention by the driver,

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). Each block may be implemented as a multiplicity of components, while a number of blocks may be implemented as one component. Even further, some components may be located externally to the car, for example some processing may be performed by a remote server being in computer communication with a processing unit within the vehicle. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.