Mobile device usage monitoring for commercial vehicle fleet management

A driver monitoring system according to an example of the present disclosure includes a camera configured to record images of a driver within a cabin of a vehicle. At least one vehicle sensor is configured to detect an anomalous driving event of the vehicle based on predefined criteria. A controller is in communication with the camera and the at least one vehicle sensor. The controller is configured to, based on the detection, obtain a particular image taken by the camera that depicts the driver during the anomalous event. The controller is configured to transmit the particular image to a fleet manager, store the particular image in a local repository of anomalous driving images, or both.

BACKGROUND

This application relates to driver monitoring, and more particularly to recording images depicting a driver during anomalous driving events.

Safety costs in the commercial vehicle space have been rising dramatically, with distracted driving being a primary cause of accidents. Cellular phone usage is believed to be a major contributor to these types of distracted driver accidents.

SUMMARY

A driver monitoring system according to an example of the present disclosure includes a camera configured to record images of a driver within a cabin of a vehicle. At least one vehicle sensor is configured to detect an anomalous driving event of the vehicle based on predefined criteria. A controller is in communication with the camera and the at least one vehicle sensor. The controller is configured to, based on the detection, obtain a particular image taken by the camera that depicts the driver during the anomalous event. The controller is configured to transmit the particular image to a fleet manager, store the particular image in a local repository of anomalous driving images, or both.

In a further embodiment of any of the foregoing embodiments, the controller is in communication with the at least one vehicle sensor through a controller area network bus of the vehicle.

In a further embodiment of any of the foregoing embodiments, the at least one vehicle sensor includes at least one of the following: an acceleration sensor configured to detect an anomalous acceleration event, a steering angle sensor configured to detect an anomalous steering event, a braking sensor operable to detect an anomalous braking event, and an object detection sensor operable to detect a near collision.

In a further embodiment of any of the foregoing embodiments, the controller is configured to adjust the predefined criteria based on at least one of traffic density, weather conditions, and object detection in the vicinity of the vehicle.

In a further embodiment of any of the foregoing embodiments, the controller is configured to determine the predefined criteria based on an experience level of the driver.

In a further embodiment of any of the foregoing embodiments, the controller is operable to obtain the particular image from a rolling video buffer recorded within a time window corresponding to the anomalous event.

In a further embodiment of any of the foregoing embodiments, the controller is configured to record additional images depicting the driver from the camera at random intervals, and transmit the additional images to a fleet manager, store the additional images in the local repository of anomalous driving images, or both.

A driver monitoring system according to an example of the present disclosure includes a gaze tracking camera configured to record images of a driver within a cabin of a vehicle and determine a gaze direction of the driver in the recorded images. A controller is in communication with the gaze tracking camera and is configured to detect a potential distracted driving event based on the gaze direction of the driver as depicted in a particular image of the recorded images being outside of a predefined alert driver area for an amount of time exceeding a predefined time threshold. The controller is configured to perform one or both of the following predefined actions based on the potential distracted driving event: transmission of the particular image to a fleet manager, and storage of the particular image in a local repository of anomalous driving images.

In a further embodiment of any of the foregoing embodiments, the controller is configured to, based on the detected potential distracted driving event, provide the particular image to a convolutional neural network that has been trained with images depicting drivers utilizing mobile devices, and determine, based on feedback from the convolutional neural network, whether the driver is utilizing a mobile device in the particular image. The performance of the predefined action is further based on the determination of whether the driver is utilizing a mobile device in the particular image.

In a further embodiment of any of the foregoing embodiments, the controller is configured to, based on a determination that the driver is not utilizing a mobile device in the particular image, omit performance of one or both of the predefined actions and further train the convolutional neural network using the particular image.

In a further embodiment of any of the foregoing embodiments, the controller is configured to transmit one or more of the following distracted driving alerts to the driver based on the potential distracted driving event: an audio notification through a vehicle speaker, an audio notification to a wireless headset worn by the driver, and a visual notification on an electronic display within the cabin.

In a further embodiment of any of the foregoing embodiments, the gaze tracking camera is a video camera, and the images are recorded in a video feed from the video camera.

In a further embodiment of any of the foregoing embodiments, the controller is configured to adjust one or both of the predefined time threshold and the predefined alert driver area based on at least one of traffic density, weather conditions, object detection external to the vehicle, and a geographic location of the vehicle.

In a further embodiment of any of the foregoing embodiments, the controller is configured to select one or both of the predefined time threshold and the predefined alert driver area based on an experience level of the driver.

In a further embodiment of any of the foregoing embodiments, the controller is configured to obtain additional images depicting the driver from the gaze tracking camera, or another camera, at random intervals, and transmit the additional images to a fleet manager, store the additional images in the local repository of anomalous driving images, or both.

A method of monitoring a driver according to an example of the present disclosure includes recording images of a driver within a cabin of a vehicle using a vehicle camera, detecting an anomalous driving event of the vehicle based on input from at least one vehicle sensor, obtaining a particular image from the camera depicting the driver during the anomalous event, and performing at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of anomalous driving images.

In a further embodiment of any of the foregoing embodiments, the detecting of the anomalous driving event of the vehicle includes detecting one of more of an anomalous acceleration event, an anomalous steering event, an anomalous braking event, and a near collision.

A method of monitoring a driver according to an example of the present disclosure includes recording images of a driver within a cabin of a vehicle, determining a gaze direction of the driver in the recorded images, and detecting a potential distracted driving event based on the gaze direction of the driver, as depicted in a particular image of the recorded images, being outside of a predefined alert driver area for an amount of time exceeding a predefined time threshold. The method includes, based on the detecting, performing at least one of transmitting the particular image to a fleet manager and storing the particular image in a local repository of anomalous driving images.

In a further embodiment of any of the foregoing embodiments, the method includes, based on the potential distracted driving event, providing the particular image to a convolutional neural network that has been trained with mobile device usage images depicting drivers utilizing mobile devices, and determining, based on feedback from the convolutional neural network, whether the driver is utilizing a mobile device in the particular image. The performance of the at least one predefined action is further based on the determination of whether the driver is utilizing a mobile device in the particular image.

In a further embodiment of any of the foregoing embodiments, the method includes, based on a determination that the driver is not utilizing a mobile device in the particular image, omitting performance of one or both of the predefined actions, and further training the convolutional neural network using the particular image.

A driver monitoring system according to an example of the present disclosure includes a wireless activity detector configured to detect signaling from a mobile device within a cabin of a vehicle, a camera configured to record images of a driver within the cabin of the vehicle, and a controller in communication with the camera and wireless activity detector. The controller is configured to, based on the detected signaling, obtain a particular image taken by the camera that depicts the driver, and transmit the particular image to a fleet manager, store the particular image in a local repository of anomalous driving images, or both.

A method of monitoring a driver according to an example of the present disclosure includes detecting wireless signaling from a mobile device within a cabin of a vehicle. The method includes, based on the detected wireless signaling: recording an image of a driver within the cabin of the vehicle, and transmitting the particular image to a fleet manager, store the particular image in a local repository of anomalous driving images, or both.

The embodiments, examples, and alternatives of described in the claims and in the following description and drawings, including any of their various aspects or respective individual features, may be taken independently or in any combination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are incompatible.

DETAILED DESCRIPTION

FIG.1schematically illustrates an example fleet management system10that includes a fleet12of vehicles14A-N operable to communicate with a fleet manager22through a wide area network (“WAN”)16, such as the Internet. The vehicles14are operable to record images depicting drivers of the vehicles14, and to store or transmit those images, optionally along with associated event data describing how the vehicles14are being operated (e.g., acceleration events, steering events, braking events, near collisions, etc.).

In one example, the vehicles14A-N transmit the images and/or event data to the fleet manager22by transmitting the images to a fleet management server18, where they can be accessed by a computing device20of the fleet manager22that supervises the fleet12. In one example, the vehicles14A-N can transmit the images and/or event data to the fleet manager22by transmitting to the computing device20of the fleet manager22, bypassing the fleet management server18. In one example, in addition to or as an alternative to transmitting the images to the fleet manager22, the vehicles14store the images in a local repository in the vehicles14. In one example, whether a given image is transmitted via the WAN16or is stored in the local repository is based on whether the vehicle14currently has connectivity to the WAN16. In the example ofFIG.1, the vehicles14are trucks, but it is understood that other commercial vehicles could be used, such as delivery vans and the like.

FIG.2schematically illustrates components of an example driver monitoring system24provided in each vehicle14. In the example ofFIG.2, the driver monitoring system24includes an electronic control unit (ECU)30which is operably connected to a telematics module32, a cabin camera34, an acceleration sensor36, a steering angle sensor38, and a braking sensor40. Although three sensors34-38are described, it is understood that fewer or more sensors could be used. For example, the ECU30may be operably connected to an exterior camera42operable to record images of a surrounding environment of the vehicle14, an object detection sensor44operable to detect objects exterior to the vehicle14, a wireless activity detector45operable to detect wireless device usage by a driver, an electronic display46, a vehicle speaker48, and/or a Bluetooth module50.

In one example, the electronic display46and speaker48are part of a driver information system (“DIS”) that provides information about a vehicle status (e.g., speed, engine RPMs, etc.). In this example, the electronic display46could be part of a vehicle instrument cluster. As another example, the electronic display46could be a center console display that is part of an infotainment system that provides a combination of vehicle information and entertainment information (e.g., current radio station, climate control, and the like). In one example, the ECU30is integrated into a DIS ECU (not shown) or the telematics module32.

In the example ofFIG.2, the ECU30is operably connected to the components32-50through a vehicle data bus52, which may be a controller area network (“CAN”) bus. Of course,FIG.2is only an example, and it is understood that the ECU could connect to certain ones of the components32-50through other connections besides the vehicle data bus52.

FIG.3schematically illustrates the ECU30in greater detail. Referring now toFIG.3, the ECU30includes a processor60operatively connected to memory62, and a communication interface64. The processor60includes one or more processing circuits, such as microprocessors, microcontrollers, application specific integrated circuits (ASICs), or the like. The memory62may include one or several types of memory such as read-only memory (ROM), random-access memory, cache memory, flash memory devices, optical storage devices, etc. The memory62includes a local repository66of anomalous driving images, and optionally may also include a convolutional neural network (“CNN”)67, a driver attention model68, and/or a driver gaze model69. The CNN67is operable to detect whether a driver is utilizing a mobile device in a cabin of the vehicle14. As used herein, a “mobile device” refers to a handheld electronic device such as a cellular phone, smartphone, tablet, personal media player, or the like. Although depicted as being part of the ECU30, it is understood that the CNN67could instead be stored external to the vehicle14, such as in the fleet management server18. The communication interface64provides for communication between the ECU30and other components (e.g., a wired connection to the vehicle data bus52).

Referring now toFIG.2, with continued reference toFIG.3, the cabin camera34, which may be a gaze tracking camera, is configured to record images of a driver within a cabin of the vehicle14, and each of the sensors36-44is configured to detect an anomalous driving event of the vehicle based on predefined criteria corresponding to distracted driving.

Based on detection of an anomalous driving event by one of the sensors36-44, the ECU30is operable to obtain a particular image taken by the cabin camera34that depicts the driver during the anomalous event. The ECU30transmits the particular image to the fleet manager22using the telematics module32and/or stores the particular image in the local repository66of anomalous driving images.

The acceleration sensor36is configured to detect an anomalous acceleration event, such as rapid acceleration or deceleration of the vehicle14, which could be indicative of distracted driving. The predefined criteria for the acceleration sensor36could include a rate of acceleration above a predefined acceleration threshold or a rate of deceleration below a predefined deceleration threshold, for example.

The steering angle sensor38is configured to detect an anomalous steering event, such as a rapid steering wheel angle change that could be indicative of swerving. For example, the predefined criteria for the steering angle sensor38could include a change in steering angle beyond a predefined angle threshold within a predefined time period while the vehicle14is traveling at a speed above a predefined speed threshold, which could be indicative of swerving as a result of distracted driving.

The braking sensor40is configured to detect an anomalous braking event, such as rapid braking of the vehicle14, and could be configured to measure changes in vehicle velocity, and/or control signals transmitted to a vehicle braking system, for example.

The object detection sensor44may be a LIDAR (“light detection and ranging”) or RADAR (“radio detection and ranging”) sensor, for example. The object detection sensor44can be used on its own, or in conjunction with the ECU30, to detect near-collision incidents in which a collision was narrowly avoided.

The telematics module32includes a wireless transceiver operable to transmit images through the WAN16. In one example, the telematics module32is configured to use a predefined protocol standard such as one or more of the 802.11 standards and/or one or more cellular standards (e.g., GSM, CDMA, LTE, etc.).

The wireless activity detector45includes an antenna configured to detect radio signals, and includes related processing circuitry for determining if a detected radio signal represents mobile device usage within a cabin of the vehicle14based on one or more predefined thresholds. The criteria used by the processing circuitry of wireless activity detector45could include any one or combination of the following: signal strength, signal duration, and mobile device identifier. Some example mobile device identifiers could include an international mobile subscriber identity (“IMSI”), an Internet protocol (“IP”) address, a media access control (“MAC”) address, and if a mobile device identifier associated with the driver is detected, it is more likely that the signal transmission corresponds to mobile device usage by the driver and not a pedestrian or driver of a nearby vehicle.

In one example, signal duration is used to distinguish between background activity, such as handovers between adjacent cells, where a driver is not actually using the mobile device, and active use of the mobile device (e.g., phone calls, video streaming, etc.) where the signal duration is more likely to exceed a predefined signal length threshold.

In one example, the wireless activity detector45is configured to limit its monitoring to frequency bands linked to known telecommunication standards, such as GSM band(s), CDMA band(s), LTE band(s), WiMax band(s), WiFi band(s), etc. In one example, the wireless activity detector45includes a plurality of antennas, each tuned for a particular one or set of frequency bands, and/or includes one or more antennas configured to sweep a plurality of such frequency bands.

In one example, the wireless activity detector45is configured to base its detection at least in part on signal strength, as a signal detected from a mobile device in the vehicle cabin is likely to be stronger than that of a mobile device in a neighboring vehicle.

In one example, the cabin camera34is a video camera operable to provide a rolling buffer of a predefined duration (e.g., 30 seconds) that overwrites itself if not backed up, and the ECU30is operable to obtain images from frames of the rolling video buffer within a time window corresponding to an anomalous driving event. This could also provide an opportunity to preserve video leading up to an anomalous driving event to see what occurred during and prior to the anomalous driving event.

In one example, the ECU30is configured to record additional images depicting the driver from the cabin camera34at random intervals which may occur outside of anomalous driving events, and transmit the additional images to the fleet manager22, store the additional images in the local repository66of anomalous driving images, or both. This random sampling could provide an additional level of compliance for the driver.

In one example, the ECU30is configured to adjust the predefined criteria used for determining anomalous driving events based on at least one of traffic density, weather conditions, and object detection in the vicinity of the vehicle14. For example, in adverse weather conditions (e.g., rain, snow, icy roads) and or high-traffic or high-pedestrian areas, the thresholds used for determining what constitutes anomalous driving may be lowered from default values to a more stringent standard, particularly when the vehicle14is a large commercial truck.

The determination of whether weather conditions are adverse could be based on a weather forecast received at the ECU30, for example. The determination of whether the vehicle14is in a high-traffic or high-pedestrian area could be based on, e.g., a traffic report received at the ECU30and/or based on object detection from the object detection sensor44or exterior camera42.

Also, the predefined criteria used to detect anomalous driving event could be selected based on an experience level of a driver. This could provide for more stringent standards for less-experienced drivers, and more tolerant standards for experienced drivers.

FIG.4is a flowchart of an example method100of monitoring a driver. The ECU30monitors one or more vehicle sensors (e.g., sensors36-44) for anomalous driving events (step102). If no anomalous driving event is detected (a “no” to step104), the ECU30keeps monitoring for anomalous driving events. If an anomalous driving event is detected (a “yes” to step104), the ECU30obtains a particular image from the cabin camera34that depicts the driver during the anomalous event (step106). The ECU30transmits the particular image to the fleet manager22and/or stores the image in the local repository66of anomalous driving images, and then resumes monitoring the vehicle sensors for anomalous driving events (step102).

In some embodiments, the cabin camera34is a gaze tracking camera configured to record images of a driver within a cabin of the vehicle14and determine a gaze direction of the driver in the recorded images. Such cameras are commercially available from SmartEye (https://smarteye.se/) and EyeSight (http://w ww.eyesight-tech.com/). In one example, the cabin camera34detects gaze by directing infrared or near-infrared light to a user's eye, and then measuring the reflection of that infrared light back from the driver's eye. Based on the angle of reflection, a gaze direction can be ascertained. In another example, the cabin camera34infers the gaze direction of the driver by determining a gaze vector from the general shape of the driver's head and/or the symmetry of the driver's face in a recorded image. Both of these techniques are well known to those of ordinary skill in the art, and therefore are not discussed in detail herein. In one example, the cabin camera34is integrated into a driver information system and/or instrument cluster.

FIG.5is a flowchart of an example method200of monitoring a driver in which the cabin camera34is a gaze tracking camera, and the ECU30utilizes a CNN67. The cabin camera34records images of a driver within a cabin of the vehicle14(step202), and determines a gaze direction of the driver in the recorded images (step203). The ECU30determines whether the gaze is outside of a predefined alert driver area for an amount of time that exceeds a predefined time threshold (step204).

Referring now toFIG.6, an example vehicle cabin70is schematically shown, along with an example predefined alert driver area72, which includes a windshield74, rearview mirror76, wing mirrors78A-B, and instrument cluster display46, but excludes center console display46B, and other areas that are likely to be indicative of distracted driving and/or mobile device usage, such as the driver's lap area80, the passenger seat82, the majority of the driver and passenger windows84A-B, etc.

In the example ofFIG.6, camera monitor system units86A-B, which are mounted to pillars92A-B, are provided within the alert driver area72. Each camera monitor system unit86A-B includes a respective electronic display88A-B for providing an external vehicle video feed, and may optionally also include a respective camera90A-B (which optionally may be used as the cabin camera34ofFIG.2if desired). Utilizing the cameras90A-B could be useful for retrofitting aspects of the driver monitoring system24into existing vehicles, where modifications to the existing software for an existing vehicle camera may be impractical. In one example, the camera monitor system units86A-B are part of the MIRROREYE system from Stoneridge, Inc.

Referring again toFIG.5, with continued reference toFIG.6, if the driver's gaze is within the alert driver area72, or is only outside of the alert driver area72for an amount of time less than the predefined time threshold (a “no” to step204), then the ECU30resumes monitoring the gaze direction of the driver.

Conversely, if the ECU30determines in step204that the driver's gaze is outside of the alert driver area72for an amount of time greater than the predefined time threshold (a “yes” to step204), then the ECU30determines that the gaze corresponds to a potential distracted driving event (step206), and provides a particular image of the driver when the driver's gaze direction is outside of the alert driver area72to the CNN67(step208).

The CNN67is trained with images depicting drivers utilizing mobile devices, and the ECU30utilizes the CNN67to process the image and determine if the driver is utilizing a mobile device in the particular image (step210). The training image set could include images of drivers texting, talking on the phone while holding the phone up to their face, having a hand near their face in a position suggesting phone usage (even if a phone is not visible), etc. Use of the CNN67helps to reduce false positive detections of distracted driver events.

If mobile device usage is detected (a “yes” to step212), the ECU30performs one or more predefined actions based on the potential distracted driving event, such as transmitting the image to the fleet manager22for review, storing the image for eventual review in the local repository66of anomalous driving, and/or providing an alert to the driver (step214). The alert could be provided as an audio notification through vehicle speaker48, as an audio notification to a wireless headset worn by the driver (e.g., using Bluetooth module50), and/or a visual notification on the electronic display46within the cabin70, for example. The particular image can then be used as additional training data for the CNN67(step216) as part of a supervised machine learning process.

Alternatively, if mobile device is not detected (a “no” to step212), one or more (e.g., all) of the predefined actions are omitted, and the ECU30proceeds to step216.

In a similar fashion to how the ECU30can adjust the anomalous driving detection thresholds, the ECU30can also adjust the thresholds use for determining when a driver's gaze is indicative of distracted driving and/or mobile device usage. For example, in certain environments, such as adverse weather conditions (e.g., rain, snow, icy roads) and or high-traffic or high-pedestrian areas, the alert driver area72could be narrowed from a default area and/or the time threshold used in step204could be shortened from a default value to enforce a more stringent level of driver attention. Conversely, in low-traffic and/or favorable weather conditions (i.e., non-icy, non-snowy, non-slippery) the alert driver area72could be expanded and/or the time threshold of step204could be lengthened.

In one example, the ECU30is configured to select one or both of the predefined time threshold of step204and the alert driver area72based on an experience level of the driver. This could provide for more stringent standards for less-experienced drivers, and more tolerant standards for experienced drivers

Similarly, the threshold for when an alert is provided to a driver in step214could be selected based on driver experience level, with the understanding that such warnings may be more appropriate and/or useful for drivers with less experience than they would be for drivers with more experience.

As discussed above, the ECU may include a driver attention model68and/or a driver gaze model69. Use of such models could provide an additional layer of refinement for the driver monitoring system24, for example, by correlating gaze direction with external objects and determining if drivers are fixating on certain objects when they should be gazing at other relevant objects in the area surrounding the vehicle. For example, the rapid movement of gaze direction from side to side could indicate a distracted driving event, even though the driver's gaze is focused externally to the vehicle (as their attention is not fixated on any situation long enough to indicate driver attention). Furthermore, the attention model indicating extended focus on a point near the driver's knee could indicate that the driver is utilizing the phone to access the internet while driving.

FIG.7is a flowchart of another example method300of monitoring a driver in which the wireless activity detector45is used to monitor wireless signal transmissions based on any of the predefined criteria discussed above (e.g., signal strength, signal duration, and mobile device identifier) (step302). If wireless signaling from a driver mobile device is detected (a “yes” to step303), an image is recorded of the driver within the vehicle cabin70, and the image is provided to the CNN67(step308). Steps310-316are performed in the same manner as steps210-216as described above.

In one example, the CNN67is omitted, the ECU30just transmits the recorded image in step306for review, or stores the image for eventual review.

In one example, if the driver's mobile device is paired with the vehicle infotainment system or a headset (e.g., through Bluetooth), steps306-316are skipped. In one example, the wireless activity detector45detects whether the mobile device is paired with a headset and/or the infotainment system by monitoring for transmissions on a Bluetooth frequency band.

In one example, steps302-303are used in the method200as an additional layer of detection prior to utilizing the CNN67(e.g., between steps206and208, such that the particular image is only provided to the CNN67if the wireless activity detector45corroborates the potential distracted driving event by detecting evidence of wireless signaling from a driver mobile device.

Although example embodiments have been disclosed, a worker of ordinary skill in this art would recognize that certain modifications would come within the scope of this disclosure. For that reason, the following claims should be studied to determine the scope and content of this disclosure.