Patent Description:
<CIT> describes a system for quantifiable assessment of vehicle driver performance based upon objective standards. The physical and/or control states of a vehicle are monitored by sensors during a driving trip. Measurement data, optionally comprising a measurement signal, is composed from parameters selected from the measured physical and/or control states. The measurement data is then compared to reference data, optionally comprising a reference signal, comprising the same or similar physical and control state parameters, for the same or analogous driving trip or portion thereof, including discrete driving tasks, as determined by one or more of: a known driver of specific attributes, a population average, or an autonomous driving algorithm. A metric of comparison may be determined as one or more characteristic metrics of a driving task, according to one or more path metrics of a driving task, or as a signal distance metric between the reference and measurement signals.

<CIT> discloses a further system for autonomous driving comparison and evaluation.

Autonomous driving comparison and evaluation is disclosed. A system for autonomous driving comparison and evaluation comprises an input interface configured to receive trip information generated during a driver controlled trip and receive information from an autonomous driving system related to actions that the autonomous driving system would have taken had it been in control during the trip; and a processor configured to compare a trip factor of the driver controlled trip with a simulated trip factor that would have occurred had the autonomous driving system been in control. The system for autonomous driving comparison and evaluation additionally comprises a memory coupled to the processor and configured to provide the processor with instructions.

In embodiments of the present invention, a vehicle event recorder mounted on a vehicle records vehicle data and anomalous vehicle events. Anomalous vehicle event types include accidents, speed limit violations, rough road events, hard maneuvering events (e.g., hard cornering, hard braking), dangerous driving events (e.g., cell phone usage, eating while driving, working too long of a shift, sleepy driving, etc.), and any other appropriate kind of anomalous vehicle events. The vehicle event recorder analyzes data from sensors (e.g., video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, radar, ADAS, etc.) to determine when an anomalous event has occurred. The vehicle event recorder transmits event data, including sensor data, to a vehicle data server, where the data is stored and analyzed. The vehicle event recorder additionally comprises a simulated driver assistance system. The simulated driver assistance system is designed to simulate driver assistance and autonomous control systems as a baseline of autonomous driving performance based on data received from the installed driver assistance and autonomous control sensors and systems installed in the vehicle. In examples not according to the claimed invention, the installed driver assistance system inputs are comprised of electronic stability control, braking assistance, adaptive cruise control, automatic lane following, automatic steering and navigation, full automated driving, or any other appropriate driver assistance system. In examples not according to the claimed invention, the installed driver assistance system comprises an autonomous driver assistance system. In examples not according to the claimed invention, the installed driver assistance system comprises a set of Advanced Driver Assistance Systems (ADAS). During normal vehicle operation, the vehicle driver controls the vehicle, and signals from the installed driver assistance system may or may not be ignored by the driving subsystems of the vehicle depending on the driver enabling or disabling these systems. Regardless of the installed driver assistance systems outputs being utilized by the driving subsystems of the vehicle, signals from the installed driver assistance systems are processed by vehicle event recorder simulated driver assistance system and the signals are used as a reference against which the driver's actions can be compared. A driver score for the trip can be determined based on the comparison of the driver actions with the simulated driver assistance system actions. In examples not according to the claimed invention, any deviation of the driver's actions from the signals of the driver assistance system causes a reduction of driver score. In examples not according to the claimed invention, a trip factor (e.g., travel time, fuel economy, trip safety, etc.) is determined for the trip and compared with a simulated trip factor for the trip indicated by the signals of the simulated driver assistance system. The trip score is determined based on the ratio of the trip factor and the simulated trip factor. The driver score is computed as an aggregate of trip scores - for example, an average or a weighted average of trip scores over many trips. In various embodiments, the weighted average includes weighting to decay the contributions with time (e.g., more recent trips are weighted more than older trips, for example the last week, is weighted more than the previous week and much more than last month or <NUM> months ago), weighting to emphasize longer trips, shorter trips, trips during certain periods of the day (e.g., working hour trips vs. non-working hour trips, etc.), weighting to emphasize customer priority (e.g., based on riskiness - for example, statistically riskier such as due to bad weather, heavy traffic, higher number of ADAS warnings, based on fuel efficiency, etc.), or any other appropriate weighting.

In examples not according to the claimed invention, a driver is benchmarked against the installed driver assistance systems (e.g., lane keep system and following distance system). In examples not according to the claimed invention, the driver is benchmarked against a simulated driver assistance systems based on data provided by the installed driver assistance components and sensors (e.g., simulate a lane keep system based on lane detection and tracking (LDAT) data from a lane departure warning (LDW) system).

<FIG> is a block diagram illustrating an embodiment of a system including a vehicle event recorder. Vehicle event recorder <NUM> comprises a vehicle event recorder mounted in vehicle <NUM> (e.g., a car or truck). In some embodiments, vehicle event recorder <NUM> includes or is in communication with a set of sensors - for example, video recorders, audio recorders, accelerometers, gyroscopes, vehicle state sensors, GPS, outdoor temperature sensors, moisture sensors, laser line tracker sensors, radar, or any other appropriate sensors. In various embodiments, vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine revolutions per minute (e.g., RPM) sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors. In some embodiments, vehicle event recorder <NUM> comprises a system for processing sensor data and detecting events. Vehicle event recorder <NUM> can comprise map data. In examples not according to the claimed invention, vehicle event recorder <NUM> comprises a system for detecting risky behavior. Vehicle event recorder <NUM> can be mounted to vehicle <NUM> in one of the following locations: the chassis, the front grill, the dashboard, the rear-view mirror, or any other appropriate location. In examples not according to the claimed invention, vehicle event recorder <NUM> comprises multiple units mounted in different locations in vehicle <NUM>. Vehicle event recorder <NUM> comprises a communications system for communicating with network <NUM>. Network <NUM> can comprise a wireless network, a wired network, a cellular network, a Code Division Multiple Accessing (CDMA) network, a Global System for Mobile (GSM) communications network, Wideband Code Division Multiple Access (W-CDMA), Long Term Evolution (LTE), a local area network, a wide area network, the Internet, or any other appropriate network. In examples not according to the claimed invention, network <NUM> comprises multiple networks, changing over time and location. Vehicle event recorder <NUM> communicates with vehicle data server <NUM> via network <NUM>. Vehicle event recorder <NUM> is mounted on vehicle <NUM>. In various embodiments, vehicle <NUM> comprises a car, a truck, a commercial vehicle, or any other appropriate vehicle. Vehicle data server <NUM> comprises a vehicle data server for collecting events and risky behavior detected by vehicle event recorder <NUM>. In some embodiments, vehicle data server <NUM> comprises a system for collecting data from multiple vehicle event recorders. In some embodiments, vehicle data server <NUM> comprises a system for analyzing vehicle event recorder data. In examples not according to the claimed invention, vehicle data server <NUM> comprises a system for displaying vehicle event recorder data. In examples not according to the claimed invention, vehicle data server <NUM> is located at a home station (e.g., a shipping company office, a taxi dispatcher, a truck depot, etc.). In examples not according to the claimed invention, events recorded by vehicle event recorder <NUM> are downloaded to vehicle data server <NUM> when vehicle <NUM> arrives at the home station. In some embodiments, vehicle data server <NUM> is located at a remote location. In some embodiments, events recorded by vehicle event recorder <NUM> are downloaded to vehicle data server <NUM> wirelessly (e.g., the home station case is wired, private wireless - for example, WiFi, etc.). In examples not according to the claimed invention, network <NUM> comprises a public generally available network (e.g., a WiFi, a cellular network, or satellite network). In examples not according to the claimed invention, a subset of events recorded by vehicle event recorder <NUM> is downloaded to vehicle data server <NUM> wirelessly. Vehicle <NUM> additionally comprises other vehicle systems <NUM> in communication with vehicle event recorder <NUM> (e.g., sensor systems, user interface systems, driving subsystems, driver assistance systems, etc.).

<FIG> is a block diagram illustrating an embodiment of a vehicle event recorder. In some embodiments, vehicle event recorder <NUM> of <FIG> comprises vehicle event recorder <NUM> of <FIG>. In the example shown, vehicle event recorder <NUM> comprises processor <NUM>. Processor <NUM> comprises a processor for controlling the operations of vehicle event recorder <NUM>, for reading and writing information on data storage <NUM>, for communicating via wireless communications interface <NUM>, for determining a position using global positioning system <NUM>, and for reading data via sensor interface <NUM>. Data storage <NUM> comprises a data storage (e.g., a random access memory (RAM), a read only memory (ROM), a nonvolatile memory, a flash memory, a hard disk, or any other appropriate data storage). In various embodiments, data storage <NUM> comprises a data storage for storing instructions for processor <NUM>, vehicle event recorder data, vehicle event data, sensor data, video data, map data, or any other appropriate data. In various embodiments, wireless communications interface <NUM> comprises one or more of a Global System for Mobile (GSM) interface, a Code Division Multiple Accessing (CDMA) interface, a Wideband Code Division Multiple Access (WCDMA) interface, a Long Term Evolution (LTE) interface, a WiFi interface, or any other appropriate interface. Global positioning system <NUM> comprises a global positioning system (e.g., GPS) for determining a system location. Sensor interface <NUM> comprises an interface to one or more vehicle event recorder sensors. In various embodiments, vehicle event recorder sensors comprise an external video camera, an internal video camera, a microphone, an accelerometer, a gyroscope, an outdoor temperature sensor, a moisture sensor, a laser line tracker sensor, vehicle state sensors, ADAS, interface to front radar, back and side radar or any other appropriate sensors. In various embodiments, vehicle state sensors comprise a speedometer, an accelerator pedal sensor, a brake pedal sensor, an engine RPM sensor, an engine temperature sensor, a headlight sensor, an airbag deployment sensor, driver and passenger seat weight sensors, an anti-locking brake sensor, an engine exhaust sensor, a gear position sensor, a cabin equipment operation sensor, or any other appropriate vehicle state sensors. In various embodiments, sensor interface <NUM> comprises an on-board diagnostics (OBD) bus (e.g., society of automotive engineers (SAE) J1939, J1708/J1587, On-Board Diagnostics (OBD)-II, controller area network (CAN) BUS, etc.), interface to a powertrain control module (e.g., a PCM), or any other appropriate interface. In some embodiments, vehicle event recorder <NUM> communicates with vehicle state sensors via OBD bus. Simulated driver assistance system <NUM> is designed to simulate driver assistance and autonomous control systems as a baseline of autonomous driving performance based on data received from the installed driver assistance and autonomous control sensors and systems installed in the vehicle. Driver performance is measured in comparison to the baseline. In various embodiments, driver assistance systems comprise one or more of the following: a lane keeping system, a following distance system, or any other appropriate driver assistance system.

<FIG> is a block diagram illustrating an embodiment of subsystems of a vehicle. In some embodiments, vehicle <NUM> comprises vehicle <NUM> of <FIG>. In the example shown, vehicle <NUM> comprises vehicle event recorder <NUM>. In some embodiments, vehicle event recorder <NUM> comprises vehicle event recorder <NUM> of <FIG>. Vehicle event recorder <NUM> receives sensor data from sensors <NUM>. In various embodiments, vehicle event recorder <NUM> processes sensor data to determine occurrence of anomalous events, to determine when installed driver assistance systems <NUM> should be activated, to determine when video data should be recorded/stored, to evaluate the risk associated with the data, or for any other appropriate reason. The risk indicates distracted driving, drowsy driving, intoxicated driving, aggressive driving, a legal infraction, icy roads, poor visibility, rain, failing brakes, a damaged tire, or any other appropriate risky situation. In some embodiments, vehicle event recorder <NUM> includes driver assistance algorithms to simulate equivalent performance to a vehicle being controlled using installed driver assistance systems <NUM>. Sensor data from sensors <NUM> are passed to installed driver assistance systems <NUM> for processing. Installed driver assistance systems <NUM> process sensor data to determine driving subsystems control information. Installed driver assistance systems <NUM> provides driving subsystems control information to driving subsystems <NUM> to control the driving elements of the vehicle. Installed driver assistance systems <NUM> comprise one or more systems for automatically assisting the driver. In examples not according to the claimed invention, installed driver assistance systems <NUM> comprise electronic stability control, braking assistance, adaptive cruise control, automatic lane following, automatic steering and navigation, full automated driving, or any other appropriate driver assistance systems. In examples not according to the claimed invention, installed driver assistance systems <NUM> comprise autonomous driver assistance systems. In examples not according to the claimed invention, installed driver assistance systems <NUM> comprise a set of Advanced Driver Assistance Systems (ADAS). In examples not according to the claimed invention, installed driver assistance systems <NUM> provide driving subsystems control information to vehicle event recorder <NUM> for processing.

Driving subsystems <NUM> comprise subsystems controlling the various driving elements of the vehicle (e.g., acceleration, brakes, steering, etc.). Driving subsystems <NUM> receive commands from the vehicle controls (e.g., accelerator pedal, brake pedal, steering wheel, etc.) and provide commands to the driving elements of the vehicle to operate the vehicle as indicated by the driver. Driving subsystems <NUM> provides vehicle control commands received from the driving elements of the vehicle to vehicle event recorder <NUM> for processing. In examples not according to the claimed invention, vehicle event recorder uses driving subsystems control information received from installed driver assistance systems <NUM> and vehicle control commands received from driving subsystems <NUM> to compute a trip score. In examples not according to the claimed invention, when a trip is complete and a trip score has been computed, the trip score is displayed to the user via user interface <NUM>. In various embodiments, a trip score is calculated on vehicle event recorder <NUM> or a vehicle data server based at least in part on data collected by vehicle event recorder <NUM>, any applicable backend data contextual data (e.g., traffic details, weather data, etc.), or any other appropriate data. In examples not according to the claimed invention, if vehicle event recorder <NUM> detects a large or significant difference between the driving subsystems control information and the vehicle control commands, it determines that an anomalous event has occurred. In examples not according to the claimed invention, the anomalous event is stored. In examples not according to the claimed invention, the anomalous event is transmitted to a vehicle data server for later review.

<FIG> is a flow diagram illustrating an embodiment of a process for autonomous driving comparison and evaluation. In examples not according to the claimed invention, the process of <FIG> is used by vehicle event recorder <NUM> of <FIG> to compute a trip score and a driver score. In the example shown, in <NUM>, trip information is received. In examples not according to the claimed invention, trip information comprises vehicle control commands (e.g., vehicle control commands received from driving subsystems). In <NUM>, driver assistance systems information is received. In examples not according to the claimed invention, driver assistance systems information comprises driving subsystems control information received from driver assistance systems. In <NUM>, a trip factor for the trip information (e.g., the trip information received in <NUM>) is determined. In various embodiments, a trip factor comprises trip time, trip fuel consumption, trip safety, human fatigue factors (e.g., hours of service (HOS) regulations), or any other appropriate trip factor. In examples not according to the claimed invention, determining trip safety comprises determining the safety of the trip route, determining the safety of the trip speed, determining the safety of the driving style (e.g., tailgating, hard acceleration, hard cornering, hard braking, following too close, not following the lane, etc.), determining safety violations (e.g., eating while driving, cell phone usage, etc.), determining if autonomous driving should have been engaged (e.g., prior to making a phone call), or determining any other appropriate trip safety information. In <NUM>, a trip factor is determined using simulated trip information. For example, the simulated trip information is determined based on the driver assistance systems information received indicating different driving from the actual driving and using this information to simulate the trip information. In examples not according to the claimed invention, determining a trip factor comprises determining a route indicated by the driver assistance systems, determining an average speed indicated by the driver assistance systems, determining a relationship between fuel consumption and speed, determining a route safety, or determining any other appropriate trip factor information. In <NUM>, a trip score based at least in part on the trip factor for the trip information and the trip factor for the autonomous driving system information is determined. In some embodiments, the trip score comprises the ratio of the trip factor for the trip information and the trip factor for the autonomous driving system information. In <NUM>, a new driver score is determined based at least in part on the trip score and on an old driver score. In examples not according to the claimed invention, the new driver score comprises an average of trip scores. In examples not according to the claimed invention, the new driver score comprises a weighted average of trip scores.

In various embodiments, trip factors comprise safety factors, efficiency factors, and compliance factors. In comparison, regulatory compliance comprises safety factors and efficiency factors. In various embodiments, speed limit is relevant to safety and fuel efficiency, HOS is relevant to efficiency, fatigue is relevant to safety factors, or any other appropriate factors.

<FIG> is a flow diagram illustrating an embodiment of a process for determining a trip factor for a simulation route. The process of <FIG> is used to implement <NUM> of <FIG>. In the example shown, in <NUM> a trip start location and a trip stop location is determined. In <NUM>, it is determined whether the simulator follows the same route as the actual route from trip start location to trip stop location. For example, a routing algorithm calculates or determines a route from the trip start location to the trip stop location. The route is checked against the actual trip route. In the event that the route is not the same as the actual route, in <NUM> the simulation route is set to calculated route, and control passes to <NUM>. In the event that the route is the same as the actual route, then in <NUM>, the simulation route is set to be the actual route. In <NUM>, it is determined whether any anomalous events occurred on the actual route. For example, an anomalous event determination might come from driver assistance system information. In various embodiments, anomalous events include one or more of the following: weather, accidents, construction, traffic, an event that the driver had to contend with (e.g., a pedestrian, a cyclist, a near collision, etc.). In some embodiments, a subset of anomalous events (e.g., route context) is factored into the simulated route if the data is available real time (e.g., route selection based on real time traffic data). In the event that no anomalous events occurred on the actual route, control passes to <NUM>. In the event that anomalous events occurred on the actual route, in <NUM> it is indicated that anomalous events occurred to the simulator, and control passes to <NUM>. In <NUM>, simulated trip information is determined for the simulation route. For example, the simulated trip information is determined using an automated driver system driving the simulation route and including any indicated anomalous events. In <NUM>, a trip factor is determined for the simulation route. For example, a trip factor (e.g., driver safety, fuel efficiency, time efficiency, etc.) are determined for the simulation route.

In examples not according to the claimed invention, simulated trip information is determined for the actual route only, and no determination is made whether other routes could/should have been taken. In examples not according to the claimed invention, no account is taken of anomalous events along the actual route, and the simulation of trip event assumes normal driving conditions.

Claim 1:
A system for autonomous driving comparison and evaluation, comprising:
a vehicle event recorder configured to:
receive (<NUM>) trip information generated during a driver controlled trip;
receive simulated driver assistance systems information (<NUM>) from a simulated driver assistance system (<NUM>) relating to actions that an installed driver assistance system (<NUM>) would have taken had it been in control during the driver controlled trip; and
wherein the vehicle event recorder comprises a processor (<NUM>) configured to:
determine simulated trip information using the simulated driver assistance systems information (<NUM>);
determine an actual trip factor (<NUM>) based at least in part on the trip information;
determine a simulated trip factor (<NUM>) based at least in part on the simulated trip information;
characterized in that:
determining the simulated trip information comprises determining whether the simulator follows a same route as an actual route.