Autonomous vehicle performance optimization system

An autonomous vehicle (AV) performance optimization system can receive vehicle data from human-driven vehicles, and determine a set of performance metrics for determining AV performance in relation to human performance based on the vehicle data. Thereafter, the system can receive AV data from AVs operating or configured for operation throughout a given region. Based on the AV data, the system can determine a performance score for each of the set of performance metrics for the AV.

BACKGROUND

Autonomous driving technology requires continuous sensor data processing of a situational environment of the self-driving vehicle (SDV). One limiting factor in the rollout of SDVs on public roads and highway is uncertainty with respect to the overall safety of such vehicles in normal driving situations as well as any contingency scenario. Traditional safety concepts may be necessary but insufficient for the SDV domain. For instance, SDV systems are extremely complex, having a very large and intricate code base that may be impractical to rigorously validate.

Additionally, the range of possible situations, interactions, and scenarios is virtually infinite, so safety guarantees for every conceivable scenario may not be possible. Thus, a rigorous design-based safety concept, which may often be preferred for typical software projects, may be impractical for extremely complex systems running in arbitrary, random, uncontrolled, and changing environments. Yet, in order to provide the necessary safeguards for SDV operation on public roads and highways, SDV manufacturers and operators must have a means for measuring and understanding just how safe an SDV is in its operation.

DETAILED DESCRIPTION

An autonomous vehicle (AV) or self-driving vehicle (SDV) performance measurement system is disclosed herein to assess the overall safety and/or performance of the SDV. According to some examples, the performance measurement system can includes a plurality of sensors, such as one or more LIDARs, cameras, inertial measurement units (IMUs), and/or global positioning system (GPS) receivers, and processing resources and memory resources for processing and storing performance data of a vehicle. Furthermore, an AV performance optimization system (e.g., either a backend remote system from the SDV or an on-board system) is provided herein that can utilize the performance data and a set of quantitative metrics to determine an overall safety and performance of the SDV. In variations, the performance measurement system can further include a housing in which various components can be housed, and can be mounted onto human-driven as well as autonomously driven vehicles. For example, the performance measurement system can be included as a single, bolt on pod mountable to the roof, or to the roof racks of a vehicle, or elsewhere on the body of the vehicle. In variations, the performance measurement system can be integrated with the SDV control system, utilizing existing sensors of the SDV to gather and store performance data.

In some examples, the performance measurement system can further include a wireless communication interface to transmit and/or receive data. In such examples, the performance measurement system can be communicatively linked to a performance optimization system, which can analyze the performance data in light of a set of quantitative metrics used to gauge the safety and performance of the vehicle. In an initial phase, the performance measurement system can be utilized with human driven vehicles to establish a set of safety, performance, and/or comfort thresholds—based on the set of quantitative metrics—that an SDV must meet in order to be adequate for public road operation. In certain implementations, the performance measurement system can be connected (e.g., via a data bus) to the SDV control system to correlate the gathered performance data with SDV actions.

Example comparison metrics can include braking, acceleration, steering, traffic law compliance, lane changing, turning, proximity, and/or following metrics. For example, the performance optimization system can analyze the performance data based on traffic law compliance metrics such as, speeding, driving too slow, slowing or stopping abruptly, stopping past a traffic line, crossing onto a shoulder, crossing a center line, running a red traffic light, lane positioning, unsafe or double lane changes, failure to yield properly, illegal passing, illegal turns or U-turns, driving without headlights, failure to signal, and the like. Examples provided herein recognize that traffic law compliance may not be black and white in many human driving scenarios, and that black and white traffic law compliance may indeed cause undesirable road and traffic situations, such as traffic backup, excessive hesitation, and even collisions. Thus, a significant hurdle must be surmounted in the rollout of SDVs, where SDVs and human drivers must be able to interact in normal as well as exigent driving conditions to build an exemplary level of trust. Traffic law compliance is paramount in building such trust, and thus analyzing SDV behaviors in light of human-based compliance metrics can prove invaluable in adjusting SDV operative and performance parameters to replicate or otherwise attempt to reproduce ideal human driving behavior.

In addition to traffic law compliance, the performance optimization system can analyze the performance data based on comfort metrics, such as the g-forces experienced when the SDV is accelerating, braking, and steering. Such comfort parameters may also be compared to a set of ideal and tolerable g-force ranges determined from established human-based metrics in the initial phase (e.g., for lateral, forward, and rearward g-force ranges). Thus, in analyzing SDV performance in light of both human-based traffic law compliance as well as ride comfort, the performance optimization system can ultimately converge on a set of SDV driving parameters that replicates what can be considered tantamount to an ideal human standard.

According to examples described herein, after the initial phase, the performance optimization system can receive performance data from SDVs operating throughout a given region, and analyze the performance data in light of the standardized metrics established in the initial phase. For example, the performance optimization system can analyze the performance data based on steering metrics established for human drivers, such as wheel jerks or unsmooth steering and steering response latency, or proximity metrics, such as tailgating or being too close to vehicles, pedestrians, bicyclists, or other objects and obstacles. Accordingly, for each of the established metrics, the performance optimization system can utilize the performance data from the performance measurement system to generate a set of improvement configurations for the SDV targeting one or more specified metrics to improve the performance, safety, and comfort of the SDV with respect to those particular metrics (e.g., individual traffic law compliance, braking latency, braking intensity, acceleration rate, steering latency, and the like).

Still further, in analyzing the performance data, the performance optimization system can ascribe or attach weightings to certain metrics that may be more predictive of overall safety and/or comfort than other metrics. Utilizing such weightings, the performance optimization system can generate an overall safety and performance score—or a set of scores—for an SDV based on received performance data. Such scores can indicate whether the SDV's performance is adequate for public road operation, or whether a number of adjustments to the SDV's operation are required to resolve any potential weaknesses. It is contemplated that such weightings for quantitative metrics may be location and/or time specific. For example, performance data received when the SDV is operating in, for instance, snowy conditions or nighttime conditions can be analyzed based on weightings specific for those conditions in order to optimize the safety and performance analysis. Thereafter, the performance optimization system can generate a configuration package for the SDV to configure one or more control system parameters to improve safety and/or performance of the SDV for one or more specified metrics. As described herein, it is further contemplated that the performance optimization system can be provided as a backend system dedicated for improving safety and performance of SDVs operating throughout a given region. However, it is also contemplated that the performance data can be gathered and analyzed on board the SDV, which can store the set(s) of quantitative metrics for analyzing the performance data.

Among other benefits, the examples described herein achieve a technical effect of providing a standardized performance and safety measurement and performance optimization system for autonomous or self-driving vehicles. Performance data from vehicles can be analyzed against ideal human driving standards to ultimately ensure a standard of safety and passenger comfort for public road operation.

As used herein, a computing device refers to devices corresponding to desktop computers, cellular devices or smartphones, personal digital assistants (PDAs), laptop computers, tablet devices, virtual reality (VR) and/or augmented reality (AR) devices, wearable computing devices, television (IP Television), etc., that can provide network connectivity and processing resources for communicating with the system over a network. A computing device can also correspond to custom hardware, in-vehicle devices, or on-board computers, etc. The computing device can also operate a designated application configured to communicate with the network service.

Some examples described herein can generally require the use of computing devices, including processing and memory resources. For example, one or more examples described herein may be implemented, in whole or in part, on computing devices such as servers, desktop computers, cellular or smartphones, personal digital assistants (e.g., PDAs), laptop computers, printers, digital picture frames, network equipment (e.g., routers) and tablet devices. Memory, processing, and network resources may all be used in connection with the establishment, use, or performance of any example described herein (including with the performance of any method or with the implementation of any system).

Numerous examples are referenced herein in context of an autonomous vehicle (AV) or self-driving vehicle (SDV). An AV or SDV may be used herein interchangeably, and can refer to any vehicle which is operated in a state of automation with respect to steering, propulsion, and braking. Different levels of autonomy may exist with respect to AVs and SDVs. For example, some vehicles may enable automation in limited scenarios, such as on highways, provided that drivers are present in the vehicle. More advanced AVs and SDVs can drive without any human assistance from within or external to the vehicle.

System Descriptions

FIG. 1Ais a block diagram illustrating an example vehicle performance measurement system, as described herein. The vehicle performance measurement system100can be implemented with human-driven vehicles as well as autonomous vehicles (AVs) or self-driving vehicles (SDVs). For example, in an initial phase, the performance measurement system100can be implemented on any number of human driven vehicles to measure various driver performance parameters. Over the course of hundreds, thousands, or millions of miles driven, a backend performance optimization system160can utilize the performance data142of human drivers from the measurement system100to establish a set of metrics upon which to subsequently gauge SDV performance. Such metrics can comprise subjective or objective values or ranges with which SDV performance may be compared, and can include safety metrics, performance metrics, and comfort metrics.

Furthermore, each established metric may be classified under a single grouping (e.g., performance, safety, comfort, or traffic law compliance), or may overlap with other groupings. Still further, in establishing the metrics, the performance optimization system160can apply weightings or rankings to each metric in order to prioritize SDV operation with respect to more important metrics (e.g., certain safety and compliance metrics) over less important metrics (e.g., certain comfort metrics). Thus, in analyzing performance data144for an SDV for any given driving session, or over the course of multiple driving sessions, the performance optimization system160can not only determine the SDV's performance in view of the various established metrics, but can further generate a set of SDV operation configurations (i.e., a configuration package162) that the SDV's control system155can utilize to bolster its overall performance.

According to examples provided herein, the vehicle performance measurement system100can include one or more cameras120, such as monocular and/or stereoscopic cameras, and one or more LIDAR sensor systems125. The camera(s)120and LIDAR sensors125can provide observational data127corresponding to a sensor view of the vehicle, whether the vehicle is human or autonomously driven. In certain aspects, the measurement system100can further include an inertial measurement unit (IMU)110comprised of one or more gyroscopic sensors, accelerometers, and/or magnetometers for providing precise measurements of the vehicle's specific force (e.g., the lateral g-forces based on the vehicle's acceleration, braking, steering, and road features), angular rate, and/or magnetic field surrounding the measurement system100. According to certain examples, the measurement system100can further include a GPS unit105to provide location data107indicating the vehicle's position with respective to the given region in which the vehicle operates.

In some examples, the observational data127from the LIDAR(s)125and the camera(s)120, the location data107from the GPS unit105, and the IMU data112from the IMU110can be managed by a data manager150, which can collectively store the data as performance data (i.e., human performance data142and SDV performance data144) in a local database140. In examples described herein, the data manager150can correspond to or be implemented by one or more controllers or processors of the measurement system100. In some examples, the data manager150can correlate the location data107, the observational data127, and the IMU data112with time and/or location stamps for subsequent data analysis. For human driven vehicles, the collective human performance data142can be analyzed by a performance optimization system160to establish a set of human performance metrics161with which SDV performance data144from SDVs can be compared and scrutinized.

In accordance with examples described herein, the measurement system100can be coupled to or mounted to any number of human-driven vehicles (e.g., one the order of hundreds or thousands of vehicles) in an initial data-collection phase. For example, one or more manufacturers of human driven vehicles with interests in autonomous driving technology can manufacture certain vehicles to include the measurement system100to gather performance data142for human drivers. The measurement system100can include a system interface130to transmit the human performance data142to the performance optimization system160. As provided herein, the human performance data142can comprise any combination of location data107, observational data127and IMU data112that provide specific forces and data correlations corresponding to any driving session for a driver in virtually any conditions and scenarios. This initial phase of performance data142collection can encompass any time frame, or can be persistent, until the performance optimization system160has built a robust set of human performance metrics161with which SDV performance data144can be compared.

In setting the human performance metrics161, the performance optimization system160can be selective to establish sensible safety and comfort ranges that SDVs must meet (e.g., in multiple different conditions and environments over a certain amount of mileage) in order to be considered road-worthy on public roads. Thus, the human performance metrics161can be selected to be as stringent as necessary for certain metrics (e.g., a red traffic light compliance metric) and somewhat flexible for other metrics (e.g., certain comfort metrics). Further description of the performance optimization system160and metric classification and weighting is provided below with respect toFIG. 3.

As provided herein, the system interface130can connect with the performance optimization system160via a wired bus or plug-in device. Accordingly, the human performance data142and/or the SDV performance data144can be transmitted to the performance optimization system160via the wired bus or plug-in device. In variations, the system interface130can comprise a wireless communication module (e.g., implementing Wi-Fi, BLUETOOTH, WiGig, WiMAX, etc.), and can transmit the human performance data142and/or SDV performance data144over one or more networks157.

In a subsequent phase, the measurement system100can be included as a component of individual SDVs to store SDV performance data144, also comprising any combination of observational data127, location data107, and IMU data112. As provided herein, one or more of the GPS unit105, the camera(s)120, the LIDAR(s)125, and the IMU110can be included in a single, bolt-on component comprising the vehicle performance measurement system100, or can be included as integrated components of the SDV (e.g., as part of a sensor array of an SDV). According to some examples, the measurement system100can further include a vehicle interface115to couple the measurement system100to the SDV control system155in order to receive control system data117indicating the various control system parameters utilized by the SDV control system155in operating the acceleration, steering, and braking systems of the SDV.

In certain implementations, the data manager150can correlate the control system data117with the location data,107, the observational data127, and/or the IMU data112, and collectively store the foregoing data as the SDV performance data144in the local database140. As described herein, the SDV performance data144can be compiled over the course of a single or multiple autonomous driving sessions. For example, prior to allowing a specified SDV, or SDV type, to operate on public roads and highways, certain safety and/or comfort standards (e.g., a set of National Highway Traffic Safety Administration standards) may need to be met. For example, these safety standards can mandate that any particular SDV must perform within a set of “green ranges” of the human performance metrics161, which can indicate that the SDV performs better than say, 95% of human drivers across each of the human performance metrics161in multiple types of conditions and over a certain amount of mileage. Such stringent standards may be necessary in order to advance SDVs beyond the testing phase and into full licensing and integration. Furthermore, in certain implementations, these safety standards can be utilized by the performance optimization system160to establish the human performance metrics161. Thus, in such examples, the human performance metrics161can be directly correlated to the safety standards that the SDV must meet in order to be fully operable and utilized in public settings.

In various implementations, the SDV performance data144can be transmitted to the performance optimization system160for scrutiny. As described herein, the performance optimization system160can determine a set of scores for the SDV based on the SDV performance data144for each of the human performance metrics161. Such scores can indicate rankings or percentiles of the SDV in comparison with human drivers for each metric161. For example, utilizing the IMU data112, the performance optimization system160can determine that the SDV performs better than 97% of human drivers in ride comfort metrics. As another example, utilizing the combined SDV performance data144, the performance optimization system160can determine that the SDV performs within the 90th percentile of human drivers across all traffic law compliance metrics (e.g., traffic light, stop sign, speed limit, yielding, and lane changing compliance metrics).

Some examples provide that the control system data117is also provided to the performance optimization system160, and can be time and location correlated with the other SDV performance data144. Thus, in analyzing the performance of the SDV, the analysis engine160can identify any potential underlying causes of certain weaknesses in SDV performance by also analyzing the SDV control system data117. For example, the SDV may only score within the 65th percentile for ride comfort metrics associated with braking. To determine a cause for such poor performance, the performance optimization system160can analyze braking events in the SDV performance data144(e.g., utilizing the combination of the location data107, IMU data112, and the observational data127).

According to examples described, the observational data127can indicate what exactly the SDV is reacting to when instigating brake inputs. The location data107can indicate where such brake inputs are being performed (e.g., whether on a highway, in city traffic, or on narrow roads). The IMU data112can indicate a g-force signature for each of the braking events. Finally, analysis of the control system data117in light of the foregoing data can indicate a potential cause for the underperformance in ride comfort. For example, the performance optimization system160can identify an overall deficiency in braking latency, which can result in irregular or spasmodic braking. While this may result in adequate safety compliance, the ride comfort may be undesirable for riders. Thus, the performance optimization system160can attempt to correct the braking latency deficiency without sacrificing, or with minimal sacrifice to the SDV's performance in the safety metrics.

The above analysis in determining a cause for poor or deficient performance under one or more human performance metrics161can be performed by the performance optimization system160for each metric outside a green range (e.g., where the SDV performs below a certain threshold standard, such as those necessary for licensure). In attempting to resolve such causes, the analysis engine160can analyze the control system data117from the SDV control system155—which specifies the control commands implemented by the SDV on the acceleration, braking, steering, and auxiliary systems—in relation to the other data107,127,112in the SDV performance data144. Accordingly to certain implementations, the performance optimization system160may then generate a configuration package162that can include a number of alterations to the manner in which the SDV control system155operates in order to attempt to resolve the deficiencies.

The performance optimization system160can transmit the configuration package162to the SDV control system155, which can process the configuration package162to adjust a number of control system parameters in order to resolve the deficiencies. For example, the configuration package162can cause the SDV control system155to adjust or soften braking inputs in order to resolve the braking deficiency. As another example, the configuration package162can cause the SDV control system155to increase a following distance to forward vehicles to provide increased reaction time and enabling the braking inputs to be tempered without sacrificing safety.

In executing the configuration package162, the vehicle performance and measurement system100can go through an additional iteration of data gathering. Thus, in some examples, previous SDV performance data144prior to execution of the configuration package162may be flushed or otherwise logged, and new SDV performance data144for the current iteration can be compiled. After satisfying operation conditions and mileage with the new configuration, this new SDV performance data144can be submitted to the performance optimization system160for a new round of scrutiny with respect to the established set of human performance metrics161. If necessary, additional iterations may be completed until the SDV has succeeded in exceeding the established safety and comfort standards for public operation.

Furthermore, it is contemplated that the performance optimization system160may be included as on-board hardware, a logical component, or a combination of hardware and logic on the SDV itself. In such implementations, the SDV may be loaded with predetermined human performance metrics161, and the SDV can periodically or continuously analyze the SDV performance data144from the vehicle performance measurement system100in order to ensure that the SDV operates within greenlit or standardized tolerance ranges for each of the performance metrics161(e.g., via implementation of machine learning techniques). Thus, it is contemplated that any detected deficiency with respect to any of the standardized performance metrics161can be resolved by executing a corrective configuration package162generated by the SDV itself.

FIGS. 1B and 1Cillustrate an example vehicle performance measurement system implemented on a vehicle, according to examples described herein. For example, the vehicle performance measurement system100shown and described with respect toFIG. 1Acan be implemented as the vehicle performance measurement system170as shown and described with respect toFIGS. 1A and 1B. Referring toFIG. 1B, in one example, the measurement system170can include a GPS unit173, an IMU175, a LIDAR179, and a number of monocular and/or stereoscopic cameras177. Each of the components can be coupled to a number of controllers and memory resources to store the collective data generated by each component. Furthermore, in some examples, the cameras177and LIDAR179can be arranged to focus on a forward operating direction of the vehicle.

In certain aspects, each of the components can be time synchronized (e.g., synched to the GPS receiver173) in order to associate the collective data with time stamps so that subsequent analysis can yield correlations between each component. For example, a hard braking event detected by the IMU175can be correlated with a stray dog running across the road as identified by the LIDAR179and/or camera(s)177. Subsequent scrutiny of the data can draw conclusions regarding the reaction to such an anomaly, such as whether the SDV braked too hard, not hard enough, too early, too late, and the like. However, since such an event can be considered anomalous, analysis of the data can further conclude that the braking event does not contribute to any fundamental operative issue with the SDV (e.g., a propensity towards over-reactive braking).

Referring toFIG. 1C, the vehicle performance measurement system170can include a housing183to house the various sensors and components. Furthermore, in some aspects, the measurement system170can include mounts181, such as roof rack mounts to enable easy installation of the measurement system170onto a vehicle roof185of any vehicle having roof racks. In variations, the mounts181can comprise a mounting assembly that enables installation of the performance measurement system170onto roof rails191on the roof185of the vehicle. As such, the mounting assembly can include a frame187that supports the vehicle performance measurement pod170. The frame187can be coupled on both the right and left side to a track189, that is then mounted to the roof rail191on both sides of the vehicle roof185. Thus, the mounting assembly can comprise a left track189and a right track189onto which the frame187is mounted, and which secures the measurement system170to the roof rails191of the vehicle.

In certain examples, the housing183securely covers the cameras177, the IMU175, and various other components (e.g., processors and memory), while leaving the LIDAR179exposed to transmit and receive laser light, as shown inFIG. 1B.

While the examples ofFIGS. 1B and 1Cshow a discrete sensor pod mountable to a vehicle, further implementations can be integrated into the vehicle body work, or designed as part of the vehicle. Furthermore, components of the measurement system170can be dispersed throughout the vehicle. For example, the LIDAR179and cameras177can be included on the roof of the vehicle, whereas the IMU175can be included under the hood. Still further, for SDV implementations, one or more of the sensors can be included in an overall sensor array of the SDV itself (e.g., the sensor array that the SDV control system obtains sensor data from for purposes of perceiving the environment around the SDV, controlling the SDV, etc.).

FIG. 2is a block diagram illustrating an example AV or SDV implementing a control system, as described herein. In an example ofFIG. 2, a control system220can autonomously operate the SDV200in a given geographic region for a variety of purposes, including transport services (e.g., transport of humans, delivery services, etc.). In examples described, an autonomously driven vehicle can operate without human control. For example, in the context of automobiles, an autonomously driven vehicle can steer, accelerate, shift, brake, and operate lighting components. Some variations also recognize that an autonomous-capable vehicle can be operated either autonomously, manually, or a combination of both.

According to some examples, the control system220can utilize specific sensor resources in order to intelligently operate the vehicle200. For example, the control system220can operate the vehicle200by autonomously operating the steering254, acceleration252, and braking systems256of the vehicle200as the vehicle progresses to a destination. The control system220can perform vehicle control actions (e.g., braking, steering, accelerating) and route planning using sensor information, as well as other inputs (e.g., transmissions from remote or local human operators, network communication from other vehicles, etc.).

In an example ofFIG. 2, the control system220includes a computer or processing system which operates to process sensor data211that is obtained on the vehicle200with respect to a road segment upon which the vehicle200operates. The sensor data211can be used to determine actions which are to be performed by the vehicle200in order for the vehicle200to continue on a route to a destination. In some variations, the control system220can include other functionality, such as wireless communication capabilities, to send and/or receive wireless communications with one or more remote sources. In controlling the vehicle200, the control system220can issue instructions and data, shown as commands235, which programmatically control various electromechanical interfaces of the vehicle200. The commands235can serve to control operational aspects of the vehicle200, including propulsion, braking, steering, and auxiliary behavior (e.g., turning lights on).

The SDV200can be equipped with multiple types of sensors201,203which can combine to provide a computerized perception of the space and the physical environment surrounding the vehicle200. Likewise, the control system220can operate within the SDV200to receive sensor data211from the collection of sensors201,203, and to control various electromechanical interfaces for operating the vehicle200on roadways.

In more detail, the sensors201,203operate to collectively obtain a complete sensor view of the vehicle200, and further to obtain situational information proximate to the vehicle200, including any potential hazards proximate to the vehicle200. By way of example, the sensors201,203can include multiple sets of cameras sensors201(video cameras, stereoscopic pairs of cameras or depth perception cameras, long range cameras), remote detection sensors203such as provided by radar or LIDAR, proximity or touch sensors, and/or sonar sensors (not shown).

Each of the sensors201,203can communicate with the control system220utilizing a corresponding sensor interface210,212. Each of the sensor interfaces210,212can include, for example, hardware and/or other logical components which are coupled or otherwise provided with the respective sensor. For example, the sensors201,203can include a video camera and/or stereoscopic camera set which continually generates image data of the physical environment of the vehicle200. As an addition or alternative, the sensor interfaces210,212can include a dedicated processing resource, such as provided with a field programmable gate array (“FPGA”) which can, for example, receive and/or process raw image data from the camera sensor.

According to one implementation, the SDV can include a set of controllers240to implement commands235from the control system220on the various systems of the SDV200. For example, the controllers240can implement the commands235on the acceleration252, steering254, braking256, and auxiliary systems258of the SDV200to control propulsion, steering, braking, and/or other vehicle behavior to operate the SDV200along a current route. Thus, while the vehicle200actively drives along the current route, the controller(s)240can continuously adjust and alter the movement of the vehicle200in response to receiving a corresponding set of commands235from the control system220.

According to examples, the commands235can specify actions to be performed by the vehicle200. The actions can correlate to one or multiple vehicle control mechanisms (e.g., steering mechanism254, brakes256, accelerator252, etc.). The commands235can specify the actions, along with attributes such as magnitude, duration, directionality, or other operational characteristics of the vehicle200. By way of example, the commands235generated from the control system220can specify a relative location of a road segment which the SDV200is to occupy while in motion (e.g., changing lanes, moving into a center divider or towards shoulder, turning the vehicle, etc.). As other examples, the commands235can specify a speed, a change in acceleration (or deceleration) from braking or accelerating, a turning action, or a state change of exterior lighting or other components. The controllers240can translate the commands235into control signals for implementation on the SDV's200operative systems252,254,256,258.

In an example ofFIG. 2, the control system220can include a route planner222, event logic224, and vehicle control logic228. The vehicle control logic228can convert alerts of event logic224(“event alert229”) into commands235that specify a set of vehicle actions. Furthermore, in operating the acceleration252, braking256, and steering systems254of the SDV200, the control system220can include a database270that stores previously recorded and processed localization maps272of the given region. As described herein, the localization maps272can comprise processed or normalized surface data that enables the control system220to compare with the sensor data211in order to identify any potential hazards while operating throughout the given region.

Additionally, the route planner222can select one or more route segments226that collectively form a path of travel for the SDV200when the vehicle200is on a current trip (e.g., servicing a pick-up request). In one implementation, the route planner222can specify route segments226of a planned vehicle path which defines turn by turn directions for the vehicle200at any given time during the trip. The route planner222may utilize the sensor interface212to receive GPS information as sensor data211. The vehicle control logic228can process route updates from the route planner222as commands235to progress along a path or route using default driving rules and actions (e.g., moderate steering and speed).

In some examples, the route planner222can receive route information or instructions over a network280from a transport management system, which can instruct the SDV200to make pick-ups and drop-offs of passengers in connection with a transportation arrangement service, such as those provided by UBER, Inc. of San Francisco, Calif.

In certain implementations, the event logic224can trigger low level responses to detected events. A detected event can correspond to a roadway condition or obstacle which, when detected, poses a potential hazard or threat of collision to the vehicle200. By way of example, a detected event can include an object in the road segment, heavy traffic ahead, and/or wetness or other environmental conditions on the road segment. The event logic224can use sensor data211from cameras, LIDAR, radar, sonar, or various other image or sensor component sets in order to detect the presence of such events as described. For example, the event logic224can detect potholes, debris, objects projected to be on a collision trajectory, and the like. Thus, the event logic224can detect events which enable the control system220to make evasive actions or plan for any potential hazards.

When events are detected, the event logic224can signal an event alert229that can classify the event and indicate the type of avoidance action to be performed. Additionally, the control system220can determine whether an event corresponds to a potential incident with a human driven vehicle, a pedestrian, or other human entity external to the SDV200. In turn, the vehicle control228can determine a low level response based on a score or classification of the event. Such response can correspond to an event avoidance action223, or an action that the SDV200can perform to maneuver based on the detected event and its score or classification. By way of example, the vehicle response can include a slight or sharp vehicle maneuver for avoidance using a steering control mechanism and/or braking component. The event avoidance action223can be signaled through the commands235for controllers240to execute on the SDV's200acceleration system252, steering system254, and braking system256.

When an anticipated dynamic object of a particular class does in fact move into position of likely collision or interference, some examples provide that event logic224can signal the event alert229to cause the vehicle control228to generate commands235that correspond to an event avoidance action223. For example, in the event of an incident in the path of the vehicle200, the event logic224can signal the event alert229to avoid a collision. The event alert229can indicate (i) a classification of the event (e.g., “serious” and/or “immediate”), (ii) information about the event, such as the type of object that generated the event alert229, and/or information indicating a type of action the vehicle200should take (e.g., location of object relative to path of vehicle, size or type of object, etc.).

According to examples described herein, the SDV200can include a communications array214to communicate over one or more networks280with an performance optimization system290, such as the performance optimization system160described with respect toFIG. 1A. The control system220of the SDV200can include data gathering logic260to compile raw data in accordance with the vehicle performance measurement system100as shown and described with respect toFIG. 1A. For example, the SDV200can include a sensor pod275comprising the various components and sensors shown inFIGS. 1B and 1C. The sensor pod275can provide raw data277comprising IMU data, observational data from the LIDAR and/or cameras, and/or GPS data from a GPS receiver. As provided herein, the data gathering logic260can also receive the sensor data211from the SDV's200on-board sensors201,203for storage as performance data262.

Thus, the data gathering logic260can compile the performance data262from the various sensors and components of the SDV's200sensors and/or the sensor pod275—which can comprise one or more LIDARs, cameras, an IMU, and a GPS receiver. As discussed herein the performance data262can be stored locally in a database (e.g., database270), and/or can be transmitted to the performance optimization system290over the one or more networks280. As further discussed herein, the performance optimization system290can process the performance data262based on a set of human performance metrics to determine the overall performance of the SDV200. Thus, the performance optimization system290may determine a score or percentile for the SDV200for each metric to determine how well the SDV200is performing in relation to an average or idealized human driver.

In certain implementations, the performance optimization system290can utilize the performance data262to generate a configuration package286comprising a set of parameters that the control system220can process to alter the manner in which the control system220interprets and reacts to sensor data211from the SDV's200camera, LIDAR, sonar, and other sensor systems. For example, the configuration package286can be processed by the vehicle control228to adjust one or more parameters in which the vehicle control228generates the commands235, which can comprise of acceleration, braking, and steering commands. In various aspects, such adjustments can seek to improve the SDV's200performance with respect to certain metrics.

FIG. 3is a block diagram illustrating an example performance optimization system in communication with a fleet of vehicles, as described herein. In the examples shown with respect toFIG. 3, a set of human driven vehicles390can be provided with a vehicle performance measurement device, such as those shown and described with respect toFIGS. 1A through 1C. As further described herein, the performance optimization system300can collect human control data392from human driven vehicles390in order to determine a set of human performance metrics333with which to compare performance data394from an SDV309or fleet of SDVs. Such metrics333can include acceleration and braking metrics (e.g., braking latency and intensity), lane positioning, consistency in maintaining distances from other vehicles and pedestrians, compliance with stop signs, stop lights, and other traffic signs and regulations, speed control, steering smoothness and consistency, and the like.

Further human performance metrics333can include overall braking comfort, specific types of braking performance (e.g., reactive braking versus anticipated braking), fuel consumption and performance efficiency, lateral g-forces, overall acceleration performance, ride smoothness, condition-based performance (e.g., any particular metric in snowy or rainy conditions), speed control, overall cautiousness, slowing or stopping inappropriately, lane changing behavior, steering jitter, traffic law compliance, traffic light compliance, stopping positioning, proximity to objects, vehicles, or people, yield behavior, subjective ratings from riders, following behavior (e.g., tailgating), braking latency, acceleration latency, steering latency (i.e., response time), use of headlights and turn signals, braking distances, and the like. Utilizing the human control data392, the performance optimization system300can build a set of quantitative models332to run subsequently acquired performance data394through in order to gauge the performance of an SDV309in relation to an idealized human driver. Such quantitative models332can be stored in a database330of the performance optimization system300.

For example, running performance data394from an SDV309through the quantitative models332can yield a performance result or score337for the SDV309for each metric333. In certain implementations, the performance optimization system300can build the quantitative models332using the human control data392from the measurement systems on human driven vehicles. The human control data392can be processed by the performance optimization system300to set ranges for each metric. For example, a data set from a particular human driven vehicle can be processed by the performance optimization system300to determine the driver's performance in relation to all other drivers for each particular metric. The driver's rating can comprises a series of scores or percentiles indicating that the driver is say, in the 45th percentile in overall traffic law compliance, or the 75th percentile in overall ride comfort. A trialing process or initial phase can be implemented for data collection and to build a robust set of quantitative models332for the performance metrics333that provide consistent, accurate, and reliable performance scores for subsequent implementation for SDVs.

Once the quantitative models332are built, according to many examples, the performance optimization system300can include a communications interface305which can receive performance data394from SDVs309operating throughout a given region. In one example, the communications interface305connects with the SDVs309wirelessly over one or more networks380. In variations, the communications interface305can include a data bus comprising an input port (e.g., a universal serial bus port) to receive the performance data394either via direct line or external memory resource.

The performance optimization system300can further include a performance analysis engine335, which can process the performance data394received from a particular SDV309using the quantitative models332. As described herein, the performance data394can include raw or preprocessed sensor data from the vehicle performance measurement system of the SDV309. Thus, the performance data394can include IMU data (e.g., accelerometer and gyroscope data), observational data from the SDV's309LIDAR and/or camera sensor systems, and location data from a GPS receiver. In one example, the performance optimization system300can include a mapping engine375that provides map data379to the performance analysis engine335to enable the performance analysis engine335to correlate the performance data394with specified locations or areas within the given region in which the SDV309operates. For example, performance data394indicating high caution by the SDV309(e.g., unusually sensitive braking performance) can be correlated to a school zone in the map data379, which the performance analysis engine335can take into consideration when executing the quantitative models332with the performance data394.

As provided herein, the performance data394can comprise data from the SDV309operating on a pre-configured course in a variety of conditions, or on public roads after an initial certification process. It is contemplated that SDV performance may deteriorate over time, requiring replacement parts necessitating configuration and integration with the other SDV components. Thus, the performance optimization system300can provide a standardized certification for not only new SDVs, but also SDVs that require fine-tuning or recertification.

According to some aspects, in running the performance data394through the quantitative models332, the performance analysis engine335can generate a set of performance scores337indicating the SDV's309performance with respect to the performance metrics333. Such scores can indicate whether the SDV309has passed or failed for each of the metrics. In variations, the performance scores337can comprise a set of charts that indicate a passable range for each performance metric333, and an indicator of the SDV's309performance on each chart with respect to each metric333.

The performance scores337can be provided to a configuration package generator355, which can analyze the SDV's309performance with respect to each metric333. In some examples, the performance optimization system300can store SDV control system parameters334for each particular type of SDV. The SDV control system parameters334can indicate the configuration settings of the SDV309that map the SDV's309behavior. For example, the parameters334can include braking strength and reaction time in relation to the SDV's309decision-making process, or general acceleration settings.

According to some examples, the configuration package generator355can utilize the SDV control system parameters334in analyzing the performance scores337of the SDV309. In some examples, the configuration package generator355can identify certain deficiencies in the SDV's309operation in the performance scores337, and utilize the SDV control system parameters334to determine a set of corrective adjustments to the SDV's309control system parameters334that can strengthen the SDV's309performance with respect to certain metrics333. As a general example, the performance scores337can indicate that the SDV309is deficient in overall ride comfort, further indicating general hard braking and aggressive steering. The configuration package generator355can utilize the SDV control system parameters334to generate a control system configuration package357that includes a number of adjustments to be implemented by the SDV's309control system in order to alleviate the ride comfort deficiency. Such adjustments can include, for example, increasing braking distances, and softening steering responses.

Accordingly, based on the performance scores337, the configuration package generator355can generate a control system configuration package357executable by the SDV's309control system to ultimately improve the SDV's309performance with respect to the specific performance metrics333. The performance optimization system300can then transmit the control system configuration package357to the SDV309for implementation. As provided herein, execution of the configuration package357by the SDV's309control system can alter the manner in which the control system responds to sensor data (e.g., LIDAR and camera data) in operating the SDV's309acceleration, braking, and steering systems. Such adjustments can also comprise general modifications to the SDV's309use of the acceleration, braking, and steering system to improve SDV compliance with traffic laws and regulation, SDV ride comfort, and various other performance metrics333with minimal sacrifice in SDV performance with respect to other metrics333.

Methodology

FIG. 4is a flow chart describing an example method of analyzing performance data from an SDV and generating an SDV configuration package to improve SDV performance, according to examples described herein. In the below description ofFIG. 4, reference may be made to reference characters representing like features as shown and described with respect toFIG. 1AthroughFIG. 3. Furthermore, the method described with respect toFIG. 4may be performance by an example performance optimization system160,290,300as shown and described with respect toFIGS. 1A, 2, and 3. Referring toFIG. 4, the performance optimization system300can receive performance data from any number of human driven vehicles390(400). The performance data can comprise IMU data (401), LIDAR data (402), camera data (403), and/or GPS data (404), and can provide the performance optimization system300with human control information392indicating human driving performance and characteristics with which to compare SDV performance.

According to examples described herein, the performance optimization system300can determine a set of performance metrics333based on the human control information392(405). In some examples, the performance metrics333can be classified as three main classifications comprising traffic law compliance (470), ride comfort (480), and performance etiquette (490). It is contemplated that many other classifications can be included, and the performance metrics333can include any conceivable attribute or trait with which to compare SDV operative behavior with human behavior. Under the traffic law compliance umbrella, performance metrics333can generally include compliance with traffic lights (472), road signs (474), yielding behavior (476), and speed limit compliance (478). However, more granular metrics may also be included, such as compliance with rules relating to rights-of-way, lane use, turning, individual intersections, pedestrian crossings, overtaking, carpool lanes, pre-emption (e.g., moving over for an emergency vehicle), and the like.

Under the ride comfort umbrella, performance metrics333can include anything having to do with an SDV's acceleration (482), braking (484), and steering (486) that can affect the comfort of passengers, as described herein. A few examples can include acceleration intensity, steering jitter or aggression, braking latency, braking intensity, fuel or energy usage efficiency, and lateral g-force control. Under the etiquette umbrella, performance metrics333can include proximity control with pedestrians, objects, and other vehicles (492), lane behavior (494) (e.g., lane positioning and lane changing behavior), and speed control (496) (e.g., avoidance of erratic or abrupt braking and acceleration). As provided herein, several metrics can overlap with other metric classifications. Furthermore, adjustments to improve performance with respect to certain metrics can affect the SDV's performance with respect to other metrics. Thus, examples described herein seek to optimize the SDV's performance across all performance metrics, giving high priority to certain metrics (e.g., red light compliance), while taking into consideration factors such as passenger comfort and etiquette with external entities (e.g., other drivers and pedestrians).

In certain implementations, the performance optimization system300can construct performance models332for scoring SDV performance with respect to each performance metric333(410). As described herein, the performance models332can attribute a priority or weighting for each performance metric333. The performance optimization system300can subsequently receive performance data394from a particular SDV309(415). In many examples, this performance data394can also comprise data from an IMU (416), a LIDAR (417), cameras (418), and/or a GPS receiver (419) of the SDV309. The performance optimization system300may then analyze the performance data394using the constructed performance models332(420), and generate a performance score337for the SDV309for each respective performance metric333(425).

In some examples, the performance optimization system300can further receive a set of configurations that the SDV309is currently implementing during operation. Such configurations can include sensitivity settings for responding to dynamic objects in sensor data, braking distances, acceleration settings, traffic law compliance settings and priorities (e.g., right-of-way priorities), and the like. Utilizing the performance scores337and the control system settings or configurations of the SDV309, the performance optimization system300can generate a configuration package357for the SDV309(430). The configuration package357can include a number of adjustments to the SDV's309control system settings that affect how the SDV309operates on public roads and in traffic conditions. The performance optimization system300may then transmit the configuration package357to the SDV309for execution to adjust a number of the SDV's309operative parameters (435). As described herein, implementation of the configuration package357can cause the SDV309to make a number of adjustments to its operation in order to bolster performance with regard to the performance metrics333, including the traffic law compliance metrics (470), the ride comfort metrics (480), and the driving etiquette metrics (490).

Hardware Diagrams

FIG. 5is a block diagram that illustrates a computer system upon which examples described herein may be implemented. A computer system500can be implemented on, for example, a server or combination of servers. For example, the computer system500may be implemented as part of a network service for providing transportation services. In the context ofFIGS. 1A, 2, and 3, the performance optimization system160,290,300may be implemented using a computer system500such as described byFIG. 5. The performance optimization system100may also be implemented using a combination of multiple computer systems as described in connection withFIG. 5.

In one implementation, the computer system500includes processing resources510, a main memory520, a read-only memory (ROM)530, a storage device540, and a communication interface550. The computer system500includes at least one processor510for processing information stored in the main memory520, such as provided by a random access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by the processor510. The main memory520also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor510. The computer system500may also include the ROM530or other static storage device for storing static information and instructions for the processor510. A storage device540, such as a magnetic disk or optical disk, is provided for storing information and instructions.

The communication interface550enables the computer system500to communicate over one or more networks580(e.g., cellular network) through use of the network link (wireless or wired). Using the network link, the computer system500can communicate with one or more computing devices, one or more servers, driver devices, and/or a fleet of SDVs. In accordance with examples, the computer system500receives human control data392from measurement systems implemented on human-driven vehicles, and constructs performance models to gauge SDV performance based on a set of human-based metrics. The computer system500may then receive performance data584from an SDV. The executable instructions stored in the memory520can include performance analysis instructions524, which the processor510can execute to analyze the performance of the SDV with respect to the set of performance metrics. The communication interface550can receive the performance data584, and the processor510—executing the performance analysis instructions522—can generate a performance score for the SDV for each of the performance metrics.

The main memory520can also store configuration instructions524, which the processor510executes to perform an optimization operation based on the performance scores in order to determine an optimal set of configuration settings for the SDV's control system. In performing the optimization, the processor510can utilize the performance scores and a set of current SDV configurations to generate a configuration package554that include a number of settings adjustments to maximize or otherwise optimize the SDV's performance with respect to the totality of the performance metrics.

The processor510is configured with software and/or other logic to perform one or more processes, steps and other functions described with implementations, such as described byFIGS. 1A through 4, and elsewhere in the present application.

FIG. 6is a block diagram illustrating a computer system upon which example SDV processing systems described herein may be implemented. The computer system600can be implemented using one or more processors604, and one or more memory resources606. In the context ofFIG. 2, the control system220can be implemented using one or more components of the computer system600shown inFIG. 6.

According to some examples, the computer system600may be implemented within an autonomous vehicle or self-driving vehicle with software and hardware resources such as described with examples ofFIG. 2. In an example shown, the computer system600can be distributed spatially into various regions of the SDV, with various aspects integrated with other components of the SDV itself. For example, the processors604and/or memory resources606can be provided in the trunk of the SDV. The various processing resources604of the computer system600can also execute control instructions612using microprocessors or integrated circuits. In some examples, the control instructions612can be executed by the processing resources604or using field-programmable gate arrays (FPGAs).

In an example ofFIG. 6, the computer system600can include a communication interface650that can enable communications over one or more networks660with a backend transport system, an performance optimization system, one or more other SDVs, and the like. In one implementation, the communication interface650can also provide a data bus or other local links to electro-mechanical interfaces of the vehicle, such as wireless or wired links to and from the SDV control system220, and can provide a network link to a performance optimization system over one or more networks660.

The memory resources606can include, for example, main memory, a read-only memory (ROM), storage device, and cache resources. The main memory of memory resources606can include random access memory (RAM) or other dynamic storage device, for storing information and instructions which are executable by the processors604. The processors604can execute instructions for processing information stored with the main memory of the memory resources606. The main memory606can also store temporary variables or other intermediate information which can be used during execution of instructions by one or more of the processors604. The memory resources606can also include ROM or other static storage device for storing static information and instructions for one or more of the processors604. The memory resources606can also include other forms of memory devices and components, such as a magnetic disk or optical disk, for purpose of storing information and instructions for use by one or more of the processors604.

According to some examples, the memory606may store a set of software instructions including, for example, control instructions612. The control instructions612may be executed by one or more of the processors604in order to implement functionality such as described with respect to the SDVs herein. As described herein, the control instructions612are executed by the SDV to generate control commands605for operating the acceleration, braking, and steering systems620. Thus, the processors604can receive live sensor data632from an on-board sensor array630(e.g., comprising LIDAR, camera, sonar, radar, and other sensor systems), and operate the acceleration, braking, and steering systems620along a current route based on the live sensor data632.

In certain variations described herein, the SDV can include an autonomous vehicle performance measuring pod that includes a set of sensors for measure raw performance data654using an IMU, LIDAR, cameras, GPS, and the like. The computer system600can transmit the performance data654to a backend performance optimization system over the network660, and receive a configuration package662comprising a number of adjustment settings. In one aspect, the configuration package662can alter the manner in which the processor604interprets the sensor data632in operating the acceleration, braking, and steering systems620. For example, the configuration package662can comprise a set of alterations to the control instructions612to adjust the way the processors604comply with traffic regulations, control speed, implement the brakes and accelerator, etc. in order to bolster SDV performance and comfort.

While examples ofFIGS. 5 and 6provide for computing systems for implementing aspects described, some or all of the functionality described with respect to one computing system ofFIGS. 5 and 6may be performed by one or more other computing systems described with respect toFIGS. 5 and 6. Thus, once or more functions of the performance optimization system described herein may, for example, be performed by the computer system600implemented on the SDV.

It is contemplated for examples described herein to extend to individual elements and concepts described herein, independently of other concepts, ideas or systems, as well as for examples to include combinations of elements recited anywhere in this application. Although examples are described in detail herein with reference to the accompanying drawings, it is to be understood that the concepts are not limited to those precise examples. As such, many modifications and variations will be apparent to practitioners skilled in this art. Accordingly, it is intended that the scope of the concepts be defined by the following claims and their equivalents. Furthermore, it is contemplated that a particular feature described either individually or as part of an example can be combined with other individually described features, or parts of other examples, even if the other features and examples make no mentioned of the particular feature. Thus, the absence of describing combinations should not preclude claiming rights to such combinations.