Patent ID: 12202512

DETAILED DESCRIPTION

The following describes the technology of this disclosure within the context of an autonomous vehicle for example purposes only. As described herein, the technology described herein is not limited to an autonomous vehicle and can be implemented for or within other autonomous platforms and other computing systems.

With reference toFIGS.1-15, example implementations of the present disclosure are discussed in further detail.FIG.1is a block diagram of an example operational scenario, according to some implementations of the present disclosure. In the example operational scenario, an environment100contains an autonomous platform110and a number of objects, including first actor120, second actor130, and third actor140. In the example operational scenario, the autonomous platform110can move through the environment100and interact with the object(s) that are located within the environment100(e.g., first actor120, second actor130, third actor140, etc.). The autonomous platform110can optionally be configured to communicate with remote system(s)160through network(s)170.

The environment100may be or include an indoor environment (e.g., within one or more facilities, etc.) or an outdoor environment. An indoor environment, for example, may be an environment enclosed by a structure such as a building (e.g., a service depot, maintenance location, manufacturing facility, etc.). An outdoor environment, for example, may be one or more areas in the outside world such as, for example, one or more rural areas (e.g., with one or more rural travel ways, etc.), one or more urban areas (e.g., with one or more city travel ways, highways, etc.), one or more suburban areas (e.g., with one or more suburban travel ways, etc.), or other outdoor environments.

The autonomous platform110may be any type of platform configured to operate within the environment100. For example, the autonomous platform110may be a vehicle configured to autonomously perceive and operate within the environment100. The vehicles may be a ground-based autonomous vehicle such as, for example, an autonomous car, truck, van, etc. The autonomous platform110may be an autonomous vehicle that can control, be connected to, or be otherwise associated with implements, attachments, and/or accessories for transporting people or cargo. This can include, for example, an autonomous tractor optionally coupled to a cargo trailer. Additionally, or alternatively, the autonomous platform110may be any other type of vehicle such as one or more aerial vehicles, water-based vehicles, space-based vehicles, other ground-based vehicles, etc.

The autonomous platform110may be configured to communicate with the remote system(s)160. For instance, the remote system(s)160can communicate with the autonomous platform110for assistance (e.g., navigation assistance, situation response assistance, etc.), control (e.g., fleet management, remote operation, etc.), maintenance (e.g., updates, monitoring, etc.), or other local or remote tasks. In some implementations, the remote system(s)160can provide data indicating tasks that the autonomous platform110should perform. For example, as further described herein, the remote system(s)160can provide data indicating that the autonomous platform110is to perform a trip/service such as a user transportation trip/service, delivery trip/service (e.g., for cargo, freight, items), etc.

The autonomous platform110can communicate with the remote system(s)160using the network(s)170. The network(s)170can facilitate the transmission of signals (e.g., electronic signals, etc.) or data (e.g., data from a computing device, etc.) and can include any combination of various wired (e.g., twisted pair cable, etc.) or wireless communication mechanisms (e.g., cellular, wireless, satellite, microwave, radio frequency, etc.) or any desired network topology (or topologies). For example, the network(s)170can include a local area network (e.g., intranet, etc.), a wide area network (e.g., the Internet, etc.), a wireless LAN network (e.g., through Wi-Fi, etc.), a cellular network, a SATCOM network, a VHF network, a HF network, a WiMAX based network, or any other suitable communications network (or combination thereof) for transmitting data to or from the autonomous platform110.

As shown for example inFIG.1, environment100can include one or more objects. The object(s) may be objects not in motion or not predicted to move (“static objects”) or object(s) in motion or predicted to be in motion (“dynamic objects” or “actors”). In some implementations, the environment100can include any number of actor(s) such as, for example, one or more pedestrians, animals, vehicles, etc. The actor(s) can move within the environment according to one or more actor trajectories. For instance, the first actor120can move along any one of the first actor trajectories122A-C, the second actor130can move along any one of the second actor trajectories132, the third actor140can move along any one of the third actor trajectories142, etc.

As further described herein, the autonomous platform110can utilize its autonomy system(s) to detect these actors (and their movement) and plan its motion to navigate through the environment100according to one or more platform trajectories112A-C. The autonomous platform110can include onboard computing system(s)180. The onboard computing system(s)180can include one or more processors and one or more memory devices. The one or more memory devices can store instructions executable by the one or more processors to cause the one or more processors to perform operations or functions associated with the autonomous platform110, including implementing its autonomy system(s).

FIG.2is a block diagram of an example autonomy system200for an autonomous platform, according to some implementations of the present disclosure. In some implementations, the autonomy system200can be implemented by a computing system of the autonomous platform (e.g., the onboard computing system(s)180of the autonomous platform110). The autonomy system200can operate to obtain inputs from sensor(s)202or other input devices. In some implementations, the autonomy system200can additionally obtain platform data208(e.g., map data210) from local or remote storage. The autonomy system200can generate control outputs for controlling the autonomous platform (e.g., through platform control devices212, etc.) based on sensor data204, map data210, or other data. The autonomy system200may include different subsystems for performing various autonomy operations. The subsystems may include a localization system230, a perception system240, a planning system250, and a control system260. The localization system230can determine the location of the autonomous platform within its environment; the perception system240can detect, classify, and track objects and actors in the environment; the planning system250can determine a trajectory for the autonomous platform; and the control system260can translate the trajectory into vehicle controls for controlling the autonomous platform. The autonomy system200can be implemented by one or more onboard computing system(s). The subsystems can include one or more processors and one or more memory devices. The one or more memory devices can store instructions executable by the one or more processors to cause the one or more processors to perform operations or functions associated with the subsystems. The computing resources of the autonomy system200can be shared among its subsystems, or a subsystem can have a set of dedicated computing resources.

In some implementations, the autonomy system200can be implemented for or by an autonomous vehicle (e.g., a ground-based autonomous vehicle). The autonomy system200can perform various processing techniques on inputs (e.g., the sensor data204, the map data210) to perceive and understand the vehicle's surrounding environment and generate an appropriate set of control outputs to implement a vehicle motion plan (e.g., including one or more trajectories) for traversing the vehicle's surrounding environment (e.g., environment100ofFIG.1, etc.). In some implementations, an autonomous vehicle implementing the autonomy system200can drive, navigate, operate, etc. with minimal or no interaction from a human operator (e.g., driver, pilot, etc.).

In some implementations, the autonomous platform can be configured to operate in a plurality of operating modes. For instance, the autonomous platform can be configured to operate in a fully autonomous (e.g., self-driving, etc.) operating mode in which the autonomous platform is controllable without user input (e.g., can drive and navigate with no input from a human operator present in the autonomous vehicle or remote from the autonomous vehicle, etc.). The autonomous platform can operate in a semi-autonomous operating mode in which the autonomous platform can operate with some input from a human operator present in the autonomous platform (or a human operator that is remote from the autonomous platform). In some implementations, the autonomous platform can enter into a manual operating mode in which the autonomous platform is fully controllable by a human operator (e.g., human driver, etc.) and can be prohibited or disabled (e.g., temporary, permanently, etc.) from performing autonomous navigation (e.g., autonomous driving, etc.). The autonomous platform can be configured to operate in other modes such as, for example, park or sleep modes (e.g., for use between tasks such as waiting to provide a trip/service, recharging, etc.). In some implementations, the autonomous platform can implement vehicle operating assistance technology (e.g., collision mitigation system, power assist steering, etc.), for example, to help assist the human operator of the autonomous platform (e.g., while in a manual mode, etc.).

Autonomy system200can be located onboard (e.g., on or within) an autonomous platform and can be configured to operate the autonomous platform in various environments. The environment may be a real-world environment or a simulated environment. In some implementations, one or more simulation computing devices can simulate one or more of: the sensors202, the sensor data204, communication interface(s)206, the platform data208, or the platform control devices212for simulating operation of the autonomy system200.

In some implementations, the autonomy system200can communicate with one or more networks or other systems with the communication interface(s)206. The communication interface(s)206can include any suitable components for interfacing with one or more network(s) (e.g., the network(s)170ofFIG.1, etc.), including, for example, transmitters, receivers, ports, controllers, antennas, or other suitable components that can help facilitate communication. In some implementations, the communication interface(s)206can include a plurality of components (e.g., antennas, transmitters, or receivers, etc.) that allow it to implement and utilize various communication techniques (e.g., multiple-input, multiple-output (MIMO) technology, etc.).

In some implementations, the autonomy system200can use the communication interface(s)206to communicate with one or more computing devices that are remote from the autonomous platform (e.g., the remote system(s)160) over one or more network(s) (e.g., the network(s)170). For instance, in some examples, one or more inputs, data, or functionalities of the autonomy system200can be supplemented or substituted by a remote system communicating over the communication interface(s)206. For instance, in some implementations, the map data210can be downloaded over a network to a remote system using the communication interface(s)206. In some examples, one or more of localization system230, perception system240, planning system250, or control system260can be updated, influenced, nudged, communicated with, etc. by a remote system for assistance, maintenance, situational response override, management, etc.

The sensor(s)202can be located onboard the autonomous platform. In some implementations, the sensor(s)202can include one or more types of sensor(s). For instance, one or more sensors can include image capturing device(s) (e.g., visible spectrum cameras, infrared cameras, etc.). Additionally, or alternatively, the sensor(s)202can include one or more depth capturing device(s). For example, the sensor(s)202can include one or more Light Detection and Ranging (LIDAR) sensor(s) or Radio Detection and Ranging (RADAR) sensor(s). The sensor(s)202can be configured to generate point data descriptive of at least a portion of a three-hundred-and-sixty-degree view of the surrounding environment. The point data can be point cloud data (e.g., three-dimensional LIDAR point cloud data, RADAR point cloud data). In some implementations, one or more of the sensor(s)202for capturing depth information can be fixed to a rotational device in order to rotate the sensor(s)202about an axis. The sensor(s)202can be rotated about the axis while capturing data in interval sector packets descriptive of different portions of a three-hundred-and-sixty-degree view of a surrounding environment of the autonomous platform. In some implementations, one or more of the sensor(s)202for capturing depth information can be solid state.

The sensor(s)202can be configured to capture the sensor data204indicating or otherwise being associated with at least a portion of the environment of the autonomous platform. The sensor data204can include image data (e.g., 2D camera data, video data, etc.), RADAR data, LIDAR data (e.g., 3D point cloud data, etc.), audio data, or other types of data. In some implementations, the autonomy system200can obtain input from additional types of sensors, such as inertial measurement units (IMUs), altimeters, inclinometers, odometry devices, location or positioning devices (e.g., GPS, compass), wheel encoders, or other types of sensors. In some implementations, the autonomy system200can obtain sensor data204associated with particular component(s) or system(s) of an autonomous platform. This sensor data204can indicate, for example, wheel speed, component temperatures, steering angle, cargo or passenger status, etc. In some implementations, the autonomy system200can obtain sensor data204associated with ambient conditions, such as environmental or weather conditions. In some implementations, the sensor data204can include multi-modal sensor data. The multi-modal sensor data can be obtained by at least two different types of sensor(s) (e.g., of the sensors202) and can indicate static object(s) or actor(s) within an environment of the autonomous platform. The multi-modal sensor data can include at least two types of sensor data (e.g., camera and LIDAR data). In some implementations, the autonomous platform can utilize the sensor data204for sensors that are remote from (e.g., offboard) the autonomous platform. This can include for example, sensor data204captured by a different autonomous platform.

The autonomy system200can obtain the map data210associated with an environment in which the autonomous platform was, is, or will be located. The map data210can provide information about an environment or a geographic area. For example, the map data210can provide information regarding the identity and location of different travel ways (e.g., roadways, etc.), travel way segments (e.g., road segments, etc.), buildings, or other items or objects (e.g., lampposts, crosswalks, curbs, etc.); the location and directions of boundaries or boundary markings (e.g., the location and direction of traffic lanes, parking lanes, turning lanes, bicycle lanes, other lanes, etc.); traffic control data (e.g., the location and instructions of signage, traffic lights, other traffic control devices, etc.); obstruction information (e.g., temporary or permanent blockages, etc.); event data (e.g., road closures/traffic rule alterations due to parades, concerts, sporting events, etc.); nominal vehicle path data (e.g., indicating an ideal vehicle path such as along the center of a certain lane, etc.); or any other map data that provides information that assists an autonomous platform in understanding its surrounding environment and its relationship thereto. In some implementations, the map data210can include high-definition map information. Additionally, or alternatively, the map data210can include sparse map data (e.g., lane graphs, etc.). In some implementations, the sensor data204can be fused with or used to update the map data210in real-time.

The autonomy system200can include the localization system230, which can provide an autonomous platform with an understanding of its location and orientation in an environment. In some examples, the localization system230can support one or more other subsystems of the autonomy system200, such as by providing a unified local reference frame for performing, e.g., perception operations, planning operations, or control operations.

In some implementations, the localization system230can determine a current position of the autonomous platform. A current position can include a global position (e.g., respecting a georeferenced anchor, etc.) or relative position (e.g., respecting objects in the environment, etc.). The localization system230can generally include or interface with any device or circuitry for analyzing a position or change in position of an autonomous platform (e.g., autonomous ground-based vehicle, etc.). For example, the localization system230can determine position by using one or more of: inertial sensors (e.g., inertial measurement unit(s), etc.), a satellite positioning system, radio receivers, networking devices (e.g., based on IP address, etc.), triangulation or proximity to network access points or other network components (e.g., cellular towers, Wi-Fi access points, etc.), or other suitable techniques. The position of the autonomous platform can be used by various subsystems of the autonomy system200or provided to a remote computing system (e.g., using the communication interface(s)206).

In some implementations, the localization system230can register relative positions of elements of a surrounding environment of an autonomous platform with recorded positions in the map data210. For instance, the localization system230can process the sensor data204(e.g., LIDAR data, RADAR data, camera data, etc.) for aligning or otherwise registering to a map of the surrounding environment (e.g., from the map data210) to understand the autonomous platform's position within that environment. Accordingly, in some implementations, the autonomous platform can identify its position within the surrounding environment (e.g., across six axes, etc.) based on a search over the map data210. In some implementations, given an initial location, the localization system230can update the autonomous platform's location with incremental re-alignment based on recorded or estimated deviations from the initial location. In some implementations, a position can be registered directly within the map data210.

In some implementations, the map data210can include a large volume of data subdivided into geographic tiles, such that a desired region of a map stored in the map data210can be reconstructed from one or more tiles. For instance, a plurality of tiles selected from the map data210can be stitched together by the autonomy system200based on a position obtained by the localization system230(e.g., a number of tiles selected in the vicinity of the position).

In some implementations, the localization system230can determine positions (e.g., relative or absolute) of one or more attachments or accessories for an autonomous platform. For instance, an autonomous platform can be associated with a cargo platform, and the localization system230can provide positions of one or more points on the cargo platform. For example, a cargo platform can include a trailer or other device towed or otherwise attached to or manipulated by an autonomous platform, and the localization system230can provide for data describing the position (e.g., absolute, relative, etc.) of the autonomous platform as well as the cargo platform. Such information can be obtained by the other autonomy systems to help operate the autonomous platform.

The autonomy system200can include the perception system240, which can allow an autonomous platform to detect, classify, and track objects and actors in its environment. Environmental features or objects perceived within an environment can be those within the field of view of the sensor(s)202or predicted to be occluded from the sensor(s)202. This can include object(s) not in motion or not predicted to move (static objects) or object(s) in motion or predicted to be in motion (dynamic objects/actors).

The perception system240can determine one or more states (e.g., current or past state(s), etc.) of one or more objects that are within a surrounding environment of an autonomous platform. For example, state(s) can describe (e.g., for a given time, time period, etc.) an estimate of an object's current or past location (also referred to as position); current or past speed/velocity; current or past acceleration; current or past heading; current or past orientation; size/footprint (e.g., as represented by a bounding shape, object highlighting, etc.); classification (e.g., pedestrian class vs. vehicle class vs. bicycle class, etc.); the uncertainties associated therewith; or other state information. In some implementations, the perception system240can determine the state(s) using one or more algorithms or machine-learned models configured to identify/classify objects based on inputs from the sensor(s)202. The perception system can use different modalities of the sensor data204to generate a representation of the environment to be processed by the one or more algorithms or machine-learned models. In some implementations, state(s) for one or more identified or unidentified objects can be maintained and updated over time as the autonomous platform continues to perceive or interact with the objects (e.g., maneuver with or around, yield to, etc.). In this manner, the perception system240can provide an understanding about a current state of an environment (e.g., including the objects therein, etc.) informed by a record of prior states of the environment (e.g., including movement histories for the objects therein). Such information can be helpful as the autonomous platform plans its motion through the environment.

The autonomy system200can include the planning system250, which can be configured to determine how the autonomous platform is to interact with and move within its environment. The planning system250can determine one or more motion plans for an autonomous platform. A motion plan can include one or more trajectories (e.g., motion trajectories) that indicate a path for an autonomous platform to follow. A trajectory can be of a certain length or time range. The length or time range can be defined by the computational planning horizon of the planning system250. A motion trajectory can be defined by one or more waypoints (with associated coordinates). The waypoint(s) can be future location(s) for the autonomous platform. The motion plans can be continuously generated, updated, and considered by the planning system250.

The motion planning system250can determine a strategy for the autonomous platform. A strategy may be a set of discrete decisions (e.g., yield to actor, reverse yield to actor, merge, lane change) that the autonomous platform makes. The strategy may be selected from a plurality of potential strategies. The selected strategy may be a lowest cost strategy as determined by one or more cost functions. The cost functions may, for example, evaluate the probability of a collision with another actor or object.

The planning system250can determine a desired trajectory for executing a strategy. For instance, the planning system250can obtain one or more trajectories for executing one or more strategies. The planning system250can evaluate trajectories or strategies (e.g., with scores, costs, rewards, constraints, etc.) and rank them. For instance, the planning system250can use forecasting output(s) that indicate interactions (e.g., proximity, intersections, etc.) between trajectories for the autonomous platform and one or more objects to inform the evaluation of candidate trajectories or strategies for the autonomous platform. In some implementations, the planning system250can utilize static cost(s) to evaluate trajectories for the autonomous platform (e.g., “avoid lane boundaries,” “minimize jerk,” etc.). Additionally, or alternatively, the planning system250can utilize dynamic cost(s) to evaluate the trajectories or strategies for the autonomous platform based on forecasted outcomes for the current operational scenario (e.g., forecasted trajectories or strategies leading to interactions between actors, forecasted trajectories or strategies leading to interactions between actors and the autonomous platform, etc.). The planning system250can rank trajectories based on one or more static costs, one or more dynamic costs, or a combination thereof. The planning system250can select a motion plan (and a corresponding trajectory) based on a ranking of a plurality of candidate trajectories. In some implementations, the planning system250can select a highest ranked candidate, or a highest ranked feasible candidate.

The planning system250can then validate the selected trajectory against one or more constraints before the trajectory is executed by the autonomous platform.

To help with its motion planning decisions, the planning system250can be configured to perform a forecasting function. The planning system250can forecast future state(s) of the environment. This can include forecasting the future state(s) of other actors in the environment. In some implementations, the planning system250can forecast future state(s) based on current or past state(s) (e.g., as developed or maintained by the perception system240). In some implementations, future state(s) can be or include forecasted trajectories (e.g., positions over time) of the objects in the environment, such as other actors. In some implementations, one or more of the future state(s) can include one or more probabilities associated therewith (e.g., marginal probabilities, conditional probabilities). For example, the one or more probabilities can include one or more probabilities conditioned on the strategy or trajectory options available to the autonomous platform. Additionally, or alternatively, the probabilities can include probabilities conditioned on trajectory options available to one or more other actors.

In some implementations, the planning system250can perform interactive forecasting. The planning system250can determine a motion plan for an autonomous platform with an understanding of how forecasted future states of the environment can be affected by execution of one or more candidate motion plans.

By way of example, with reference again toFIG.1, the autonomous platform110can determine candidate motion plans corresponding to a set of platform trajectories112A-C that respectively correspond to the first actor trajectories122A-C for the first actor120, trajectories132for the second actor130, and trajectories142for the third actor140(e.g., with respective trajectory correspondence indicated with matching line styles). The autonomous platform110can evaluate each of the potential platform trajectories and predict its impact on the environment.

For example, the autonomous platform110(e.g., using its autonomy system200) can determine that a platform trajectory112A would move the autonomous platform110more quickly into the area in front of the first actor120and is likely to cause the first actor120to decrease its forward speed and yield more quickly to the autonomous platform110in accordance with a first actor trajectory122A.

Additionally or alternatively, the autonomous platform110can determine that a platform trajectory112B would move the autonomous platform110gently into the area in front of the first actor120and, thus, may cause the first actor120to slightly decrease its speed and yield slowly to the autonomous platform110in accordance with a first actor trajectory122B.

Additionally or alternatively, the autonomous platform110can determine that a platform trajectory112C would cause the autonomous vehicle to remain in a parallel alignment with the first actor120and, thus, the first actor120is unlikely to yield any distance to the autonomous platform110in accordance with first actor trajectory122C.

Based on comparison of the forecasted scenarios to a set of desired outcomes (e.g., by scoring scenarios based on a cost or reward), the planning system250can select a motion plan (and its associated trajectory) in view of the autonomous platform's interaction with the environment100. In this manner, for example, the autonomous platform110can interleave its forecasting and motion planning functionality.

To implement selected motion plan(s), the autonomy system200can include a control system260(e.g., a vehicle control system). Generally, the control system260can provide an interface between the autonomy system200and the platform control devices212for implementing the strategies and motion plan(s) generated by the planning system250. For instance, control system260can implement the selected motion plan/trajectory to control the autonomous platform's motion through its environment by following the selected trajectory (e.g., the waypoints included therein). The control system260can, for example, translate a motion plan into instructions for the appropriate platform control devices212(e.g., acceleration control, brake control, steering control, etc.). By way of example, the control system260can translate a selected motion plan into instructions to adjust a steering component (e.g., a steering angle) by a certain number of degrees, apply a certain magnitude of braking force, increase/decrease speed, etc. In some implementations, the control system260can communicate with the platform control devices212through communication channels including, for example, one or more data buses (e.g., controller area network (CAN), etc.), onboard diagnostics connectors (e.g., OBD-II, etc.), or a combination of wired or wireless communication links. The platform control devices212can send or obtain data, messages, signals, etc. to or from the autonomy system200(or vice versa) through the communication channel(s).

The autonomy system200can receive, through communication interface(s)206, assistive signal(s) from remote assistance system270. Remote assistance system270can communicate with the autonomy system200over a network (e.g., as a remote system160over network170). In some implementations, the autonomy system200can initiate a communication session with the remote assistance system270. For example, the autonomy system200can initiate a session based on or in response to a trigger. In some implementations, the trigger may be an alert, an error signal, a map feature, a request, a location, a traffic condition, a road condition, etc.

After initiating the session, the autonomy system200can provide context data to the remote assistance system270. The context data may include sensor data204and state data of the autonomous platform. For example, the context data may include a live camera feed from a camera of the autonomous platform and the autonomous platform's current speed. An operator (e.g., human operator) of the remote assistance system270can use the context data to select assistive signals. The assistive signal(s) can provide values or adjustments for various operational parameters or characteristics for the autonomy system200. For instance, the assistive signal(s) can include way points (e.g., a path around an obstacle, lane change, etc.), velocity or acceleration profiles (e.g., speed limits, etc.), relative motion instructions (e.g., convoy formation, etc.), operational characteristics (e.g., use of auxiliary systems, reduced energy processing modes, etc.), or other signals to assist the autonomy system200.

Autonomy system200can use the assistive signal(s) for input into one or more autonomy subsystems for performing autonomy functions. For instance, the planning subsystem250can receive the assistive signal(s) as an input for generating a motion plan. For example, assistive signal(s) can include constraints for generating a motion plan. Additionally, or alternatively, assistive signal(s) can include cost or reward adjustments for influencing motion planning by the planning subsystem250. Additionally, or alternatively, assistive signal(s) can be considered by the autonomy system200as suggestive inputs for consideration in addition to other received data (e.g., sensor inputs, etc.).

The autonomy system200may be platform agnostic, and the control system260can provide control instructions to platform control devices212for a variety of different platforms for autonomous movement (e.g., a plurality of different autonomous platforms fitted with autonomous control systems). This can include a variety of different types of autonomous vehicles (e.g., sedans, vans, SUVs, trucks, electric vehicles, combustion power vehicles, etc.) from a variety of different manufacturers/developers that operate in various different environments and, in some implementations, perform one or more vehicle services.

For example, with reference toFIG.3A, an operational environment can include a dense environment300. An autonomous platform can include an autonomous vehicle310controlled by the autonomy system200. In some implementations, the autonomous vehicle310can be configured for maneuverability in a dense environment, such as with a configured wheelbase or other specifications. In some implementations, the autonomous vehicle310can be configured for transporting cargo or passengers. In some implementations, the autonomous vehicle310can be configured to transport numerous passengers (e.g., a passenger van, a shuttle, a bus, etc.). In some implementations, the autonomous vehicle310can be configured to transport cargo, such as large quantities of cargo (e.g., a truck, a box van, a step van, etc.) or smaller cargo (e.g., food, personal packages, etc.).

With reference toFIG.3B, a selected overhead view302of the dense environment300is shown overlaid with an example trip/service between a first location304and a second location306. The example trip/service can be assigned, for example, to an autonomous vehicle320by a remote computing system. The autonomous vehicle320can be, for example, the same type of vehicle as autonomous vehicle310. The example trip/service can include transporting passengers or cargo between the first location304and the second location306. In some implementations, the example trip/service can include travel to or through one or more intermediate locations, such as to onload or offload passengers or cargo. In some implementations, the example trip/service can be prescheduled (e.g., for regular traversal, such as on a transportation schedule). In some implementations, the example trip/service can be on-demand (e.g., as requested by or for performing a taxi, rideshare, ride hailing, courier, delivery service, etc.).

With reference toFIG.3C, in another example, an operational environment can include an open travel way environment330. An autonomous platform can include an autonomous vehicle350controlled by the autonomy system200. This can include an autonomous tractor for an autonomous truck. In some implementations, the autonomous vehicle350can be configured for high payload transport (e.g., transporting freight or other cargo or passengers in quantity), such as for long distance, high payload transport. For instance, the autonomous vehicle350can include one or more cargo platform attachments such as a trailer352. Although depicted as a towed attachment inFIG.3C, in some implementations one or more cargo platforms can be integrated into (e.g., attached to the chassis of, etc.) the autonomous vehicle350(e.g., as in a box van, step van, etc.).

With reference toFIG.3D, a selected overhead view of open travel way environment330is shown, including travel ways332, an interchange334, transfer hubs336and338, access travel ways340, and locations342and344. In some implementations, an autonomous vehicle (e.g., the autonomous vehicle310or the autonomous vehicle350) can be assigned an example trip/service to traverse the one or more travel ways332(optionally connected by the interchange334) to transport cargo between the transfer hub336and the transfer hub338. For instance, in some implementations, the example trip/service includes a cargo delivery/transport service, such as a freight delivery/transport service. The example trip/service can be assigned by a remote computing system. In some implementations, the transfer hub336can be an origin point for cargo (e.g., a depot, a warehouse, a facility, etc.) and the transfer hub338can be a destination point for cargo (e.g., a retailer, etc.). However, in some implementations, the transfer hub336can be an intermediate point along a cargo item's ultimate journey between its respective origin and its respective destination. For instance, a cargo item's origin can be situated along the access travel ways340at the location342. The cargo item can accordingly be transported to transfer hub336(e.g., by a human-driven vehicle, by the autonomous vehicle310, etc.) for staging. At the transfer hub336, various cargo items can be grouped or staged for longer distance transport over the travel ways332.

In some implementations of an example trip/service, a group of staged cargo items can be loaded onto an autonomous vehicle (e.g., the autonomous vehicle350) for transport to one or more other transfer hubs, such as the transfer hub338. For instance, although not depicted, it is to be understood that the open travel way environment330can include more transfer hubs than the transfer hubs336and338and can include more travel ways332interconnected by more interchanges334. A simplified map is presented here for purposes of clarity only. In some implementations, one or more cargo items transported to the transfer hub338can be distributed to one or more local destinations (e.g., by a human-driven vehicle, by the autonomous vehicle310, etc.), such as along the access travel ways340to the location344. In some implementations, the example trip/service can be prescheduled (e.g., for regular traversal, such as on a transportation schedule). In some implementations, the example trip/service can be on-demand (e.g., as requested by or for performing a chartered passenger transport or freight delivery service).

To improve the performance of an autonomous platform, such as an autonomous vehicle controlled at least in part using autonomy system200(e.g., the autonomous vehicles310or350), the autonomous platform can implement validation techniques described herein can be implemented according to example aspects of the present disclosure.

FIG.4is a block diagram of an evaluation system400, according to some implementations of the present disclosure. AlthoughFIG.4illustrates an example implementation of an evaluation system400having and interacting with various components, it is to be understood that the components can be rearranged, combined, omitted, etc. within the scope of and consistent with the present disclosure.

Reference data402can include data describing a reference scene404. Reference scene404can be a snapshot or recording of sensor data describing an environment. Execution environment405can host a system under test (SUT)406that is to be evaluated by evaluation system400. SUT406can include a perception system that processes the sensor data of reference scene404to generate an understanding of the environment described thereby.

Evaluation system400can obtain labeled object data408that identifies objects described by reference scene404. Evaluation system400can obtain object detection data410generated using SUT406. Evaluation system400can compare labeled object data408and object detection data410using measurement block412. Measurement block412can generate divergence metrics414that represent differences between labeled object data408and object detection data410. Additionally, measurement block412can evaluate one or more context metrics416that provide additional information regarding the context in which the measured differences of divergence metrics414arise. Measurement block412can output those measured differences and the context for those differences to machine-learned calibration model(s)418.

Machine-learned calibration model418can process the measurements and contexts from measurement block412and generate a score420that can characterize whether object detection data410are materially different from labeled object data408. For example, machine-learned calibration model418can process divergence metrics414and SUT contexts416and evaluate whether the detected differences amount to an overall material difference. For example, machine-learned calibration model418can determine the influence of divergence metrics414on the resulting score420. In this manner, for instance, machine-learned calibration model418can calibrate the influence of the differences between labeled object data408and object detection data410.

Evaluation system400can output an evaluation state422for SUT406based on score420. For instance, evaluation system400can output a positive evaluation state if score420is below a threshold (e.g., if object detection data410are not materially different from labeled object data408).

Reference data402can include recorded instances of real-world or simulated driving. The recorded data can include data collected by sensors onboard one or more vehicles (e.g., autonomous vehicles, non-autonomous vehicles, etc.). The recorded data can include data collected from other sources (e.g., roadside cameras, aerial vehicles, etc.). Reference data402from simulated scenarios can include probabilistic data, such as data sampled from a distribution fitted to a number of observations.

Reference data402can include trajectory data. For example, reference data402can include recorded trajectories of an actor and data describing the environment in which the actor moves (e.g., map data, perception data). Reference data402can include real or synthetic trajectories. Real trajectories can include trajectories traversed by a vehicle in a real-world environment (e.g., by a human-driven vehicle, an autonomous vehicle). Synthetic trajectories can include trajectories traversed by a simulated vehicle in a simulated environment (e.g., a simulation implementing an autonomous vehicle control system to control a simulated vehicle). Synthetic trajectories can include trajectories drawn or otherwise annotated using a review system (e.g., by a human annotator, an automated annotator) to indicate a trajectory that a vehicle should travel in a given situation.

Trajectory data can be used to generate synthetic perception data by simulating outputs of sensors simulated to be moving through a simulated environment along a given trajectory. For instance, a raycasting origin location can be moved through a simulated environment over time in alignment with a trajectory from trajectory data. In this manner, for instance, synthetic sensor data can be generated from recorded trajectories.

Reference data402can include logged sensor data or otherwise recorded sensor data. Sensor data can be logged during real-world or simulated driving. Sensor data can be logged by sensors on vehicles or sensors not on vehicles. Sensor data can be obtained from sensors used on autonomous devices or systems or non-autonomous devices or systems.

Reference data402can include various kinds of data. Reference data402can include real or simulated LIDAR data, RADAR data, image data, audio data, position data, velocity data, acceleration data, orientation data, or any other data captured by or otherwise obtained using sensors that can record information describing a surrounding environment or a system's interaction with the environment.

Reference scene404can include a portion of reference data402that describes a scene or portion of an environment that is of interest. Reference scene404can be characterized as a segment of a larger driving sequence. For example, reference data402can include data describing a certain number of seconds or minutes of a larger driving log. Reference scene404can include data that focuses on a particular scene in which a particular object was recorded by one or more sensors. Reference scene404can include a subset of or all of the data types from reference data402describing an environment, including objects, actors, infrastructure features, etc.

Evaluation system400can select reference scene404for evaluating SUT406based on one or more attributes of reference scene404. For example, reference scene404can describe a particular situation of interest. Evaluation system400can evaluate the performance of an autonomous vehicle system (e.g., perception system) in that particular situation of interest by deploying the autonomous vehicle system as SUT406. For example, reference scene404can correspond to a standardized test scenario established by regulation (e.g., regulations promulgated by one or more government agencies). For example, reference scene404can include a particular object in a particular environment viewed from a standard perspective. For example, a standard test scene can be configured to test object detection under high occlusion or other challenging conditions.

Evaluation system400can select reference scene404from a benchmark set of tests maintained by evaluation system400. For example, evaluation system400can maintain a benchmark set that identifies reference scenes404(or labeled objects408) that explore edge cases of detection under different conditions. The benchmark set can be configured to provide examples that help define a boundary of SUT behavior. If a version of an SUT fails one or more of the benchmark tests, it may be determined that the SUT did not achieve or had regressed in performance from a desired level.

For example, the benchmark set can include a first reference scene in which an object is in a position that has a high tolerance for error (e.g., a position far removed from a roadway with minimal chance of intercept) and a second reference scene in which the object is in a different position that has a lower tolerance for error (e.g., nearer to the roadway). While it may be difficult to precisely identify the exact position at which a certain amount of detection error does not materially affect the quality of the perception of the object, evaluating the SUT against both the first and second reference scenes can provide an initial indication that the SUT can make the correct or expected decisions in each of the reference scenes. In this manner, for instance, evaluating SUT406against a benchmark set that contains examples that constrain decision boundaries can help identify whether SUT406adheres to the desired decision boundaries. As more and more reference scenes are added to the benchmark set, the decision boundaries of SUT406can be tested with increasing precision. If SUT406can achieve satisfactory performance over the benchmark set, evaluation system400can determine that SUT406has demonstrated at least a benchmark level of performance in a baseline range of situations and is likely to resolve novel situations consistent with the boundaries constrained by the benchmark set.

Execution environment405may be a real or simulated environment. For example, if SUT406include a perception system that is configured to receive sensor data inputs and generate perception data outputs (e.g., tracked object data), an operating platform can include a real or virtual machine with one or more real or virtual processors, memory, storage, etc. that enable execution of one or more operations of the perception system. Execution environment405can facilitate operation or simulation of sensor devices as well for end-to-end evaluation of SUT406.

System under test (SUT)406can be or include one or more operational systems of an autonomous vehicle. For instance, SUT406can include one or more autonomy systems or one or more systems operating in support of an autonomy system. For instance, SUT406can include one or more portions of autonomy system200, such as a localization subsystem230, a perception subsystem240, a planning subsystem250, a control subsystem260, etc. In some examples, SUT406can include real or simulated sensor(s)202, communication interface(s)206, remote assistance system270, platform control devices212, etc. SUT406can include one or more machine-learned models.

Labeled object data408can include ground truth object detection data. For instance, a labeled object can include data describing an object (e.g., LIDAR points associated with an object, image data depicting an object, RADAR data, etc.) paired with a label identifying the object (e.g., a bounding box, a centroid marker, an image segment, etc.). In general, labeled object data408can be any data registering a semantically meaningful object identity with a portion of sensor data. Labeled object data408can include spatial labels (e.g., bounding boxes, points, markers, etc.), temporal labels (e.g., keyframes, interval endpoints, etc. associated with an object appearance), semantic labels (e.g., an object type, object description, etc.), motion labels (e.g., heading, velocity, acceleration, etc.), etc.

Labeled object data408can be manually or automatically generated, or some combination thereof. For instance, an image recognition system can process images to automatically generate labels. Those labels can be reviewed manually. The automatically generated labels or the manually confirmed labels can be stored in association with the images to provide a ground truth example of a set of inputs (e.g., the sensor data) and the desired label. Labeled object data408can include 2D or 3D bounding boxes. A bounding box can be drawn over sequences of sensor data captures in time to obtain an additional dimension.

Labeled object data408can be filtered or otherwise restricted to information that is knowable to the SUT406at test time. For instance, review of all sensor data in an offline setting can enable fully or partially occluded objects to be fully recognized, labeled, and characterized. However, at test time, SUT406may only have limited visibility of the object. For instance, a vehicle pulling a trailer might be positioned such that SUT406does not have visibility of the trailer. In this situation, a label that indicates the position and orientation of the trailer may not be helpful for evaluating SUT406. For example, it may not be helpful to penalize SUT406for not knowing unknowable information.

Object detection data410can describe outputs describing objects detected by SUT406based on processing reference scene404. Object detection data410can include, for instance, bounding boxes or other recognition markers or outputs generated by SUT406around objects detected in reference scene404. In general, object detection data410can be any data registering a semantically meaningful object identity with a portion of sensor data. Object detection data410can include spatial labels (e.g., bounding boxes, points, markers, etc.), temporal labels (e.g., keyframes, interval endpoints, etc. associated with an object appearance), semantic labels (e.g., an object type, object description, etc.), motion labels (e.g., heading, velocity, acceleration, etc.), etc.

Measurement block412can include logic executed by evaluation system400to extract salient features of labeled object data408and object detection data410. Measurement block412can include one or more machine-learned models or components. For example, measurements by measurement block412can be directly extracted from labeled object data408or object detection data410, can be inferred therefrom using a machine-learned model, or can be obtained using one or more transformations applied to labeled object data408or object detection data410.

In general, measurement block412can generate values that represent aspects in which object detection data410diverge from labeled object data408. Divergence can be represented by differences between labeled object data408and object detection data410. Differences can include differences between predicted and actual spatial labels, predicted and actual temporal labels, predicted and actual semantic labels, etc.

Measurement block412can estimate or approximate divergences using a collection of divergence metrics that can be aggregated to obtain an overall measure of divergence.

Divergence metrics414can include functions, operators, or other components that are configured to compute divergence values that characterize differences between labeled object data408and object detection data410. Divergence metrics414can compute differences between labeled object data408and object detection data410. For instance, divergence metrics414can compute a difference in a bounding box predicted by SUT406and a bounding box associated with labeled object data408. For example, a divergence metric can compute box intersection over union (IoU) (e.g., 2D or 3D). The IoU can be computed in various different planes. For instance, the boxes can be projected into an image plane as viewed from the perspective of the ego vehicle associated with SUT406. A divergence metric can compute a difference between how close the nearest point of each box is to a reference point (e.g., a point on a vehicle associated with the SUT, such as an autonomous vehicle operating, in reality or in simulation, the SUT). A divergence metric can compute differences in position or orientation. A divergence metric can compute differences in a rate of change of position or orientation (e.g., velocity, acceleration, jerk, etc.). A divergence metric can compute a difference in volume of a portion of space occupied by sensor data associated with a given object.

A divergence metric can compute differences at one or more points in time. For instance, a test time can be associated with the capture time of the sensor data. Small errors at test time can lead to increasingly large errors over time. For instance, a small divergence in heading at capture time could lead to larger divergences at later times. To evaluate these later-time divergences, evaluation system400can compare a forecast of an object position against a labeled object position at a later time. For instance, SUT406or evaluation system400can generate a forecasted object position at the later time based on the detected object position indicated in object detection data410. To provide a reference, SUT406or evaluation system400can generate a forecasted object position at the later time based on the labeled object position indicated in labeled object data408. To provide a reference, evaluation system400can retrieve a labeled object position associated with the later time. For example, the divergence metrics can be compared at capture time and 500 ms in the future, 1 s in the future, 2 s in the future, 5 s in the future, etc.

Divergence metrics414can evaluate differences in latent or implicit attributes. Divergence metrics414can compute projected or embedded features that implicitly encode meaningful information regarding the scene. Divergence metrics414can include one or more machine-learned components. For example, a divergence encoder can be configured to generate a respective divergence value by processing, using one or more machine-learned parameters, at least a portion of reference scene404, labeled object data408, detected object data410, etc. The divergence encoder can be trained end-to-end within evaluation system400to generate divergence values that correspond to meaningful differences.

SUT context metrics416can include or be based on data describing pertinent characteristics of object detection data410with respect to reference scene404. For example, context metrics416can correspond to a relative importance of a particular divergence metric414. For example, a divergence metric can compute a difference in orientation of a bounding box. This difference might be immaterial, however, if the bounding box is for a parked vehicle on the shoulder of a roadway. Thus, an example SUT context metric416can determine the lane position of the object. Such a value can contextualize the generated divergence value (e.g., scale, weight, deprioritize, etc.).

Context metrics416can be generated by hand-tuned or engineered components. Engineered components can implement inductive or deductive operations. For instance, an engineered logic or rule can be deduced a priori from laws of physics, kinematics, known constraints, etc. For example, lane position can be an engineered context because it is derived from an a priori understanding of preferences and expectations for road users (e.g., that shouldered vehicles behave differently than vehicles in a travel lane).

Context metrics416can be generated by machine-learned components. Machine-learned components can perform inference over inputs to generate outputs. For instance, machine-learned components can infer, based on patterns seen across many training examples, that a particular input maps to a particular output. For example, a context value that contextualizes a particular comparison value can be generated by a machine-learned model. The model can be trained to contextualize comparison values in a manner that improves an evaluation capability of evaluation system400(e.g., decreases false positives, decreases false negatives, etc.).

Context metrics416can be continuous, piecewise continuous, or discretized. For example, context metrics416can define bins of contextual features that adjust importance for divergence metrics414when the bin is satisfied. For instance, a context metric can include a weather status (e.g., raining, not raining, etc.). Certain divergence metrics can be more impactful if inclement weather impedes visibility, decreases road surface friction, etc. Based on the presence of rain or no rain, a different context value can be obtained that can modify (e.g., weight, scale, etc.) a divergence value.

Measurement block412can generate values that are strictly non-increasing in “goodness” or desirability. Measurement block412can generate values that are strictly non-decreasing in divergence. For example, values of divergence metrics414, context metrics416, or both (e.g., the products thereof) can be determined such that as the magnitude of the values of divergence metrics414, context metrics416, or both (e.g., the products thereof) increase, the agreement or match between labeled object data408and object detection data410can be strictly non-increasing. This constraint can facilitate efficient construction of decision boundaries for individual parameters. For instance, under such a constraint, it can be noted that an increase in one metric orthogonally to all others (e.g., “all else being equal”) will decrease alignment between labeled object data408and object detection data410. This can increase the interpretability of evaluation system400.

Machine-learned calibration model418can reason over the outputs of measurement block412to generate a score420. Machine-learned calibration model418can include various different architectures, models, and model components. Machine-learned calibration model418can be or include a linear model. Machine-learned calibration model418can be or include a nonlinear model.

Machine-learned calibration model418can calibrate the influence of the differences between labeled object data408and object detection data410. For example, learnable parameters of machine-learned calibration model418can weight the values of divergence metrics414, context metrics416, or both (e.g., the products thereof). For example, machine-learned calibration model418can generate a learned linear combination of divergence metrics414, context metrics416, or both (e.g., the products thereof). For example, machine-learned calibration model418can include or generate attention values over divergence metrics414, context metrics416, or both (e.g., the products thereof) that indicate how much to attend to respective values of divergence metrics414, context metrics416, or both (e.g., the products thereof) when generating an overall score.

Score420can represent an overall match or alignment between labeled object data408and object detection data410. Score420can quantify how much object detection data410diverge from labeled object data408. Score420can be an aggregate score that indicates an aggregate divergence between labeled object data408and object detection data410. Evaluation system400can compare score420against a threshold to determine whether an amount of divergence is material. For instance, a score below a threshold can correspond to immaterial divergence (e.g., object detection data410are effectively as “good” as labeled object data408, even if they are different in some respects). A score above a threshold can correspond to material divergence (e.g., object detection data410are not considered to be as “good” as labeled object data408according to a desired validation precision). Evaluation system400can output evaluation state422based on score420.

Evaluation state422can indicate a determination of quality of the object detection. An evaluation state can be a validation state. A validation state can indicate a positive validation or a lack of validation. A validation state can indicate that SUT406at least satisfies a benchmark level of performance.

Evaluation system400(e.g., machine-learned calibration model418) can be trained using a set of labeled matches. Each set of labeled matches can be a unit test. For example, labeled matches can include sets of data that are confirmed to diverge either materially or immaterially. For example, labeled matches can include an object detection and a corresponding label that are confirmed to diverge in a material manner. Labeled matches can include an object detection and a corresponding label that are confirmed to not diverge in a material manner. Training evaluation system400can include updating learnable parameters of machine-learned calibration model418until evaluation system400correctly labels the input labeled matches (e.g., correctly determines that the matches either diverge materially or do not diverge materially). For example, evaluation system400can include a type of support vector machine, and labeled matches can provide support vectors that help define a desired decision boundary.

If machine-learned calibration model418does not or cannot converge to a set of weights that enables correct labeling of all unit tests, then evaluation system400can add additional expressivity to more fully model the task. For example, evaluation system400can compute additional divergence metrics. Evaluation system400can use additional context metrics (or more nuanced or granular versions of existing metrics). Evaluation system400can add additional learnable parameters to machine-learned model418. Increasing the expressivity of evaluation system400can increase a precision with which evaluation system400can model a desired decision boundary between detections that “match” or are aligned closely enough to ground truth and detections that do not “match” or are not aligned closely enough.

FIG.5is a block diagram of a measurement block412according to some implementations of the present disclosure. Measurement block412can process a reference scene500. Reference scene500can describe an environment containing an ego vehicle (e.g., an AV or other vehicle) at ego position502. Reference scene500can describe a labeled object position504-L. SUT406can generate a detected object position504-D based on reference scene500. Measurement block412can process ego position502, labeled object position504-L, and detected object position504-D using divergence metrics414and context metrics416.

For example, divergence metrics414can compute a number of divergence values506-1,506-2, . . . ,506-M that characterize M differences between labeled object position504-L and detected object position504-D. An example divergence metric can compute a divergence value that indicates difference in orientation (e.g., an angular displacement) between labeled object position504-L and detected object position504-D.

Context metrics416can compute N context values508-1,508-2, . . . ,508-N that characterize N attributes of any one or more of ego position502, labeled object position504-L, or detected object position504-D. An example context metric can compute a context value that corresponds to a lane position of the object. For instance, in scene500, the object is shouldered. Another example context metric can compute a relative distance between labeled object position504-L and ego position502. For example, certain errors in detected object position504-D may be less material when labeled object position504-L is farther away from ego position502. Similarly, another example context metric can compute a relative distance between detected object position504-D and ego position502. For example, certain errors in detected object position504-D may be more material when detected object position504-D closer to ego position502(e.g., false positive detections that may induce unnecessary evasive maneuvers).

Another example context metric can compute a measure of how occluded the detected object is from the point of view of the ego vehicle (e.g., a value indicating a proportion of the object that is visible or is not visible). Another example context metric can compute a measure of a distance between the labeled object position504-L or the detected object position504-D and a road surface (e.g., how far away from the roadway).

Another example context metric can compute a measure of an estimated time to arrival at a present or forecasted location associated with the object (e.g., a present or forecasted labeled object position504-L or the detected object position504-D). For instance, an example context metric can compute a number of seconds until the ego vehicle could be expected to intersect a boundary of the object. The number of seconds can be estimated at a minimum, a lower bound, or under a set of assumed conditions such that the estimated interval is associated with a high probability of being less than the actual value. For instance, the estimated interval can be determined under a set of assumed conditions such that the estimate falls in a low percentile of a distribution of possible intervals (e.g., 1%, 0.1%, 0.01%, etc.).

Another example context metric can indicate a likelihood associated with arrival at a present or forecasted location associated with the object (e.g., a present or forecasted labeled object position504-L or the detected object position504-D). For instance, an example context metric can indicate a likelihood associated with a control sequence that would lead to arrival at a present or forecasted location associated with the object. For instance, an example context metric can indicate a likelihood associated with a trajectory that includes an ego vehicle position that overlaps a location associated with the object (e.g., to intersect a boundary of the object).

Machine-learned calibration model418can process the divergence values and the context values to generate score510. Score510can indicate an aggregate divergence between labeled object box504-L and detected object box504-D.

InFIG.5, the error between labeled object box504-L and detected object box504-D may be determined to be immaterial. For instance, score510may not satisfy a threshold that indicates a material aggregate divergence. It may be intuitively understood, for example, that errors in position of a shouldered vehicle may be relatively less important than other perception errors. As such, detected object box504-D may be sufficiently accurate in that context.

FIG.6illustrates evaluation of a different scene. InFIG.6, measurement block412receives data describing a reference scene600that is associated with ego position602and labeled object box604-L (which can be the same as or different from labeled object box504-L). Measurement block412can receive a different detected object box604-D in response to SUT406processing reference scene600.

Measurement block412can process ego position602, labeled object position604-L, and detected object position604-D using divergence metrics414and context metrics416. For example, divergence metrics414can compute a number of divergence values606-1,606-2, . . . ,606-M that characterize M differences between labeled object position604-L and detected object position604-D. Context metrics416can compute N context values608-1,608-2, . . . ,608-N that characterize N attributes of any one or more of ego position602, labeled object position604-L, or detected object position604-D. Machine-learned calibration model418can process the divergence values and the context values to generate score610. Score610can indicate an aggregate divergence between labeled object box604-L and detected object box604-D.

InFIG.6, the error between labeled object box604-L and detected object box604-D may be determined to be material. For instance, score610can satisfy a threshold that indicates a material aggregate divergence. For example, as compared to the error illustrated inFIG.5, the error illustrated inFIG.6occurs for a vehicle in a lane of traffic (as compared to a shouldered vehicle). In the context of a vehicle in a traffic lane, even the same amount of measured difference (e.g., angular displacement) may be material. Reflecting this increased significance, a context metric associated with lane position can generate a higher context value as compared to reference scene500, because the prediction relates to a more significant lane position.

Context metrics can compute context values based on one or more inputs. Context metrics can include linear or nonlinear functions of one or more parameters. Context metrics can be continuous, piecewise continuous, or discrete or disjoint. In an example, context values can be piecewise continuous over one or more semantically meaningful subdivisions of a domain of an input space. For example, a quantified expression of significance or importance may be different in different basins of context. For example, a cost associated with angular position error may have a steep slope in an adjacent travel lane (e.g., to cause a high cost of error for vehicles moving directly alongside the ego vehicle). In comparison, a cost associated with angular position error may have a gentler slope for vehicles parked on a shoulder.

FIG.7is an example tree structure700that illustrates an approach to binning a domain of an input space for leveraging a piecewise context metric. Example tree structure700can divide an input space into two subdivisions based on, at702, a query about whether the object is positioned in a travel lane. If the response to702is True, tree structure700can further subdivide the domain at704based on whether the object is in a travel lane adjacent to the ego vehicle. If the response to704is True, tree structure700can associate a given object detection with bin706. Bin706can correspond to a specific context function adapted to situations in which an object is in an adjacent travel lane. If the response to704is False, tree structure700can associate a given object detection with bin708. Bin708can correspond to a specific context function adapted to situations in which an object is in a travel lane that is not adjacent (e.g., further away from the ego vehicle).

If the response to702is False, tree structure700can further subdivide the domain at710based on whether the object is moving. If the response to710is True, tree structure700can associate a given object detection with bin712. Bin712can correspond to a specific context function adapted to situations in which an object is moving in a shoulder lane. If the response to710is False, tree structure700can associate a given object detection with bin714. Bin714can correspond to a specific context function adapted to situations in which an object is parked in a shoulder lane.

Adaptation of context functions in each bin can include learned adaptation. For instance, a context function can have one or more learnable parameters (e.g., a constant weight, a linear slope, coefficients of a nonlinear function, weights in a neural network, etc.). By using tree structure700to apply particular functions in certain basins of context, and then by training the context functions based on their performance in their respective contexts, a system can learn specific context functions adapted to particular situations.

Contexts for various scenarios can be binned based on interpretable features that correspond to hand-crafted heuristics. In this manner, for instance, the categorization can facilitate high-confidence confirmation that performance in specific contexts will be prioritized. For instance, whether the object is on a shoulder of a roadway is known to change how significant various divergences can be. By building this world knowledge into the tree structure, the system can be biased to learn prioritizations that align with a priori understandings of significance to a driving task.

Additionally, or alternatively, the features that define the binning can be latent context features learned by a machine-learned model. For instance, machine-learned mixture models or other clustering models can be configured to describe a distribution of contexts or scenarios to identify and cluster groups of contexts that should be evaluated similarly.

FIG.8is a block diagram of an example configuration of a machine-learned calibration model418according to example aspects of the present disclosure. Context values508-1,508-2, . . . ,508-N can modify one or more of divergence values506-1,506-2, . . . ,506-M to obtain P matching features800-1,800-2, . . . ,800-P (e.g.,800-1can be a product of506-1and508-1). The matching features can reflect a coarse or initial estimate of an influence of various divergence values on score420based on the context value(s). Machine-learned calibration model418can include one or more weights that calibrate the influences of the P matching features800-1,800-2, . . . ,800-P. In an example, machine-learned calibration model418can include P weights802-1,802-2, . . . ,802-P that respectively correspond to the P matching features800-1,800-2, . . . ,800-P.

Matching features can be a linear product of a divergence value and a context value. Matching features can be based on the divergence values and the context values in more complex arrangements. For instance, context metrics can map to one or more bins of context states. One approach is to have a different context value for each bin, where the different context values can modify the same divergence metric. Another approach is to generate different divergence metric instances for each bin, such that as contexts for a particular object detection fall into a bin, the corresponding divergence metric is used for computing a divergence. This divergence metric can directly supply a matching feature x that may not be further adjusted by an explicit context value before being processed by machine-learned calibration model418.

A constraint over the context features can force the context metrics to have a monotonic effect on the resulting matching feature. For instance, the effect of the context can be constrained such that as the context value increases, the matching feature value increases as well gets larger, the match score is weighted more highly. This can be accomplished by using, as the indicator functions, step basis functions. The step basis functions can be activated at the beginning of the domain covered to each bin. Instead of indicator functions that “turn off” when exiting the bin (e.g., generating one-hot vector over a distribution of bins), example step functions can continue to be activated after an initial threshold is satisfied, such that the overall matching feature value grows cumulatively. For example, a resulting indicator vector, rather than having elements of value 0 everywhere with 1 at only one element, instead can contain multiple 1 values after an initial activation (e.g. [0,0,1,1,1] rather than [0,0,1,0,0]). In an example, this can provide a piecewise constant and monotonic function of the context. The magnitudes of the step functions can be determined by corresponding context values (e.g., corresponding to the bins) and thereby weight the divergence values differently in each bin.

A context value itself can be a piecewise linear and monotonic function. For instance, instead of step functions, ramp or other piecewise linear functions can be used in h (y). The ramps can be centered on a sequence of knot points or joints that anchor a particular range over which the context value is to vary. The joints can be positioned using learned transitions or hard-coded.

An efficient computational technique for computing these different metrics uses a tensor product (e.g., a Kronecker product) of a vector of indicator functions h (y) where y is a context value and a vector of divergence values x. Computing the tensor product can include generating a block matrix that contains a number and arrangement of blocks that respectively correspond to a number and arrangement of values in a first matrix (e.g., a 2×2 first matrix leads to a product with 2×2 blocks). In each block, a corresponding value of the first matrix uniformly scales an entire second matrix, such that the block size is the size of the second matrix. This computational structure using a linear classifier can allow for efficient expansion of a feature set.

For example, machine-learned calibration model418can generate a linear combination of the P matching features. In an example, let x represent a vector of the matching feature values and w represent a vector of P weights of machine-learned calibration model418. An example score810can be computed as wTx to indicate an aggregate divergence of the input detections. The vector w can be constrained to not flip a direction of a value in x (e.g., constrained to be positive) so as to not alter the predetermined effect of a divergence metric. This can aid interpretability and increase expressivity of the model, as the model is freed from having to learn the underlying physical, legal, or other causes of increased cost. A match can be determined as wTx<θ, where θ is a threshold value for determining a match.

In an example expression, let b=eigen_pwl(x, k) be an operator that receives an input divergence metric value x and a vector of knot points k and returns a basis vector b such that a dot product of the basis vector b with a vector of slopes s provides a piecewise linear function F having the specified slopes.

An example divergence metric Fdivergencefor a measurement m can then be expressed as
Fdivergence(m)=eigen_pwl(m,k)·s
where Fdivergence(m) is a piecewise linear function in m where when m≤k0, the slope is s0, when k0<m≤k1, the slope is s1, etc.

Similarly, an example context metric Fcontextfor a measurement n can then be expressed as
Fcontext(n)=eigen_pwl(n,v)·z
where Fcontext(n) is a piecewise linear function in n where when n≤v0, the slope is z0, when v0<n≤v1, the slope is z1, etc.

An example matching feature then can be expressed as:
Fmatching(m,n)=Fdivergence(m)·Fcontext(n)=d·[eigen_pwl(n,v)⊗(eigen_pwl(m,k)]
where a set of weights d can be factorized as
d=z⊗s.

For example, z and s can be individually learned and selected based on engineered logic or physical constraints or principles. Alternatively, values of d can be learned directly (e.g., corresponding to weights of machine-learned calibration model418), subject to various constraints (e.g., constraints on a change of sign to regularize the learned model). For example, the effect of the context and the divergence can be initialized with unit-valued functions, and machine-learned calibration model418can calibrate the magnitudes of the slopes to dial in the result. In this manner, for instance, the resulting function Fmatching(m, n) can be linear in all its parameters (e.g., the slopes) and can be jointly piecewise linear in the inputs (e.g., m and n).

In some scenarios, a highly regularized, constrained model can provide improved performance with high interpretability and low risk of overfitting, thereby enabling strong out-of-domain performance. In some implementations, more complex machine-learned models can be used.

For instance,FIG.9is a block diagram illustrating an example implementation in which a neural network900can process input data based on divergence metrics414and context metrics416to generate a score910. Although a fully connected network is illustrated, various different architectures can be used, such as transformer-based architectures, CNNs, RNN, LSTM, feedforward networks, etc. The network can be linear or nonlinear.

The neural network can include a number of weights that can outnumber the quantity of divergence metrics414and context metrics416. The neural network can be small. The neural network can have only a small number of layers, such as one, two, or three layers, although more layers can be used (e.g., less than 10, less than 20, etc.). A single-layer linear neural network can effectively represent a linear weighted combination as described above.

An input dimension of the neural network can match a quantity of divergence metrics414. A number of channels of an input layer can include a channel for divergence values and a channel for context values. Divergence values and context values can be concatenated and processed in one channel.

To train machine-learned calibration model418(e.g., weights802-1,802-2, . . . ,802-P), evaluation system400can leverage a number of unit tests. A unit test can include a pair of object detections that are confirmed to not have a material divergence or a pair of object detections that are confirmed to have material divergence. For I positive unit tests (a positive match) and J negative unit tests (no match), learning weights702-1,702-2, . . . ,702-P can include optimizing weights702-1,702-2, . . . ,702-P such that

maxi∈{1,…,I}[wT⁢xi]<minj∈{1,…,J}[wT⁢xj]
where xirepresents the set of matching features for the i-th member of the set of I positive unit tests and where xjrepresents the set of matching features for the j-th member of the set of J negative unit tests. In other words, for an example optimization, all the computed scores for all the positive unit tests should be less than all the computed scores for all the negative unit tests, since all the negative unit tests by definition have more material divergences. The system can learn weights702-1,702-2, . . . ,702-P with additional objectives, such as finding the smallest set of such weights that satisfy the above criterion.

The output can be reshaped using various scaling and transformations to obtain a score mapped to a desired range (e.g., [0, 1]) having a desired threshold. For example, the output can be rescaled using

11+ewT⁢x-θ.

FIG.10is a flow chart of a process for updating evaluation system400using a set of unit tests1000, according to some aspects of the present disclosure. Unit tests1000can include one or more unit tests1002-1,1002-2, . . . ,1002-n.

An example unit test1002-ican include a recorded ego vehicle position1004-i, a detected object box1006-i, and a labeled object box1008-i. Unit test1002-ican be associated with a ground truth evaluation state1010-ithat records a validation or evaluation state of the match between detected object box1006-iand labeled object box1008-i. For instance, unit test1002-ican be associated with a value “Detection”: True that indicates that detected object box1006-iis valid and matches labeled object box1008-i.

An example unit test1002-jcan include a recorded ego vehicle position1004-j, a detected object box1006-j, and a labeled object box1008-j. Unit test1002-jcan be associated with a ground truth evaluation state1010-jthat records a validation or evaluation state of the match between detected object box1006-jand labeled object box1008-j. For instance, unit test1002-jcan be associated with a value “Detection”: False that indicates that detected object box1006-jis a valid detection but does not sufficiently match the labeled object box1008-j.

An example unit test1002-kcan include a recorded ego vehicle position1004-k, a detected object box1006-k, and a labeled object box1008-k. Unit test1002-kcan be associated with a ground truth evaluation state1010-kthat records a validation or evaluation state of the match between detected object box1006-kand labeled object box1008-k. For instance, unit test1002-kcan be associated with a value “Detection”: None that indicates that detected object box1006-kis not a valid detection with respect to labeled object box1008-k(e.g., the error is so great as to be spurious).

Evaluation system400can process one or more unit tests to determine whether evaluation system400correctly identifies the detection state. For instance, evaluation system400can include one or more adjustable thresholds. A first threshold can be set such that scores above the threshold correspond to “Detection”: True. The first threshold can be set such that scores below the threshold correspond to “Detection”: False. A second threshold can be set such that scores below the threshold correspond to “Detection”: None.

Evaluation system400can process all unit tests1000. A system can update/optimize (iteratively/numerically or analytically) parameters of evaluation system400(e.g., weights, thresholds, etc.) such that all unit tests pass. Passing a unit test can include generating an evaluation state that aligns with the stored evaluation state associated with the unit test. For instance, each unit test can represent a confirmed judgment (e.g., a human judgment) that a given object detection matches a label, does not match a label, or is spurious. Evaluation system400can pass a unit test if it correctly identifies the evaluation state for the unit test. If one or more unit tests fail, then the behavior of evaluation system400may be deviating from expectations.

In some situations, a failed unit test indicates a suboptimal selection of learnable parameters of evaluation system400, and further training can produce a set of parameters that cause evaluation system400to pass all unit tests. In some cases, however, the expressive power of evaluation system400is too constrained to satisfy all unit tests. In such cases, for example, additional terms, degrees of freedom, parameters, etc. can be added to evaluation system400to enable evaluation system400to fully model all unit tests.

In a simplified example, for instance, if a unit test required that evaluation system400severely penalize any error in detections of red objects—and evaluation system400did not have any context metric associated with object color—then there may be no set of optimal parameters which would cause evaluation system400to pass all unit tests, so long as evaluation system400is unable to recognize object color and penalize errors accordingly. Resolving the impasse can include adding a context metric that weights one or more errors based on a detected object color. In this manner, for instance, if evaluation system400is unable to satisfy all unit tests, then additional features can be added (e.g., additional divergence metrics, additional context metrics) to increase an expressive power of evaluation system400.

In an example, using three levels of values of Detection can enable evaluation system400to both evaluate a quality of object detections and prune spurious detections using a single framework (e.g., using the score(s) computed using divergence metrics414and context metrics416).

For example, a perception system can use an instance of evaluation system400to determine whether a new detection input matches an existing object track (or a forecast therefrom). Based on a score output from evaluation system400, the perception system can assign a new detection to an existing track (with or without errors, which can be handled separately) or can discard the new detection as spurious. If a detection is assigned to a track with errors (e.g., “Detection”: False), then the perception system can initiate recovery methods to improve the alignment of the detection with the track, either by updating the object track to reflect the new world state as recorded in the new detection or by cross-checking and confirming the new detection with additional/backup sensors or other processing algorithms (e.g., different filters, etc.).

In this manner, for instance, evaluation system400can be used online or offline. For instance, evaluation system400can be used online to evaluate a quality of a new detection of a perception system. Evaluation system400can be used offline to evaluate a benchmark performance of a new perception system over a set of benchmark scenes to evaluate the new perception system for performance advancement, regression, or minimum performance baselines.

FIG.11is an illustration of an example interface of a user input system1100that human operators can use to input labeled object data408(e.g., object bounding boxes, anchors, etc.). An interface1102can present a rendering of log data that can be “replayed”—that is, log data at various time steps can be presented in sequence (e.g., controlled by playback controls1104) to facilitate review of scenes of an environment.

A user can interact with the interface1102to draw a labeled box, such as a box1108around an object. Drawing a box can include tracing a path across an input surface (e.g., touch-sensitive input surface, using a cursor, etc.). Drawing a box can include selecting coordinates at which to anchor vertices of the box. A user can interact with interface1102to designate time intervals within which the annotations are valid. For example, interface1102can receive inputs that associate points on a timeline with beginning and ending times of a time interval (e.g., inputs selecting positions1110and1112on a timeline element).

User input system1100can facilitate review of pairs of labeled boxes and detected boxes for labeling the pairs as positive matches or negative matches. For example, interface1102can render two boxes: one labeled box and one box generated by SUT406. Interface1102can display an input element that, when selected, causes user input system1100to store a label indicating that the boxes do not materially diverge. Interface1102can display an input element that, when selected, causes user input system1100to store a label indicating that the boxes do materially diverge.

User input system1100can facilitate review of object detection data410in different reference scenarios. For example, user input system1100can cause input interface1102to display object detection data410(e.g., boxes) that are ranked or filtered according to one or more criteria. For instance, a value of an individual divergence metric or context metric can be used for ranking or filtering. For example, user input system1100can receive an input requesting a listing of object detection data410(e.g., boxes) that have the highest divergence value in a particular metric (e.g., position, forecasted position, etc.). In this manner, for instance, the structure of the evaluation system400itself into interpretable divergence metrics can facilitate more granular interrogation of how SUT406is diverging to reveal potential underlying causes of such divergences.

FIG.12is a flowchart of a method1200for evaluating a system under test (e.g., a perception system) according to aspects of the present disclosure. One or more portions of example method1200can be implemented by the computing systems described with reference to the other figures (e.g., autonomous platform112, vehicle computing system180, remote system160, a system ofFIGS.1to15, etc.). Each respective portion of example method1200can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portions of example method1200can be implemented on the hardware components of the devices described herein (e.g., as inFIGS.1to15, etc.).

FIG.12depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure.FIG.12is described with reference to elements/terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method1200can be performed additionally, or alternatively, by other systems.

At1202, example method1200can include (a) obtaining an object detection from a perception system that describes an object in an environment of the autonomous vehicle. For instance, the object detection can include data describing an object's dimensions, position, orientation, heading, type, etc. The object detection can be part of an object track that collects detection observations over time (e.g., over one or more execution cycles of the perception system) to generate a trace of object behavior over time. In an example, the object detection can include a bounding box predicted for at least a portion of the object.

At1204, example method1200can include (b) obtaining, from a reference dataset, a label that describes a reference position of the object in the environment. For instance, the label can include a ground truth bounding box for the object. For example, a sensor data capture describing the environment can be annotated using an annotation system that generates ground truth annotations of objects in the environment (e.g., using user inputs to record, e.g., bounding boxes around objects). The label can include data that has been determined (e.g., manually, automatically) to indicate one or more attributes of the object in the environment.

At1206, example method1200can include (c) determining a plurality of component divergence values respectively for a plurality of divergence metrics. For example, a divergence metric can compute a divergence value based on one or more input features (e.g., data obtained from the perception system). The input features can include the object detection data, the label data, or other context data obtained from the perception system. In an example, a respective divergence value characterizes a respective difference between the object detection and the label. In some implementations of example method1200, the plurality of divergence metrics are evaluated between a labeled bounding box and a detected bounding box from the object detection.

At1208, example method1200can include (d) providing the plurality of component divergence values to a machine-learned model to generate a score that indicates an aggregate divergence between the object track and the label. The machine-learned model can include a plurality of learned parameters defining an influence of the plurality of component divergence values on the score. For instance, the aggregate divergence can be an overall measure of the extent to which the object detection is meaningfully different from the label. The machine-learned model can be a machine-learned calibration model418that calibrates the relative importance of measured differences between the label and the object detection.

The aggregate divergence can aggregate over the component divergence values as individual signals. The individual signals of the component divergence values can be weighted using the machine-learned model. The weighting can be explicit (e.g., with a weight associated with a particular signal for combining in a weighted sum). In some implementations of example method1200, the plurality of learned parameters respectively correspond to the plurality of divergence metrics. The weighting can be implicit (e.g., with weights of a neural network operating over multiple inputs to generate an aggregate divergence that reflects a learned prioritization over different signals). In some implementations of example method1200, the plurality of learned parameters respectively correspond to the plurality of divergence metrics.

In some implementations of example method1200, the score includes a weighted combination of the plurality of component divergence values, wherein the plurality of learned parameters are used to perform the weighting in the weighted combination.

In some implementations of example method1200, the weighted combination is a linear combination.

In some implementations of example method1200, the score is generated using a piecewise function that is linear with respect to each component divergence metric.

In some implementations of example method1200, the piecewise function includes learnable parameters that are constrained from flipping a direction of a contribution of a component divergence metric to the score.

In some implementations of example method1200, the score is generated using a piecewise function that is linear with respect to each component divergence metric. In some implementations of example method1200the piecewise function includes one or more segment slopes and one or more segment intercepts include learnable parameters that are obtained using the plurality of unit tests. In some implementations of example method1200, the piecewise function can be expressed using a tensor product of a vector of basis functions of context and a vector of component divergence values.

At1210, example method1200can include (c) evaluating a quality of a match between the object track and the label based on the score. For example, a quality of the match can correspond to whether the generated object detection is close enough to the labeled object data to facilitate adequate operations of downstream components from the perception system (e.g., motion planning systems). A quality measure can have continuous or discrete values.

In some implementations, example method1200includes assigning an evaluation state to a component of the perception system based on the score. For instance, an evaluation state can be indicated by a numerical score. An evaluation state can be indicated by a Boolean value (e.g., True, False).

An evaluation state can be indicated by a hierarchy of flags. For instance, a first flag can indicate a good match. A second flag can indicate a bad match. A third flag can indicate a failure to generate any valid match. These flags can be generated based on a comparison of a raw score to one or more thresholds.

In some implementations, the evaluation state is a validation state that indicates that the perception system has achieved a benchmark level of performance.

In some implementations, example method1200includes generating, using the machine-learned model, a respective score for a respective object detection of a plurality of object detections generated by the perception system to determine a quality of a match with a respective label corresponding to the respective object track. For example, the plurality of object detections can be generated by processing a set of unit tests.

In some implementations of example method1200, assigning a validation state to the component of the perception system based on the score includes determining a proportion of matches for the plurality of object detections that satisfy a threshold quality. For instance, across a set of unit tests, a pass rate can be computed by determining a proportion of tests for which the perception system generated object detections that sufficiently aligned with the corresponding labels.

In some implementations of example method1200, assigning a validation state to the component of the perception system based on the score includes comparing the proportion to a target threshold proportion. The target proportion can be a fraction of the unit tests up to and including an entirety of the unit tests.

In some implementations of example method1200, assigning a validation state to the component of the perception system based on the score includes assigning the validation state based on the comparison. For instance, based on achieving a target level of performance, a validation state of “Validated” can be assigned to the perception system.

In some implementations, example method1200includes, prior to (c), determining the plurality of learned parameters by fitting a linear model to a plurality of unit tests, wherein a respective unit test includes an example object detection and a unit test label indicating reference data for the object. In some implementations of example method1200, the respective unit test is stored in association with a ground truth designation indicating whether there is a material divergence between the example object track and the unit test label.

For example, instead of using unit tests to validate a performance of the perception system, a set of unit tests with known evaluation states (e.g., an object detection that is known to match a label, known not to match a label, etc.) can be used to evaluate the evaluation system's ability to accurately recognize and determine the evaluation state for a particular unit test.

The evaluation system can be updated to align with the decision boundary reflected in the unit tests. For instance, the set of unit tests can trace a decision boundary around what types of differences matter in different contexts. For instance, one unit test can include a particular divergence between an object detection and a label in a first scene. In this unit test, the aggregate divergence can be material. Another unit test can include the same or a similar divergence in a second, different scene. In this unit test, the divergence can be immaterial. The evaluation system can learn to distinguish between such unit tests.

In some implementations of example method1200, the object detection and the label are obtained from a unit test associated with a positive match. In some implementations of example method1200, (e) includes determining that the score indicates a negative match based on a failure of the score to satisfy a threshold. In some implementations of example method1200, (e) includes updating one or more of the plurality of learned parameters to cause the score to satisfy the threshold.

In some implementations, example method1200includes weighting a respective contribution of the respective component divergence value using a context value obtained using a context metric, wherein the context value is based on an attribute of the object. The attribute of the object can be obtained from the object detection. The attribute of the object can be obtained from the label data.

An attribute of the object can include a relationship between the object and the environment. For instance, a relationship between the object and the environment can include a distance from another object in the environment. A relationship between the object and the environment can include a distance from an ego vehicle in the environment (e.g., a vehicle associated with generating the object detection, such as a vehicle operating the perception system). A relationship between the object and the environment can include a lane position.

An attribute of the object can include a type of the object. For example, a type of the object can include an object classification output from an object classifier. Example object types can include vehicles, infrastructure elements, pedestrians, etc.

In some implementations, example method1200includes determining, using the context metric and based on an attribute of the object detection or the label, a context domain for the respective component divergence value. In some implementations, example method1200includes weighting the respective contribution of the respective component divergence value based on a weighting parameter associated with the context domain. For example, context domains can correspond to ranges of a context parameter. A context metric can be a piecewise function over the ranges of the context parameter. In each range, the context metric can apply a different computation to generate a context value (e.g., a weighting parameter) for weighting a corresponding divergence value. Example context domains are illustrated in tree structure700.

In some implementations of example method1200, the plurality of divergence metrics include at least one of the following divergence metrics: a three-dimensional intersection over union of the labeled bounding box and the detected bounding box; an intersection over union of a projection of the labeled bounding box into a range view and a projection of the detected bounding box into the range view; a difference in volume between the labeled bounding box and the detected bounding box; a difference between: a detected distance between the labeled bounding box and a position associated with the autonomous vehicle, and a detected distance between the detected bounding box and the position associated with the autonomous vehicle; a difference between: a predicted distance between an expected position of the labeled bounding box and a position associated with the autonomous vehicle, and a predicted distance between an expected position of the detected bounding box and the position associated with the autonomous vehicle; a difference in a detected forward velocity associated with the object and a labeled forward velocity associated with the object; or a difference in a detected heading associated with the object and a labeled heading associated with the object.

In some implementations of example method1200, the plurality of divergence metrics can indicate future-time divergences. An example future-time divergence can include a difference between a predicted distance between an expected position of the labeled bounding box and a position associated with the autonomous vehicle. In some implementations of example method1200, the plurality of divergence metrics include a difference between a predicted distance between an expected position of the detected bounding box and the position associated with the autonomous vehicle. In some implementations of example method1200, a respective contribution of a respective component divergence value using the difference is weighted based on a time horizon for which the predicted distances are obtained.

In some implementations of example method1200, a score satisfying the first threshold indicates that the perception system satisfactorily tracked the object (e.g., “Detection”: True). In some implementations of example method1200, a score satisfying the second threshold but not the first threshold indicates that the perception system suboptimally tracked the object (e.g., “Detection”: False).

In some implementations, example method1200includes training, using a plurality of example matches having scores that satisfy the second threshold but not the first threshold, the perception system to improve a tracking performance. In some implementations, example method1200includes training, using a plurality of example matches having scores that do not satisfy the second threshold or the first threshold (e.g., “Detection”: None), the perception system to discard invalid tracks.

FIG.13is a flowchart of an example method1300for updating a machine-learned component of an evaluation system according to aspects of the present disclosure. One or more portions of example method1300can be implemented by the computing systems described with reference to the other figures (e.g., autonomous platform110, vehicle computing system180, remote system160, a system ofFIGS.1to15, etc.). Each respective portion of example method1300can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portions of example method1300can be implemented on the hardware components of the devices described herein (e.g., as inFIGS.1to15, etc.).

FIG.13depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure.FIG.13is described with reference to elements/terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method1300can be performed additionally, or alternatively, by other systems.

At1302, example method1300can include determining a first evaluation state for at least one test detection with respect to at least one corresponding reference detection. For example, evaluation system400can generate, for unit test1002-i, the first evaluation state comparing detected object box1006-iand labeled object box1008-i.

At1304, example method1300can include providing the first evaluation state for review. For example, the first evaluation state can be compared against a ground truth evaluation state. For instance, unit test1002-ican include a ground truth evaluation state1010-iagainst which the first evaluation state can be compared.

At1306, example method1300can include receiving a corrective signal assigning a second, different evaluation state to the at least one test detection. For instance, if the first evaluation state disagrees with the ground truth evaluation state, a corrective signal can include an indication of the disagreement (e.g., a cost, a loss, a boolean failure signal, a penalty, etc.).

For example, a first evaluation state can be a false positive or a false negative. For instance, a false positive evaluation state can indicate an absence of a material divergence (e.g., an indicated “match”). A false positive evaluation state can correspond to an indicated match when the detection does in fact materially diverge from the label. A false negative evaluation state can indicate the presence of a material divergence (e.g., an indicated failure to “match”). A false negative evaluation state can correspond to an indicated material divergence when the detection does not in fact materially diverge from the label.

At1308, example method1300can include updating parameters of the machine-learned model to refine the decision boundary based on the at least one test detection and the corresponding at least one reference detection.

For example, learnable parameters of evaluation system400(e.g., of machine-learned model418) can be re-learned until satisfactorily evaluating the at least one test detection and the corresponding at least one reference detection. For example, numerical optimization of the parameters can search over a parameter space to return a set of parameters that correctly evaluate all unit tests as well as evaluating the at least one test detection and the corresponding at least one reference detection.

In some implementations, the at least one test detection and the corresponding at least one reference detection can be used to form a new unit test. For example, a batch of new detections can be processed using evaluation system400. A labeling system can return ground truth labels associated with the real or simulated sensor data that was processed to generate the new detections. Evaluation system400can compare the new detections to the ground truth labels. The outputs of evaluation system400can be reviewed in whole or in part. For instance, detection failures can be reviewed to evaluate whether the detections are appropriately classified as failures. Detection successes can be reviewed to evaluate whether the detections are appropriately classified as successful detections.

This review can reveal false positive or false negative evaluations. To improve the performance of evaluation system400, the underlying detections, labels, and ground truth evaluation states for these false positive or false negative evaluations can form new unit tests. Learnable parameters of evaluation system400can be re-learned based on the updated set of unit tests.

If machine-learned calibration model418does not or cannot converge to a set of weights that enables correct labeling of all unit tests (e.g., including the updated set of unit tests), then evaluation system400can add additional expressivity to more fully model the task. For example, evaluation system400can compute additional divergence metrics. Evaluation system400can use additional context metrics (or more expressive or granular versions of existing metrics). Evaluation system400can add additional learnable parameters to machine-learned model418. Increasing the expressivity of evaluation system400can increase a precision with which evaluation system400can model a desired decision boundary between object detections that “match” or are aligned closely enough and object detections that do not “match” or are not aligned closely enough.

FIG.14is a flowchart of a method1400for training one or more machine-learned operational models, according to aspects of the present disclosure.

One or more portions of example method1400can be implemented by a computing system that includes one or more computing devices such as, for example, the computing systems described with reference to the other figures (e.g., autonomous platform110, vehicle computing system180, remote system160, a system ofFIGS.1to15, etc.). Each respective portion of example method1400can be performed by any (or any combination) of one or more computing devices. Moreover, one or more portions of example method1400can be implemented on the hardware components of the devices described herein (e.g., as inFIGS.1to15, etc.), for example, to validate one or more systems or models.

FIG.14depicts elements performed in a particular order for purposes of illustration and discussion. Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the methods discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure.FIG.14is described with reference to elements/terms described with respect to other systems and figures for exemplary illustrated purposes and is not meant to be limiting. One or more portions of example method1400can be performed additionally, or alternatively, by other systems.

At1402, example method1400can include obtaining training data for training a machine-learned operational model. The training data can include a plurality of training instances.

The training data can be collected using one or more autonomous platforms (e.g., autonomous platform110) or the sensors thereof as the autonomous platform is within its environment. By way of example, the training data can be collected using one or more autonomous vehicles (e.g., autonomous platform110, autonomous vehicle350, etc.) or sensors thereof as the vehicle operates along one or more travel ways. In some examples, the training data can be collected using other sensors, such as mobile-device-based sensors, ground-based sensors, aerial-based sensors, satellite-based sensors, or substantially any sensor interface configured for obtaining and/or recording measured data.

The training data can include a plurality of training sequences divided between multiple datasets (e.g., a training dataset, a validation dataset, or testing dataset). Each training sequence can include a plurality of pre-recorded perception datapoints, point clouds, images, etc. In some implementations, each sequence can include LIDAR point clouds (e.g., collected using LIDAR sensors of an autonomous platform), images (e.g., collected using mono or stereo imaging sensors, etc.), and the like. For instance, in some implementations, a plurality of images can be scaled for training and evaluation.

At1404, example method1400can include selecting a training instance based at least in part on the training data.

At1406, example method1400can include inputting the training instance into the machine-learned operational model.

At1408, example method1400can include generating one or more loss metrics and/or one or more objectives for the machine-learned operational model based on outputs of at least a portion of the machine-learned operational model and labels associated with the training instances.

At1410, example method1400can include modifying at least one parameter of at least a portion of the machine-learned operational model based at least in part on at least one of the loss metrics and/or at least one of the objectives. For example, a computing system can modify at least a portion of the machine-learned operational model based at least in part on at least one of the loss metrics and/or at least one of the objectives.

In some implementations, the machine-learned operational model can be trained in an end-to-end manner. For example, in some implementations, the machine-learned operational model can be fully differentiable.

After being updated, the operational model or the operational system including the operational model can be provided for validation (e.g., according to example implementations of example method1200, etc.). In some implementations, a validation system can evaluate or validate the operational system. The validation system can trigger retraining, decommissioning, etc. of the operational system based on, for example, failure to satisfy a validation threshold in one or more areas.

FIG.15is a block diagram of an example computing ecosystem10according to example implementations of the present disclosure. The example computing ecosystem10can include a first computing system20and a second computing system40that are communicatively coupled over one or more networks60. In some implementations, the first computing system20or the second computing40can implement one or more of the systems, operations, or functionalities described herein for validating one or more systems or operational systems (e.g., the remote system160, the onboard computing system180, the autonomy system200, etc.).

In some implementations, the first computing system20can be included in an autonomous platform and be utilized to perform the functions of an autonomous platform as described herein. For example, the first computing system20can be located onboard an autonomous vehicle and implement autonomy system for autonomously operating the autonomous vehicle. In some implementations, the first computing system20can represent the entire onboard computing system or a portion thereof (e.g., the localization system230, the perception system240, the planning system250, the control system260, or a combination thereof, etc.). In other implementations, the first computing system20may not be located onboard an autonomous platform. The first computing system20can include one or more distinct physical computing devices21.

The first computing system20(e.g., the computing devices21thereof) can include one or more processors22and a memory23. The one or more processors22can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. Memory23can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

Memory23can store information that can be accessed by the one or more processors22. For instance, the memory23(e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) can store data24that can be obtained (e.g., received, accessed, written, manipulated, created, generated, stored, pulled, downloaded, etc.). The data24can include, for instance, sensor data, map data, data associated with autonomy functions (e.g., data associated with the perception, planning, or control functions), simulation data, or any data or information described herein. In some implementations, the first computing system20can obtain data from one or more memory devices that are remote from the first computing system20.

Memory23can store computer-readable instructions25that can be executed by the one or more processors22. Instructions25can be software written in any suitable programming language or can be implemented in hardware. Additionally, or alternatively, instructions25can be executed in logically or virtually separate threads on the processors22.

For example, the memory23can store instructions25that are executable by one or more processors (e.g., by the one or more processors22, by one or more other processors, etc.) to perform (e.g., with the computing devices21, the first computing system20, or other systems having processors executing the instructions) any of the operations, functions, or methods/processes (or portions thereof) described herein. For example, operations can include implementing system validation (e.g., as described herein).

In some implementations, the first computing system20can store or include one or more models26. In some implementations, the models26can be or can otherwise include one or more machine-learned models (e.g., a machine-learned operational system, etc.). As examples, the models26can be or can otherwise include various machine-learned models such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbors models, Bayesian networks, or other types of models including linear models or non-linear models. Example neural networks include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the first computing system20can include one or more models for implementing subsystems of the autonomy system200, including any of: the localization system230, the perception system240, the planning system250, or the control system260.

In some implementations, the first computing system20can obtain the one or more models26using communication interface27to communicate with the second computing system40over the network60. For instance, the first computing system20can store the models26(e.g., one or more machine-learned models) in memory23. The first computing system20can then use or otherwise implement the models26(e.g., by the processors22). By way of example, the first computing system20can implement the models26to localize an autonomous platform in an environment, perceive an autonomous platform's environment or objects therein, plan one or more future states of an autonomous platform for moving through an environment, control an autonomous platform for interacting with an environment, etc.

The second computing system40can include one or more computing devices41. The second computing system40can include one or more processors42and a memory43. The one or more processors42can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, a FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory43can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, one or more memory devices, flash memory devices, etc., and combinations thereof.

Memory43can store information that can be accessed by the one or more processors42. For instance, the memory43(e.g., one or more non-transitory computer-readable storage media, memory devices, etc.) can store data44that can be obtained. The data44can include, for instance, sensor data, model parameters, map data, simulation data, simulated environmental scenes, simulated sensor data, data associated with vehicle trips/services, or any data or information described herein. In some implementations, the second computing system40can obtain data from one or more memory devices that are remote from the second computing system40.

Memory43can also store computer-readable instructions45that can be executed by the one or more processors42. The instructions45can be software written in any suitable programming language or can be implemented in hardware. Additionally, or alternatively, the instructions45can be executed in logically or virtually separate threads on the processors42.

For example, memory43can store instructions45that are executable (e.g., by the one or more processors42, by the one or more processors22, by one or more other processors, etc.) to perform (e.g., with the computing devices41, the second computing system40, or other systems having processors for executing the instructions, such as computing devices21or the first computing system20) any of the operations, functions, or methods/processes described herein. This can include, for example, the functionality of the autonomy system200(e.g., localization, perception, planning, control, etc.) or other functionality associated with an autonomous platform (e.g., remote assistance, mapping, fleet management, trip/service assignment and matching, etc.). This can also include, for example, validating a machined-learned operational system.

In some implementations, second computing system40can include one or more server computing devices. In the event that the second computing system40includes multiple server computing devices, such server computing devices can operate according to various computing architectures, including, for example, sequential computing architectures, parallel computing architectures, or some combination thereof.

Additionally, or alternatively to, the models26at the first computing system20, the second computing system40can include one or more models46. As examples, the models46can be or can otherwise include various machine-learned models (e.g., a machine-learned operational system, etc.) such as, for example, regression networks, generative adversarial networks, neural networks (e.g., deep neural networks), support vector machines, decision trees, ensemble models, k-nearest neighbors models, Bayesian networks, or other types of models including linear models or non-linear models. Example neural networks include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. For example, the second computing system40can include one or more models of the autonomy system200.

In some implementations, the second computing system40or the first computing system20can train one or more machine-learned models of the models26or the models46through the use of one or more model trainers47and training data48. The model trainer47can train any one of the models26or the models46using one or more training or learning algorithms. One example training technique is backwards propagation of errors. In some implementations, the model trainer47can perform supervised training techniques using labeled training data. In other implementations, the model trainer47can perform unsupervised training techniques using unlabeled training data. In some implementations, the training data48can include simulated training data (e.g., training data obtained from simulated scenarios, inputs, configurations, environments, etc.). In some implementations, the second computing system40can implement simulations for obtaining the training data48or for implementing the model trainer47for training or testing the models26or the models46. By way of example, the model trainer47can train one or more components of a machine-learned model for the autonomy system200through unsupervised training techniques using an objective function (e.g., costs, rewards, heuristics, constraints, etc.). In some implementations, the model trainer47can perform a number of generalization techniques to improve the generalization capability of the models being trained. Generalization techniques include weight decays, dropouts, or other techniques.

For example, in some implementations, the second computing system40can generate training data48according to example aspects of the present disclosure. For instance, the second computing system40can generate training data48. For instance, the second computing system40can implement methods according to example aspects of the present disclosure. The second computing system40can use the training data48to train models26. For example, in some implementations, the first computing system20can include a computing system onboard or otherwise associated with a real or simulated autonomous vehicle. In some implementations, models26can include perception or machine vision models configured for deployment onboard or in service of a real or simulated autonomous vehicle. In this manner, for instance, the second computing system40can provide a training pipeline for training models26.

The first computing system20and the second computing system40can each include communication interfaces27and49, respectively. The communication interfaces27,49can be used to communicate with each other or one or more other systems or devices, including systems or devices that are remotely located from the first computing system20or the second computing system40. The communication interfaces27,49can include any circuits, components, software, etc. for communicating with one or more networks (e.g., the network60). In some implementations, the communication interfaces27,49can include, for example, one or more of a communications controller, receiver, transceiver, transmitter, port, conductors, software or hardware for communicating data.

The network60can be any type of network or combination of networks that allows for communication between devices. In some implementations, the network can include one or more of a local area network, wide area network, the Internet, secure network, cellular network, mesh network, peer-to-peer communication link or some combination thereof and can include any number of wired or wireless links. Communication over the network60can be accomplished, for instance, through a network interface using any type of protocol, protection scheme, encoding, format, packaging, etc.

FIG.15illustrates one example computing ecosystem10that can be used to implement the present disclosure. For example one or more systems or devices of ecosystem10can implement any one or more of the systems and components described in the preceding figures. Other systems can be used as well. For example, in some implementations, the first computing system20can include the model trainer47and the training data48. In such implementations, the models26,46can be both trained and used locally at the first computing system20. As another example, in some implementations, the computing system20may not be connected to other computing systems. Additionally, components illustrated or discussed as being included in one of the computing systems20or40can instead be included in another one of the computing systems20or40.

Computing tasks discussed herein as being performed at computing devices remote from the autonomous platform (e.g., autonomous vehicle) can instead be performed at the autonomous platform (e.g., via a vehicle computing system of the autonomous vehicle), or vice versa. Such configurations can be implemented without deviating from the scope of the present disclosure. The use of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. Computer-implemented operations can be performed on a single component or across multiple components. Computer-implemented tasks or operations can be performed sequentially or in parallel. Data and instructions can be stored in a single memory device or across multiple memory devices.

Aspects of the disclosure have been described in terms of illustrative implementations thereof. Numerous other implementations, modifications, or variations within the scope and spirit of the appended claims can occur to persons of ordinary skill in the art from a review of this disclosure. Any and all features in the following claims can be combined or rearranged in any way possible. Accordingly, the scope of the present disclosure is by way of example rather than by way of limitation, and the subject disclosure does not preclude inclusion of such modifications, variations or additions to the present subject matter as would be readily apparent to one of ordinary skill in the art. Moreover, terms are described herein using lists of example elements joined by conjunctions such as “and,” “or,” “but,” etc. It should be understood that such conjunctions are provided for explanatory purposes only. Lists joined by a particular conjunction such as “or,” for example, can refer to “at least one of” or “any combination of” example elements listed therein, with “or” being understood as “and/or” unless otherwise indicated. Also, terms such as “based on” should be understood as “based at least in part on.”

Those of ordinary skill in the art, using the disclosures provided herein, will understand that the elements of any of the claims, operations, or processes discussed herein can be adapted, rearranged, expanded, omitted, combined, or modified in various ways without deviating from the scope of the present disclosure. Some of the claims are described with a letter reference to a claim element for exemplary illustrated purposes and is not meant to be limiting. The letter references do not imply a particular order of operations. For instance, letter identifiers such as (a), (b), (c), . . . , (i), (ii), (iii), . . . , etc. can be used to illustrate operations. Such identifiers are provided for the case of the reader and do not denote a particular order of steps or operations. An operation illustrated by a list identifier of (a), (i), etc. can be performed before, after, or in parallel with another operation illustrated by a list identifier of (b), (ii), etc.

The term “can” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X can perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.

The term “may” should be understood as referring to a possibility of a feature in various implementations and not as prescribing an ability that is necessarily present in every implementation. For example, the phrase “X may perform Y” should be understood as indicating that, in various implementations, X has the potential to be configured to perform Y, and not as indicating that in every instance X must always be able to perform Y. It should be understood that, in various implementations, X might be unable to perform Y and remain within the scope of the present disclosure.