Patent ID: 12233918

In the drawings, like reference numbers generally indicate identical or similar elements. Additionally, generally, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears.

DETAILED DESCRIPTION

Provided herein are system, apparatus, device, method, and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for determining objects that are kinematically capable, even if non-compliant with rules-of-the-road, of affecting a trajectory of a vehicle. According to some aspects, the perception system (e.g., computational system, inference system, etc.) of an autonomous vehicle (AV) and/or the like may determine if an object and/or road actor, such as a vehicle, a person, an animal, and/or any item (e.g., a static item, a moving item, etc.) is both detectable and has a kinematically feasible trajectory (even if non-compliant with the standard rules of the road) that allows it to reach the AV's route. For example, the perception system may receive data/information regarding objects and/or actors within a field of view of one or more sensors (e.g., Light Detection and Ranging (lidar) sensors, ultrasonic sensors, depth-sensing devices, Radio Detection and Ranging (RADAR) devices, cameras, etc.) associated with the AV, and may use a detectability metric to determine, based on the amount (e.g., percentage, etc.) of each object and/or actor that is occluded, whether the object and/or actor is detectable. According to some aspects, the perception system, for objects and/or actors determined to be detectable, may determine an approximation to the AV's trajectory (e.g., a likely and/or intended path of the AV, etc.) based on a current route and velocity of the AV.

The system, apparatus, device, method, and/or computer program product embodiments, and/or combinations and sub-combinations thereof, for determining perceptual relevancy of objects and/or road actors provide improvements over conventional systems (e.g., AV perception systems, etc.). For example, conventional AV perception systems are unable to determine tracks associated with objects and/or actors that could potentially collide with an AV without unnecessarily taxing the computational resources of the AV by outputting significantly more tracks than the tracks that will interact with the planned trajectory of the AV. Conventional AV perception systems are unable to filter datasets for perception verification in such a way that it accelerates the testing of the long tails (e.g., outlier circumstances that could affect the trajectory of an AV, etc.) and avoids instances of false positives (e.g., perception instances that may cause an AV to act on irrelevant objects, etc.) and false negatives (e.g., instances where the perception system misses objects and/or misclassify objects/actors that are critical for the planning and control systems, of an AV, etc.). As such, conventional AV perception systems routinely fail to identify tracks (e.g., tracking information for objects and/or actors, etc.) for which near-perfect perception performance is required to avoid collisions.

The system, apparatus, device, method, and/or computer program product embodiments, and/or combinations and sub-combinations thereof, enable detection of objects that are kinematically capable, even if non-compliant with rules-of-the-road, of affecting a trajectory of a vehicle by enabling datasets that indicate spatially relevant objects (SRO) that are perceived by the perception system of an AV to be generated. According to some aspects, a severity ranking may be assigned to the SROs to enable the perception system to accelerate the testing of long tails and provide data/information that may be used to avoid damaging and/or potentially life-threatening collisions between the AV and SROs. Accelerated generation of datasets that indicate SROs may be used to prioritize scarce computational resources of the on-board computer and/or the like of an AV, as well as selecting objects and road actors that should be included in machine learning (ML) training datasets (e.g., ML datasets used to improve, optimize, and/or access failures of a perception pipeline used in the production of AVs, etc.). These and other technological advantages are described herein.

As used herein, the term “vehicle” refers to any moving form of conveyance that is capable of carrying either one or more human occupants and/or cargo and is powered by any form of energy. The term “vehicle” includes, but is not limited to, cars, trucks, vans, trains, autonomous vehicles, aircraft, aerial drones, and/or the like. An “autonomous vehicle” (or “AV”) is a vehicle having a processor, programming instructions, and drivetrain components that are controllable by the processor without requiring a human operator. An autonomous vehicle may be fully autonomous in that it does not require a human operator for most or all driving conditions and functions, or it may be semi-autonomous in that a human operator may be required in certain conditions or for certain operations, or that a human operator may override the vehicle's autonomous system and may take control of the vehicle, or that a human operator may primarily drive the vehicle and an autonomous system may monitor the operations of the vehicle and take control of vehicle operations to avoid collisions.

Notably, the methods and systems for determining the perceptual relevancy of objects and/or road actors are being described herein in the context of an autonomous vehicle. However, the methods and systems are not limited to autonomous vehicle applications. The methods and systems described herein may be used in other applications such as robotic applications, radar system applications, metric applications, and/or system performance applications.

FIG.1illustrates an exemplary autonomous vehicle system100, in accordance with aspects of the disclosure. System100comprises a vehicle102athat is traveling along a road in a semi-autonomous or autonomous manner. Vehicle102ais also referred to herein as AV102a. AV102acan include, but is not limited to, a land vehicle (as shown inFIG.1), an aircraft, a watercraft, and/or the like.

AV102ais generally configured to detect objects102b,114,116in proximity thereto. The objects can include, but are not limited to, a vehicle102b, cyclist114(such as a rider of a bicycle, electric scooter, motorcycle, or the like) and/or a pedestrian116. According to some aspects, as described further herein, the AV102a(e.g., via on-board computing device113, etc.) may identify and/or determine whether the objects102b,114,116are spatially relevant to the AV102a. A spatially relevant road actor (e.g., object within proximity to the AV102a) may be defined as any road actor (RA) that could reach an intended trajectory and/or path the AV102ausing a kinematically feasible trajectory (without regard to road and/or traffic rule compliancy. The degree of spatial relevancy for road actors such as the objects102b,114,116may be classified and/or ranked, for example, according to the severity of a collision that could occur between the road actors and the AV102a.

As illustrated inFIG.1, the AV102amay include a sensor system111, an on-board computing device113, a communications interface117, and a user interface115. The AV102amay further include certain components (as illustrated, for example, inFIG.2) included in vehicles, which may be controlled by the on-board computing device113using a variety of communication signals and/or commands, such as, for example, acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.

The sensor system111may include one or more sensors that are coupled to and/or are included within the AV102a, as illustrated inFIG.2. For example, such sensors may include, without limitation, a lidar system, a radio detection and ranging (RADAR) system, a laser detection and ranging (LADAR) system, a sound navigation and ranging (SONAR) system, one or more cameras (e.g., visible spectrum cameras, infrared cameras, etc.), temperature sensors, position sensors (e.g., global positioning system (GPS), etc.), location sensors, fuel sensors, motion sensors (e.g., inertial measurement units (IMU), etc.), humidity sensors, occupancy sensors, or the like. The sensor data can include information that describes the location of objects within the surrounding environment of the AV102a, information about the environment itself, information about the motion of the AV102a, information about a route of the vehicle, or the like. As AV102atravels over a surface, at least some of the sensors may collect data pertaining to the surface.

As will be described in greater detail, AV102amay be configured with a lidar system, e.g., lidar system264ofFIG.2. The lidar system may be configured to transmit a light pulse104to detect objects located within a distance or range of distances of AV102a. Light pulse104may be incident on one or more objects (e.g., AV102b) and be reflected back to the lidar system. Reflected light pulse106incident on the lidar system may be processed to determine a distance of that object to AV102a. The reflected light pulse may be detected using, in some embodiments, a photodetector or array of photodetectors positioned and configured to receive the light reflected back into the lidar system. Lidar information, such as detected object data, is communicated from the lidar system to an on-board computing device, e.g., on-board computing device220ofFIG.2. The AV102amay also communicate lidar data to a remote computing device110(e.g., cloud processing system) over communications network108. Remote computing device110may be configured with one or more servers to process one or more processes of the technology described herein. Remote computing device110may also be configured to communicate data/instructions to/from AV102aover network108, to/from server(s) and/or database(s)112.

It should be noted that the lidar systems for collecting data pertaining to the surface may be included in systems other than the AV102asuch as, without limitation, other vehicles (autonomous or driven), robots, satellites, etc.

Network108may include one or more wired or wireless networks. For example, the network108may include a cellular network (e.g., a long-term evolution (LTE) network, a code division multiple access (CDMA) network, a 3G network, a 4G network, a 5G network, an XG network, any other type of next-generation network, etc.). The network may also include a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., the Public Switched Telephone Network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber optic-based network, a cloud computing network, and/or the like, and/or a combination of these or other types of networks.

AV102amay retrieve, receive, display, and edit information generated from a local application or delivered via network108from database112. Although only a single database112is shown, the database112may include any number of databases, data repositories, data lakes, third-party data sources, and/or the like. Database112may be configured to store and supply raw data, indexed data, structured data, map data, program instructions, or other configurations as are known. For example, the database112may provide remote computing device110with ground truth data/information, such as JavaScript Object Notation (JSON) files and/or the like that contain labels (e.g. road actor classification information) for road actors (e.g., objects102b,114,116, etc.), SE3 (e.g., proper rigid transformations in 3-dimensional Euclidean space) transformations to AV frame, velocity/speed information, bounding cuboids, and/or the like. The remote computing device110may provide the ground truth data/information to the AV102a(e.g., to the on-board computing device113via the communication interface117, etc.). The remote computing device110may provide the AV102avector maps (e.g., SQLite files, etc.) corresponding to the ground truth data/information that may be used to extract information about a drivable area, lane segments that belong route travels by the AV102a, lane segments speed, and/or any other traffic and/or driving area related information. The remote computing device110may provide the AV102awith parameters such as max long acceleration/deceleration, maximum centripetal accelerations, and/or minimal turning radii for the AV102aand/or road actors (e.g., objects102b,114,116, etc.).

The communications interface117may be configured to allow communication between AV102aand external systems, such as, for example, external devices, sensors, other vehicles, servers, data stores, databases, etc. The communications interface117may utilize any now or hereafter known protocols, protection schemes, encodings, formats, packaging, etc. such as, without limitation, Wi-Fi, an infrared link, Bluetooth, etc. The user interface system115may be part of peripheral devices implemented within the AV102aincluding, for example, a keyboard, a touch screen display device, a microphone, a speaker, etc.

FIG.2illustrates an example system architecture200for a vehicle, in accordance with aspects of the disclosure. Vehicles102aand/or102bofFIG.1can have the same or similar system architecture as shown inFIG.2. Thus, the following discussion of system architecture200is sufficient for understanding vehicle(s)102a,102bofFIG.1. However, other types of vehicles are considered within the scope of the technology described herein and may contain more or fewer elements as described in association withFIG.2. As a non-limiting example, an airborne vehicle may exclude brake or gear controllers, but may include an altitude sensor. In another non-limiting example, a water-based vehicle may include a depth sensor. One skilled in the art will appreciate that other propulsion systems, sensors, and controllers may be included based on a type of vehicle, as is known.

As shown inFIG.2, system architecture200includes an engine or motor202and various sensors204-218for measuring various parameters of the vehicle. In gas-powered or hybrid vehicles having a fuel-powered engine, the sensors may include, for example, an engine temperature sensor204, a battery voltage sensor206, an engine Rotations Per Minute (“RPM”) sensor208, and a throttle position sensor210. If the vehicle is electric or hybrid, then the vehicle may have an electric motor, and accordingly includes sensors such as a battery monitoring system212(to measure current, voltage and/or temperature of the battery), motor current214and voltage216sensors, and motor position sensors218such as resolvers and encoders.

Operational parameter sensors that are common to both types of vehicles include, for example: a position sensor236such as an accelerometer, gyroscope and/or inertial measurement unit; a speed sensor238; and an odometer sensor240. The vehicle also may have a clock242that the system uses to determine vehicle time during operation. The clock242may be encoded into the vehicle on-board computing device, it may be a separate device, or multiple clocks may be available.

The vehicle also includes various sensors that operate to gather information about the environment in which the vehicle is traveling. These sensors may include, for example: a location sensor260(e.g., a Global Positioning System (“GPS”) device); object detection sensors such as one or more cameras262; a lidar system264; and/or a radar and/or a sonar system266. The sensors also may include environmental sensors268such as a precipitation sensor and/or ambient temperature sensor. The object detection sensors may enable the vehicle to detect objects that are within a given distance range of the vehicle200in any direction, while the environmental sensors collect data about environmental conditions within the vehicle's area of travel.

During operations, information is communicated from the sensors to a vehicle on-board computing device220. The on-board computing device220(e.g., the on-board computing device113ofFIG.1, etc.) may be implemented using the computer system1600ofFIG.16. The vehicle on-board computing device220analyzes the data captured by the sensors and optionally controls operations of the vehicle based on results of the analysis. For example, the vehicle on-board computing device220may control: braking via a brake controller222; direction via a steering controller224; speed and acceleration via a throttle controller226(in a gas-powered vehicle) or a motor speed controller228(such as a current level controller in an electric vehicle); a differential gear controller230(in vehicles with transmissions); and/or other controllers. Auxiliary device controller254may be configured to control one or more auxiliary devices, such as testing systems, auxiliary sensors, mobile devices transported by the vehicle, etc.

Geographic location information may be communicated from the location sensor260to the on-board computing device220, which may then access a map of the environment that corresponds to the location information to determine known fixed features of the environment such as streets, buildings, stop signs and/or stop/go signals. Captured images from the cameras262and/or object detection information captured from sensors such as lidar system264is communicated from those sensors) to the on-board computing device220. The object detection information and/or captured images are processed by the on-board computing device220to detect objects in proximity to the vehicle200. Any known or to be known technique for making an object detection based on sensor data and/or captured images can be used in the embodiments disclosed in this document.

According to some aspects, the on-board computing device220may receive information from multiple sensors that are used to determine and/or provide pose related information, such as an inertial measurement unit (IMU) (not shown), the speed sensor238, the location sensor260, and the on-board computing device220may fuse (e.g., via one or more algorithms, etc.) the information from the multiple sensors and compare the fused information with lidar information high-definition map information.

Lidar information is communicated from lidar system264(e.g., the sensor system111ofFIG.1, etc.) to the on-board computing device220. Additionally, captured images are communicated from the camera(s)262(e.g., the sensor system111ofFIG.1, etc.) to the vehicle on-board computing device220. The lidar information and/or captured images are processed by the vehicle on-board computing device220to detect objects in proximity to the vehicle200. The manner in which the object detections are made by the vehicle on-board computing device220includes such capabilities detailed in this disclosure.

The on-board computing device220may include and/or may be in communication with a routing controller231that generates a navigation route from a start position to a destination position for an autonomous vehicle. The routing controller231may access a map data store (e.g., the database112ofFIG.1, a local storage of the on-board computing device220, etc.) to identify possible routes and road segments that a vehicle can travel on to get from the start position to the destination position. The routing controller231may score the possible routes and identify a preferred route to reach the destination. For example, the routing controller231may generate a navigation route that minimizes Euclidean distance traveled or other cost functions during the route, and may further access the traffic information and/or estimates that can affect an amount of time it will take to travel on a particular route. According to some aspects, the routing controller231can determine routes that avoid certain areas, such as keep out zones (KOZ), and/or the like. According to some aspects, the routing controller231may generate one or more routes using various routing methods, such as Dijkstra's algorithm, Bellman-Ford algorithm, or other algorithms. The routing controller231may also use the traffic information to generate a navigation route that reflects expected conditions of the route (e.g., current day of the week or current time of day, etc.), such that a route generated for travel during rush-hour may differ from a route generated for travel late at night. The routing controller231may also generate more than one navigation route to a destination and send more than one of these navigation routes to a user for selection by the user (e.g., via the user interface115ofFIG.1, etc.) from among various possible routes.

According to some aspects, the on-board computing device220may determine perception information of the surrounding environment of the AV102a. Based on the sensor data provided by one or more sensors and location information that is obtained, the on-board computing device220may determine perception information of the surrounding environment of the AV102a. The perception information may represent what an ordinary driver would perceive in the surrounding environment of a vehicle. The perception data may include information relating to one or more objects in the environment of the AV102a. For example, the on-board computing device220may process sensor data (e.g., lidar or radar data, camera images, etc.) in order to identify objects and/or features in the environment of AV102a. The objects may include traffic signals, roadway boundaries, other vehicles, pedestrians, and/or obstacles, etc. The on-board computing device220may use any now or hereafter known object recognition algorithms, video tracking algorithms, and computer vision algorithms (e.g., track objects frame-to-frame iteratively over a number of time periods) to determine the perception.

According to some aspects, the on-board computing device220may also determine, for one or more identified objects in the environment, the current state of the object. The state information may include, without limitation, for each object: current location; current speed and/or acceleration, current heading; current pose; current shape, size, or footprint; type (e.g., vehicle vs. pedestrian vs. bicycle vs. static object or obstacle); and/or other state information.

The on-board computing device220may perform one or more prediction and/or forecasting operations. For example, the on-board computing device220may predict future locations, trajectories, and/or actions of one or more objects. For example, the on-board computing device220may predict the future locations, trajectories, and/or actions of the objects based at least in part on perception information (e.g., the state data for each object comprising an estimated shape and pose determined as discussed below), location information, sensor data, and/or any other data that describes the past and/or current state of the objects, the AV102a, the surrounding environment, and/or their relationship(s). For example, if an object is a vehicle and the current driving environment includes an intersection, the on-board computing device220may predict whether the object will likely move straight forward or make a turn. If the perception data indicates that the intersection has no traffic light, the on-board computing device220may also predict whether the vehicle may have to fully stop prior to entering the intersection.

According to some aspects, the on-board computing device220may determine a motion plan for the autonomous vehicle. For example, the on-board computing device220may determine a motion plan for the autonomous vehicle based on the perception data and/or the prediction data. Specifically, given predictions about the future locations of proximate objects and other perception data, the on-board computing device220can determine a motion plan for the AV102athat best navigates the autonomous vehicle relative to the objects at their future locations.

According to some aspects, the on-board computing device220may receive predictions and decide how to handle objects and/or actors in the environment of the AV102a. For example, for a particular actor (e.g., a vehicle with a given speed, direction, turning angle, etc.), the on-board computing device220decides whether to overtake, yield, stop, and/or pass based on, for example, traffic conditions, map data, state of the autonomous vehicle, etc. Furthermore, the on-board computing device220also plans a path for the AV102ato travel on a given route, as well as driving parameters (e.g., distance, speed, and/or turning angle). That is, for a given object, the on-board computing device220decides what to do with the object and determines how to do it. For example, for a given object, the on-board computing device220may decide to pass the object and may determine whether to pass on the left side or right side of the object (including motion parameters such as speed). The on-board computing device220may also assess the risk of a collision between a detected object and the AV102a. If the risk exceeds an acceptable threshold, it may determine whether the collision can be avoided if the autonomous vehicle follows a defined vehicle trajectory and/or implements one or more dynamically generated emergency maneuvers is performed in a pre-defined time period (e.g., N milliseconds). If the collision can be avoided, then the on-board computing device220may execute one or more control instructions to perform a cautious maneuver (e.g., mildly slow down, accelerate, change lane, or swerve). In contrast, if the collision cannot be avoided, then the on-board computing device220may execute one or more control instructions for execution of an emergency maneuver (e.g., brake and/or change direction of travel).

As discussed above, planning and control data regarding the movement of the autonomous vehicle is generated for execution. The on-board computing device220may, for example, control braking via a brake controller; direction via a steering controller; speed and acceleration via a throttle controller (in a gas-powered vehicle) or a motor speed controller (such as a current level controller in an electric vehicle); a differential gear controller (in vehicles with transmissions); and/or other controllers.

Returning toFIG.1, as described inFIG.2, the on-board computing device113(e.g., the on-board computing device220ofFIG.2) may determine perception information of the surrounding environment of the AV102a. For example, according to some aspects, the on-board computing device113may include a perception module120. The sensor system111may provide the perception module120sensor and/or sensor-related data/information, and the on-board computing device113may provide the perception module120vehicle control and operation information. For example, the sensor system111may provide the perception module120with data logs, such as log slices, that describe all data collected by the sensor system111with predefined time intervals (e.g., 16-second intervals, etc.). The on-board computing device113may provide the perception module120with data logs, such as log slices, that describe all data collected by on-board computing device113with predefined time intervals (e.g., 16-second intervals, etc.). For example, data logs may include data/information such as route response messages, route progress messages, sensor information (e.g., lidar information, etc.), autonomous vehicle pose information (e.g., orientation information, pose, etc.), and/or the like pertaining to the AV102aand/or road actors (e.g., objects102b,114,116, etc.). The perception module120may use data/information from components of the system100, such as sensor and/or sensor-related data/information from the sensor system111, vehicle control and operation information from the on-board computing device113, and raw data, indexed data, structured data, map data, and/or program instructions from the database112to identify and/or determine objects (e.g., spatially relevant objects, etc.) affecting a trajectory of the AV102a.

According to some aspects, the perception module120may include a multi-layered architecture and/or data infrastructure configured to identify and/or determine and/or road actors affecting a trajectory of the AV102abased on the perceptual relevancy of the objects and/or road actors. For example, the perception module120may include a data pre-processing layer, a computational layer, and a visualization layer.

According to some aspects, the data pre-processing layer of the perception module120may be configured to collect and/or prepare all data/information received by the perception module120to be used by the computational layer to determine objects (e.g., spatially relevant objects/road actors, etc.) affecting a trajectory of the AV102a. According to some aspects, the data pre-processing layer may be used to load, associate, and/or map labels to data structures (e.g., data structures that indicate the AV102a, road actors, and/or objects sensed by the sensing system111, etc.). For example, each log slice received by the data pre-processing layer may be associated with a label file (e.g., ground truth data/information, a BlinkyGroundTruth.json, etc.). The data pre-processing layer may receive label files from the database112and/or the like, load and parse the label files, and extract information such a time of validity (TOV), amodal cuboid, shrinkwrapped cuboid (e.g., a cuboid that is tightly fit to returned lidar points and therefore does not include any occluded portions of the vehicle, etc.), and velocity/speed for any tracked object. For example, the data pre-processing layer may create a TOV reference list from Lidar sweep messages received from the sensor system111.

According to some aspects, the data pre-processing layer may determine timing labels for tracks based on the TOV of the lidar sweep messages. The data pre-processing layer may execute a function that reads lidar sweep messages, extracts the TOV, and constructs a list of integer values. The list of integer values may be compared, by the calculation layer of the perception module120when determining objects (e.g., spatially relevant objects/road actors, etc.) affecting a trajectory of the AV102a, to the TOV of a track to determine if the track is visible and/or labeled in a given frame or not.

According to some aspects, the data pre-processing layer may automatically perform interpolation to determine the location and speed (or velocity) of the AV102a. According to some aspects, the data pre-processing layer may be configured with a function that reads the pose messages from each log slice and use the pose messages to create a command that may be used by the output and visualization layer to generate a pose output and/or representation of the AV102a. For example, the data pre-processing layer may determine the speed of the AV102aat a specific TOV and execute a function at the specific TOV using SE3 transformations to AV frame data/information to generate a pose output and/or representation of the AV102a.

According to some aspects, the data pre-processing layer may determine and/or extract a route for the AV102afrom received log slices. For example, the data pre-processing layer may execute a function that extracts route messages from each log slice. A route message may include lane segments, denoted by universally unique identifiers (UUIDs), associated with a drivable area (e.g., for where the AV102ais supposed to go, etc.), such as along a route traveled by the AV102a. According to some aspects, within historic (old) logs where a route response message does not exist, the function may read a route progress message to extract the lane segments traveled by the AV102a, and then extend the list through the lane segment successors up to a predefined constant.

According to some aspects, the data pre-processing layer may prepare vector map data that may be used to determine objects (e.g., spatially relevant objects, etc.) affecting a trajectory of the AV102a. Each log may correspond to a specific map version, the data pre-processing layer may determine and/or read vector map data/information (e.g., via a vector map layer, etc.) for the specific map from one or more interfaces provided by the remote computing device110. For example, the data pre-processing layer may execute a function and/or logic that causes the data pre-processing layer to determine and/or read vector map data/information for a specific map from one or more interfaces provided by the remote computing device110. Execution of a function and/or logic may load lane segment data/information, such as a lane segment list, etc., and generate a drivable area around the route of the AV102a. The function also executes a union operation on the route lane segments to generate a map corridor with a computed centerline path. The centerline is assumed the path of the AV102aand simulated longitudinal acceleration/deceleration trajectories can be interpolated to the centerline, for example, by the computational layer of the perception module120. An algorithm for generating a map corridor with a computed centerline path is shown in Algorithm 1 below.

Algorithm 1

Input: lane segments that belong to the AV routeOutput: centerline of the AV routeSteps:1. Create Av_route polygon using shapely union operation of on all input lane_segements2. Extract the exterior points of the av_route polygon3. For each point in the list-sort it to either left point or right point4. Create a centerline by triangulation between the left and right points then add the midpoint to the map_corridor_cl_list to be returned5. Compute the heading for each lane segment

FIG.3Ais an example result of Algorithm 1.FIG.3Amay be output by the visualization layer of the perception module120.FIG.3Adepicts an extracted drivable area301and lane segments302belonging to a route303of the AV102a.FIG.3Bis an example output of Algorithm 1.FIG.3Bmay be output by the visualization layer.FIG.3Bdepicts a map corridor and extracted centerline. According to some aspects,FIG.3Bdepicts predicted lane changes304associated with the route303(A person of ordinary skill would appreciate that the AV102acould perform a different maneuver for lane change and thatFIG.3Bprovides a reasonable approximation).

Returning toFIG.1, according to some aspects, the data pre-processing layer may extract lidar extrinsic data/inform from data logs (e.g., log slices, etc.) received from the sensor system111. SE3 transformations to AV frame data/information may be used by the calculation layer of the perception module120to calculate the detectability of road actors and transform a 3-dimensional cuboid representing the frame of the AV102ato a lidar frame. The data pre-processing layer may execute a function that reads rotation and translation matrices from a JSON file and/or the like from the log directory via an interface and/or the like.

According to some aspects, the calculation layer of the perception module120may compute motion profiles for the AV102a. For example, the calculation layer of the perception module120may compute motion profiles for the AV102aby executing a spatially relevant object (SRO) algorithm and/or the like for each TOV reference list entry (e.g., per frame, etc.). According to some aspects, the calculation layer may determine an acceleration profile for the AV102a. The calculation layer may execute trajectory optimization functions to calculate a longitudinal trajectory between a current speed of the AV102aextracted from the pose interface and a target speed of lane segment speed. According to some aspects, if the target speed is reached before a 4-second interval, the longitudinal trajectory may be extended at a constant speed. According to some aspects, if the target speed is reached after a 4-second interval the trajectory may be cropped to four-second.

According to some aspects, the calculation layer may determine a deceleration profile for the AV102a. The calculation layer may execute trajectory optimization functions to calculate a longitudinal trajectory between a current speed of the AV102aextracted from the pose interface and a target speed of zero. The trajectory optimization functions provide an optimal longitudinal trajectory for the AV102ato achieve a final condition (e.g., target speed, pose, acceleration, etc.) from initial conditions (e.g., initial speed, pose, acceleration, etc.) within constraints (e.g., max allowed acceleration, deceleration, max allowed Jerk, etc.). According to some aspects, if the target speed is reached before a 4-second interval, the longitudinal trajectory may be extended at 0 speed (in place) until the 4-second interval. According to some aspects, if the target speed is reached after a 4-second interval the trajectory may be cropped to four-second.

For each of the motion profiles (e.g., the acceleration motion profile, the deceleration motion profile, etc.), the longitudinal acceleration trajectory is projected/interpolated on the map corridor centerline starting from the closest point of the pose of the Av102A at THE current TOV. The calculation layer may sample trajectory, for example, at 10 HZ. The calculation layer may use information from the two motion trajectories to calculate isotemporal regions and collision severity when determining the severity of a road actor.FIG.3Cshows an example of the longitudinal acceleration trajectory projected to the centerline determined from Algorithm 1.FIG.3Cmay be output by the visualization layer.

According to some aspects, the calculation layer of the perception module120may determine and/or compute isotemporal regions for the AV102a. It will be appreciated that although the route of the AV102ais determined (and depicted) at the lane segments level, the route could be very wide and the AV102amight occupy any space of that route (note that inFIG.3Cthe AV102adoes not follow the centerline304). It will be appreciated that when determining objects (e.g., spatially relevant objects, etc.) affecting a trajectory of the AV102a, the perception module120may consider the whole map corridor (e.g., route303, etc.). To reduce the use of the computational resources of the on-board computer113, when determining objects (e.g., spatially relevant objects, etc.) affecting a trajectory of the AV102a, the perception module120may discretize the motion profile for the AV102aalong the map corridor at (e.g., the route303, etc.) one-second intervals. The resulting polygons are called isotemporal polygons.

According to some aspects, to determine isotemporal regions for the Av102A, the perception module120may use a map corridor (e.g., the route303, etc.) centerline heading to draw perpendicular lines at set and/or defined intervals (e.g., one-second intervals, etc.), intersecting points between these lines with the left and right boundary lines of the map corridor to create the isotemporal region for the acceleration or deceleration profiles of the AV102a. It will be appreciated that the isotemporal regions may overlap with each other as the AV102ahas non-zero width and length.FIG.3Dshows an example of isotemporal regions305for the acceleration trajectory of the AV102a.FIG.3Dmay be output by the visualization layer.

As previously described herein, the perception module120may access and/or receive ground truth information (e.g., a ground truth data file, etc.), for example, from the remote computing device110and/or the like that includes labels for road actors that are partially occluded. To ensure that the perception module120is not penalized for not tracking occluded road actors above certain thresholds, the computational layer may compute a detectability value (e.g., an occlusion percentage, etc.) and perform a thresholding operation to determine if road actors and/or objects, such as the objects102b,114,116, are detectable or not. The computational layer may perform a detectability based on a detectability algorithm. For example, an algorithm for determining if an object and/or road actor is detectable is shown in Algorithm 2 below.

Algorithm 2

Input: Road Object/actors' shrinkwrapped cuboid and amodals_cuboid, lidar SE3 vehicleOutput: Boolean value indicates if an object/actor is detectable or notSteps:1. Transform both input cuboids from vehicle frame to lidar frame2. Transform each point of the cuboids (lidar frame) to azimuth-elevation frame (project into space), (after this step all the faces of each cuboid will be transformed to polygons in the same plane)3. Merge resulting polygons from each cuboid through union operation to determine two polygons—one from the shrinkwrapped and another one from the amodal cuboid4. Compute intersection over union (IOU) between the two polygons5. Compare the resulting IOU with a threshold (e.g., IOU_AZI_ELE_THRESHOLD, etc.) to determine detectability

FIG.4Ais an example output of Algorithm 2 for a road actor401.FIG.4Amay be output by the visualization layer of the perception module120. The track for the road actor401may include a UUID (e.g., track_id=42). An information element402may display relevant information such as the UUID for the road actor401, a speed of the road actor401, and an indication of the thresholded IOU for the road actor401(e.g., the road actor401satisfies the detectability threshold, etc.).FIG.4Bshows an example of a shrinkwrapped and amodal representation of the road actor401resulting from Algorithm 2.FIG.4Bmay be output by the visualization layer. A shrinkwrapped representation403and an amodal representation404are shown in a lidar frame.FIG.4Cshows an example of the shrinkwrapped representation403and the amodal representation404projected to an azimuth-elevation frame.

According to some aspects, the calculation layer of the perception module120may determine and/or compute motion profiles for road actors within a drivable area of the AV102a. For example, motion profiles may be determined for large vehicles, regular vehicles, and/or motorcycle actors that pass the detectability check. For detected road actors, the calculation layer may generate deceleration and acceleration (including maintaining lane speed) trajectories to model the behavior of the road actors in the next 4 seconds. The direction of both acceleration and deceleration trajectories may be limited by lateral accelerations and minimum turning radii. According to some aspects, acceleration trajectories may also simulate road actor behaviors/actions such as u-turns, left turns, and right turns. For example, deceleration profiles may be generated and/or determined for detected road actors based on Algorithm 3 shown below.

Algorithm 3

Input: lateral acceleration limit, initial velocity, deceleration limits, jerk limits, map_SE3_actor, minimum turning radiusOutput: deceleration trajectoriesSteps:1. Compute starting turning radius using the following equation:
r0=max(initial_velocity2/lateral acceleration limit, minimum turning radius)2. For r=r0to 10,000 m (nearly straight line):a. Compute longitudinal trajectory using bang-off-bang trajectory optimization functions, that evaluates the vehicle from to νstart=ν0mph to νend=ν0mph, using deceleration and jerk limits input parametersb. To determine longitudinal trajectory, extend the beginning of the longitudinal trajectory by tlatencyat a constant speed of ν0to simulate vehicle reaction timec. If the duration of the trajectory is less than a value (e.g., four seconds, etc.), extend in place until four seconds. If the trajectory length is more than four seconds, trim the trajectory to four secondsd. Compute the (xi, yi) locations along the trajectory using the following equation:

θi=sir⁢(xi,yi)=(r*sin⁢θi,r*(cos⁢θi-1))e. Transform to map coordinates using map_SE3_actorf. Create a trajectory from position, velocity information computed above (xi, yi). Above (xi, yi) corresponds to the center of the rear axle of the road actorg. Append trajectory to output list

According to some aspects, longitudinal trajectories to stop a road actor from arbitrary speed ν0may be generated and/or determined based on Algorithm 4 shown below.

Algorithm 4

Input: lateral acceleration limit, initial lateral acceleration limit, initial velocity, deceleration limits, jerk limitsOutput: longitudinal trajectory that satisfies initial and final conditions within vehicle capability constraintsSteps:1. Compute stopping distance from ν0, for example, using a motion control function from a third-party motion control module. For example, the function “PyLongitudinalPropagator.minimum_stopping_distance_m( )” may be used to compute the stopping distance2. Use the computed stopping distance as seed for the function compute_time_optimal_trajectory_bob( ) with the following constraints:νstart=ν0;νend=0.0 mps;astart=0.0 mps;aend=0.0 mps;sstart=0 m;send=sf

mina⁢c⁢c⁢e⁢l=-max_decel-g*sin⁡(arctan⁡(grade1⁢0⁢0));maxjerk=max_jerk;**grade, max_decel and max_jerk are user defined3.a. If the return of the function compute_time_optimal_trajectory_bob( ) is “Not successful”, recall the function with the same constraints but with send+=0.25 m until the function returns “Success”b. If the return of the function compute_time_optimal_trajectory_bob( ) is “Successful”, recall the function with same constraints but with send−=0.25 m until the function returns “Not Success”c. Return the last successful bang-off-bang computed trajectory which contains the following information (after downsampling to 100 points):time_stamp along the trajectory=t[ ];Longitudinal distance along the trajectory=s[ ];Longitudinal velocity along the trajectory=ν[ ];Longitudinal acceleration along the trajectory=a[ ];

FIGS.5A and5Bshow examples of road actor deceleration profiles at different speeds.FIGS.5A and5Bmay be output by the visualization layer of the perception module120.FIG.5Ashows a road actor501and deceleration trajectories502. An information element503may display relevant information such as the UUID (e.g., track_id=41) for the road actor501, a speed of the road actor501, and a severity ranking (S1_2) which represents, for example, a degree of damage caused by a potential collision with the AV102a(e.g., if it is miss detected and/or undetected by the perception module120, etc.), and an indication of the thresholded IOU for the road actor501(e.g., an indication of that the road actor501satisfies the detectability threshold, etc.).

FIG.5Bshows a road actor504and deceleration trajectories505. An information element506may display relevant information such as the UUID (e.g., track_id=17) for the road actor504, a speed of the road actor501, and an indication of the thresholded IOU for the road actor504(e.g., an indication of that the road actor501is not relevant because it does not satisfy the detectability threshold, etc.).

According to some aspects, the calculation layer of the perception module120may determine acceleration profiles for road actors. An acceleration profile simulates a road actor accelerating from its current speed (extracted from the ground truth labels) to the lane speed (or maintaining its current speed if it is at lane speed). The perception module120may consider non-compliant actors by modeling the acceleration path as driving on circles of different radii for distance, then continuing on a straight line from the last heading at the end of the circular movement. The minimum turning radius may be dependent on the road actor (e.g., the information may be extracted from JAMA object parameters and/or determined by assumption.FIGS.6A-6Dshow trajectories of non-compliant road actors by modeling the acceleration path as driving on circles of different radii for distance, then continuing on a straight line from the last heading at the end of the circular movement.FIGS.6A-6Dmay be output by the visualization layer of the perception module120

FIG.6Ashows an example of acceleration motion for a road actor from the last heading of the circular motion.FIG.6Ashows acceleration motion for a road actor600performing a non-compliant turn in place in front of the AV102astarting from a speed of zero miles per hour.FIG.6Bshows an example of circular motion and linear motion for a road actor from the last heading of the circular motion.FIG.6Bshows circular motion601and linear motion602for the road actor600.

FIG.6Cshows an example of acceleration motion for a road actor from the last heading of the circular motion.FIG.6Cshows acceleration motion for a road actor603performing a turn in place in front of the AV102astarting from a speed of thirty miles per hour. As shown inFIG.6C, a U-turn, a left-turn, and a right turn is not feasible for the road actor603at thirty miles per hour.FIG.6Dshows an example of circular motion604and linear motion605for the road actor603.

According to some aspects, acceleration profiles for road actors may be generated and/or determined based on Algorithm 5 shown below.

Algorithm 5

Input: lateral acceleration limit, initial velocity, deceleration limits, jerk limits, map_SE3_actor, minimum turning radiusOutput: acceleration trajectoriesSteps:1. Compute starting radius using the following equation:
r0=max(initial_velocity2/lateral acceleration limit, minimum turning radius)2. For r=r0to 10,000 m (nearly straight line):a. Compute longitudinal trajectory that takes the road actor from from νstart=road actor's current speed, to final speed (max lane segment speed within a circle of DISTANCE_FROM_OV_FOR_LANE_SPEED_SEARCH=20 m radius centered around the road actor)b. If the duration of the trajectory is less than a value (e.g., four seconds, etc.), extend the duration using constant speed until four seconds. If the trajectory length is more than four seconds, trim the trajectory to four seconds.c. Compute the (xi, yi) locations along the trajectory using the following equation:

θi=sir(xi,yi)=(r*sin⁢θi,r*(cos⁢θi-1));for⁢si<d(xi,yi)=(yi-si*sin⁢θi,yi+si*(cos⁢θi-1));for⁢⁢si>dd. Transform (xi, yi) to map coordinates using map_SE3_actore. Create a trajectory from position, velocity information computed above (xi, yi). Above (xi, yi) corresponds to the center of the rear axle of the road actorf. Append trajectory to output list

According to some aspects, if a road actor is found to be detectable and within a drivable area of the AV102a, the calculation layer of the perception module120may use Algorithm 6, shown below, to find if the road actor is relevant or not.

Algorithm 6

Input:Evaluation_list=[(ov_accel_trajectories, av_isotemporal_accel_polygon),(ov_accel_trajectories, av_isotemporal_decel_polygon),(ov_decel_trajectories, av_isotemporal_accel_polygon),(ov_decel_trajectories, av_isotemporal_decel_polygon)];drivable area, stationary road actorsOutput: a list of trajectories that make the road actor intersect with the AV (AV102a) for each (ov_trajectoies, av_isotemporal_polygons) pair in evaluation listSteps:1. Down-select trajectories that intersect with the union of av_isotemporal_polygon2. Trim any trajectories that leave the drivable area or intersect with other static objects or road actors3. Downselect trajectories that intersect with the isotemporal polygons within their respective time windows4. Return the output of step 3

FIGS.7A-7Cshow example outputs for each step of Algorithm 6.FIGS.7A-7Cmay be output by the visualization layer of the perception module120.FIG.7Ashows trajectories700of a road actor701that intersect with at least one isotemporal polygon representation702of the AV102aat a respective time window along an intended path of the AV102awithin a drivable area703.FIG.7Bshows trajectories700of the road actor701that leave the drivable area703(into a non-drivable area704) or intersect with other static objects or road actors within the drivable area703.FIG.7Cshows trajectories700of the road actor701that intersect with the isotemporal polygons within their respective time windows.

According to some aspects, the perception module120may manage special and/or outlier situations where objects (e.g., spatially relevant objects, etc.) affect a trajectory of the AV102a. For example, the perception module120considers parked vehicles along a route of the AV102a. If a road actor has a speed of less than a preset value (e.g., 0 mph, etc.), the perception module120may consider the road actor relevant if its polygon intersects with any of the acceleration or deceleration isotemporal regions of the AV102a. The perception module120considers situations where road actors may be traveling in reverse by determining if the reversing motion profiles of the road actors intersect with any of the acceleration or deceleration isotemporal regions of the AV102a. If the reversing motion profiles of the road actors intersect with any of the acceleration or deceleration isotemporal regions of the AV102a, the road actor will be considered relevant. The perception module120considers situations where road actors (either parked or in motion) have any open doors and/or protruding elements, and may dilate the isotemporal polygon of the AV102aaccordingly. The perception module120considers situations where the AV102acomes to a stop behind a stopping/stopped road actor causing the respective braking isotemporal polygons will have an intersection but the respective accelerating polygons may not. The perception module120considers situations, where the AV102acomes to a stop behind a stopping/stopped road actor spatially relevant because the road actor may not always be able to accelerate (e.g., if there is at a red light, etc.). The perception module120considers situations, where a road actor is parked and pulls out into a lane traveled by the AV102a. The respective braking isotemporal polygons of the road actor and the AV102awill not intersect, but the respective accelerating isotemporal polygons will. The perception module120considers situations, where a road actor is parked and pulls out into a lane traveled by the AV102ato be spatially relevant to the AV102abecause this is a common scenario that the AV102amust decelerate for. The perception module120considers situations where the AV102amay decelerate to avoid a collision with a road actor as part of the motion profile for the AV102ato avoid causing the AV102ato cause another collision by avoiding collision. According to some aspects, the perception module120may manage any special and/or outlier situations where objects (e.g., spatially relevant objects, etc.) affect a trajectory of the AV102aby considering the motion profiles of all perceived road actors in multiple situations (both potential situations and hypothetical situations).

According to some aspects, the calculation layer of the perception module120may determine and/or compute a classification, such as a severity ranking and/or the like, for each perceived road actor. The perception module120may execute logic and/or a function in a class that assumes that the perception module120did not output a track for a road actor and the resulting worst-case severity of a collision between the AV102aand the road actor. For each road actor trajectory identified and/or determined to be relevant to a trajectory of the AV102a, the logic and/or a function intersects that trajectory with the corresponding trajectory of the AV102ato find the collision points and then assigns and/or determines the collision severity at that point. The perception module120considers that a road actor could collide with the AV102ain multiple locations because its motion profiles could have multiple trajectories. Accordingly, the calculation layer may assign and/or determine the severity of the road to be the maximum severity at the multiple collision points.

FIG.8shows an example of a situation where a trajectory of a road actor intersects the trajectory of the AV102a.FIG.8may be output by the visualization layer of the perception module120. The trajectory of the AV102aand the trajectory of the road actor are transformed to a collision point800and the collision severity is determined and/or assigned at the collision point800. A vector801represents the relative velocity at the collision point800.

FIG.9shows an example of a situation where a trajectory of a road actor does not intersect the trajectory of the AV102abut intersect with an isotemporal polygon of the AV102a.FIG.9may be output by the visualization layer of the perception module120. According to some aspects, in situations where a trajectory900of a road actor901does not intersect a trajectory902of the AV102abut intersects with an isotemporal polygon of the AV102a, the calculation layer of the perception module120may consider all possible interactions between the road actor901and the AV102a. The calculation layer of the perception module120may consider all possible interactions between the road actor901and the AV102aby, for each point along its trajectory900, transform/project isotemporal polygons of the AV102to each point along trajectory902. The calculation layer may determine the time for each point along trajectory902. If the time for a point903along the trajectory902corresponds to the isotemporal region of the AV102a, the calculation layer may transform/project the road actor901on its trajectory900to the point903and determine/compute a severity ranking (e.g., maximum severity, etc.) for the collision. If a collision does not occur based on the transformation/projection, the severity ranking may be zero (e.g., no severity, etc.).

FIG.10shows an example situation where a road actor is parked but within proximity to a trajectory of the AV102a.FIG.10may be output by the visualization layer of the perception module120. According to some aspects, in situations where a road actor1000is parked but within proximity to a trajectory1001of the AV102a, the representation of the road actor1000may be transformed/projected (projection line1002) to the closest point on the trajectory1001and a severity ranking may be determined and/or assigned. The speed of the road actor1000may be considered zero miles per hour.

According to some aspects, the perception system120, for example via the visualization layer may provide multiple interfaces to visualize the data describing objects (e.g., spatially relevant objects, etc.) affecting a trajectory of the AV102a. For example, the visualization layer may include integration applications that provide integration between notebooks, documents, and activities such as JupyterLab, Eclipse, PyCharm, AWS Cloud9, Kite, Wing Python IDE, Selenium IDE, ae like. The visualization layer may execute one or more functions to visualize any algorithm used to determine perceptually relevant objects (e.g., spatially relevant objects/road actors, etc.) that may affect a trajectory of the AV102a(e.g.,FIGS.7A-7C, etc.), visualize detectability of road actors (e.g.,FIGS.4A-4C, and/or visualize severity interactions (e.g.,FIGS.8-10, etc.). According to some aspects, the visualization layer of the perception module120may output classifications (e.g., indications of severity, relevancy, etc.) for road actors.

According to some aspects, the visualization layer of the perception module120may output and/or cause a display of relevancy statistics for road actors. For example, the perception module120may execute a function that provides an interface to allow the spatially relevant object (SRO) algorithms to run at scale on thousands of log slices. For each log slice, the output may be a data file, such as a Python dictionary and/or the like, that includes stored data elements/values (e.g., maps, etc.) that may be serialized and imported to other systems for further analysis or visualization.

According to some aspects,FIG.11provides a high-level overview of operations performed at each layer of the perception module120. As shown inFIG.11and according to some aspects, the data pre-processing layer of the perception module120may facilitate loading labels to data structures. For example, the data pre-processing layer of the perception module120may load and/or facilitate loading ground truth labels that include labels for road actors that are partially occluded, timing labels for tracks based on the TOV of the lidar sweep messages, amodal cuboid labels, shrinkwrapped labels, and/or velocity/speed labels for any tracked object based on information received from a remote source, third-party, and/or a local repository. The data pre-processing layer of the perception module120may use and/or facilitate the use of sensor data/information to create/generate TOV reference list(s) from lidar sweep messages. The data pre-processing layer of the perception module120may create/generate a pose interface that displays and/or represent a pose (e.g., orientation, etc.) for the AV102aand/or any sensed objects/actors. The data pre-processing layer of the perception module120may extract and/or facilitate extraction of lidar extrinsics, generate and/or prepare vector map data (e.g., data used to represent a drivable area, lane segments, etc.), and/or determine (e.g., extract from on-board computing data, etc.) a route of the AV102a.

As shown inFIG.11and according to some aspects, the computational layer of the perception module120may execute algorithms and/or functions to compute and/or determine motion profiles from road actors, compute and/or determine a motion profile for the AV102a, and compute and/or determine the detectability of the road actors. The computational layer of the perception module120may execute algorithms and/or functions to check the perceptual relevancy of the road actors for the AV102a, compute and/or determine isotemporal regions for the AV102a, and based on the relevancy of the road actors and respective motion profiles, compute, determine, and/or assign a severity ranking for a collision between the road actors and the AV102a.

As shown inFIG.11and according to some aspects, the visualization layer of the perception module120may be used to visualize and/or display the output and/or execution of algorithms and/or functions executed by the computational layer. The visualization layer may be used to visualize and/or display severity interactions between road actors and the AV102a, and/or visualize and/or display relevancy statistics for road actors. For example, according to some aspects, the visualization layer may include, generate, and/or communicate with a user interface that displays and/or outputs relevancy and/or severity information for any object and/or road actor.

FIG.12shows a flowchart of an example method400determining perceptual relevancy of objects and/or road actors, according to some aspects. Method1200can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG.12, as will be understood by a person of ordinary skill in the art.

Method1200shall be described with reference toFIGS.1-11. However, method1200is not limited to the aspects of those figures. The on-board computing device113(e.g., the perception module120, etc.) may facilitate detection and/or determination of objects and/or road actors kinematically capable, even if non-compliant with rules-of-the-road, of affecting a trajectory of a vehicle.

In1201, on-board computing device113creates a time of validity (TOV) reference list from lidar sweep messages. According to some aspects, the lidar sweep messages may be based on pre-defined intervals, such as 0.1 seconds and/or the like. The lidar sweep messages may provide 360-degree sensing information relative to an autonomous vehicle (AV) to which it is associated. The on-board computing device113may use any method to create a TOV reference list.

In1202, on-board computing device113loads labels to data structures sensed by a sensor (e.g., lidar sensor, the sensor system111, etc.). According to some aspects, the on-board computing device113may determine the labels from ground truth information provided to the on-board computing device113, for example, via the database11and/or the like. The on-board computing device113may use any method to associate labels with data structures.

In1203, on-board computing device113creates and/or generates a pose interface. According to some aspects, the pose interface may display and/or represent a pose (e.g., orientation, etc.) for the AV and/or any sensed objects/actors. The on-board computing device113may use any method to generate a pose interface and/or determine a pose of the AV and/or road actors.

In1204, on-board computing device113prepares vector map data. According to some aspects, vector map data may be sent to the on-board computing device113, for example, by a remote computing device (e.g., remote computing device110, etc.) and/or the like. According to some aspects, the on-board computing device113may be preconfigured with the vector map data, and/or the vector map data (or portions thereof) may be extracted from a database associated with the AV and/or a third-party source. According to some aspects, the vector map data may include polygons, lines, and points which make up all the features on a map describing the drivable area. For example, the vector map data may provide representation and/or description of a drivable area for the AV and/or lane segment information for any road/street within the drivable area. The on-board computing device113may use any method to prepare vector map data.

In1205, on-board computing device113determines a route for the AV. According to some aspects, computing device113may extract the route for the AV from log data stored by the on-board computing device113. According to some aspects, the route for the AV may be sent to the on-board computing device113, for example, by a remote computing device (e.g., remote computing device110, etc.) and/or the like. The on-board computing device113may use any method to determine the route for the AV.

In1306, on-board computing device113determines sensor data. For example, the on-board computing device113may extract sensor data such as lidar extrinsic data from logs received from a sensor (e.g., lidar sensor, the sensor system111, etc.). According to some aspects, lidar extrinsic data may include rotation and translation matrices needed to transform/project3D cuboids to a lidar frame. The on-board computing device113may use any method to determine data needed to transform/project3D cuboids to any reference frame.

In1207, on-board computing device113determines motion profiles for the AV. The motion profiles may include acceleration and deceleration trajectories for the AV. Acceleration and deceleration trajectories for the AV may include longitudinal trajectories between a current speed of the AV and a target speed (e.g., a lane segment speed, etc.) within a drivable area. Acceleration and deceleration trajectories for the AV may be determined and/or depicted for various time intervals, such as four-second look-ahead time intervals and/or the like. The on-board computing device113may use any method to determine motion profiles for the AV.

In1208, on-board computing device113determines isotemporal regions for the AV. The on-board computing device113may determine isotemporal regions for the AV by discretizing the motion profiles for the AV along the AV route at set time intervals, such as one-second intervals and/or the like. The on-board computing device113may use any method to determine isotemporal regions for the AV.

In1209, on-board computing device113determines motion profiles for road actors. For road actors that pass a detectability check, the on-board computing device113may generate deceleration and acceleration (including maintaining lane speed) trajectories to model the behavior of the road actors in the next four seconds. According to some aspects, the direction of both acceleration and deceleration trajectories are limited by lateral acceleration and minimum turning radii of the road actors. Acceleration trajectories may simulate driving maneuvers of the road actors such as u-turns, left-turns, and/or right turns. The on-board computing device113may use any method to determine motion profiles for road actors.

In1210, on-board computing device113determines road actor detectability. The on-board computing device113may determine a detectability value (occlusion percentage) for each road actor and perform a thresholding operation to determine if a road object or actor is detectable or not. According to some aspects, the thresholding operation may determine and consider how much (e.g., percentage, etc.) of a road actor is occluded, what occludes the road actor, the motion profile of the road actor, and/or the like. Detectability values that satisfy (e.g., meet, exceed, etc.) a detectability threshold may be considered relevant. Detectability values that do not satisfy (e.g., are less than, etc.) a detectability threshold may be considered not relevant.

For example, a road actor such as a large vehicle may be assigned a threshold satisfying detectability value and be considered relevant because the large vehicle is not occluded, is moving at a speed within a drivable area. A road actor such as a small vehicle may be assigned a threshold satisfying detectability value and be considered relevant because the small vehicle is 80% occluded, but moving at a speed within a drivable area. A road actor may be assigned a detectability value that does not satisfy a detectability threshold and be considered not relevant because the road actor is 2% occluded, but not moving at a speed within a drivable area. The on-board computing device113may use any method to determine road actor detectability.

In1211, on-board computing device113assigns and/or determines a classification for road actors determined to be relevant. The on-board computing device113may determine a classification (e.g., an indication of severity, relevancy, etc.) for detected road actors. For example, severity may be assigned to each detected road actor by determining, based on the respective motion profiles, road actors with trajectories that intersect an isotemporal polygon representation of the AV. According to some aspects, a severity ranking may be based on the type of AV, the type of road actor, the velocity of the AV and/or road actor, a collision point, and/or any other criteria. According to some aspects, the criteria for severity may be user-determined, predefined, updated/modified, determined by a predictive model and/or AI of the on-board computing device113, and/or the like. The on-board computing device113may use any method to determine a classification for road actors determined to be relevant.

In1212, on-board computing device113determines if all road actors have been processed. The on-board computing device113may determine if each road actor determined to be relevant has been assigned a severity. For example, the on-board computing device113may determine if the number of road actors assigned a severity corresponds to the number of road actors deemed relevant. If the number of road actors assigned a severity does not correspond to the number of road actors deemed relevant, the process may return to step1209. If the number of road actors assigned a severity corresponds to the number of road actors deemed relevant, the process progresses to1208.

In1213, on-board computing device113determines if all frames (e.g., TOV list entries, etc.) have been processed. The on-board computing device113determines each frame received with sensor data. For example, each frame may correspond to a TOV of a lidar sweep message. The on-board computing device113may determine if each lidar sweep message has been processed. If each lidar sweep message has not been processed, the process may return to step1207. If each lidar sweep message has been processed, the process progresses to1214.

In1214, on-board computing device113outputs the relevancy and/or severity for all road actors, and enables visualization of each considered scenario for determining the relevancy and/or severity for all road actors. For example, the on-board computing device113may include a user interface that enables visualization of the relevancy and/or severity for all road actors and enables visualization of each considered scenario for determining the relevancy and/or severity for all road actors. The on-board computing device113may output/send one or more data files, API calls, and/or the like that enable another device to cause display of the relevancy and/or severity for all road actors, and enables another device to cause display of each considered scenario for determining the relevancy and/or severity for all road actors.

FIG.13shows a flowchart of an example method1300determining objects (e.g., spatially relevant objects, etc.) that are kinematically capable, even if non-compliant with rules-of-the-road, of affecting a trajectory of a vehicle, according to some aspects. Method1300can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executing on a processing device), or a combination thereof. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG.13, as will be understood by a person of ordinary skill in the art.

Method1300shall be described with reference toFIGS.1-12. However, method1300is not limited to the aspects of those figures. The on-board computing device113(e.g., the perception module120, etc.) may facilitate identifying, detecting, and/or determining objects (e.g., spatially relevant objects, etc.) that are kinematically capable, even if non-compliant with rules-of-the-road, of affecting a trajectory of a vehicle.

In1310, on-board computing device113generates and/or determines a trajectory for a vehicle and a respective trajectory for each object of a plurality of objects within a field of view (FOV) of a sensing device associated with the vehicle. The on-board computing device113may receive sensor information from the sensing device. The sensing device may include at least one of a Light Detection and Ranging (lidar) sensing device, an ultrasonic sensing device, a depth-sensing device, a Radio Detection and Ranging (RADAR) device, or a camera. The on-board computing device113may identify, detect, and/or determine, based on the sensor information, each object of the plurality of objects within the FOV. For example, each object of the plurality of objects within the FOV may satisfy a perception threshold that indicates an amount of an object sensed by the sensing device that is occluded by an item.

The on-board computing device113may generate and/or determine the trajectory for a vehicle and a respective trajectory for each object of a plurality of objects within the FOV of the sensing device based on the sensor information. For example, for the vehicle and each object of the plurality of objects, the sensor information may indicate a respective position and a respective velocity. According to some aspects, the respective position and the respective velocity may be determined and/or identified by a perception system, received from a source (e.g., the remote computing device110, etc.), and/or received from a ground truth human labeling system with offline processing. Generating and/or determining the trajectory for the vehicle and the respective trajectories for each object of the plurality of objects may be based on the respective positions and the respective velocities. According to some aspects, the trajectory for the vehicle and the respective trajectories for each object of the plurality of objects may include constraints based on an object or actor type, such as maximum speed, maximum acceleration, maximum jerk, turning radius, etc.

In1320, on-board computing device113identifies and/or determines objects of the plurality of objects with trajectories that intersect the trajectory for the vehicle. The on-board computing device113may identify and/or determine objects of the plurality of objects with trajectories that intersect the trajectory for the vehicle based on the respective trajectories for each object of the plurality of objects.

In1330, on-board computing device113removes, from the objects with trajectories that intersect the trajectory for the vehicle, objects with trajectories that at least one of exit the FOV or intersect with other objects of the plurality of objects within the FOV.

In1340, on-board computing device113selects and/or determines, from remaining objects with trajectories that intersect the trajectory for the vehicle, objects with trajectories that indicate a respective collision between the object and the vehicle.

In1350, on-board computing device113assigns and/or determines, for each object of the objects with the trajectories that indicate the respective collision between the object and the vehicle, a severity of the respective collision. Assigning and/or determining, for each object of the objects with the trajectories that indicate the respective collision, the severity of the respective collision may include inputting, into a predictive model, a velocity of the vehicle, a velocity for the object, and an object type for the object. The on-board computing device113may receive from the predictive model, based on the velocity of the vehicle, a position of the vehicle, the velocity for the object, and the object type, an indication of the severity of the respective collision.

According to some aspects, method1300may further include the on-board computing device113causing, for each object of the objects with the trajectories that indicate the respective collision between the object and the vehicle, display of the respective trajectory and the trajectory for the vehicle. According to some aspects, method1300may further include the on-board computing device113sending, to a user device, an indication of the severity of the respective collision for each object of the objects with the trajectories that indicate the respective collision between the object and the vehicle.

According to some aspects, method1300may further include the on-board computing device113causing the vehicle to perform a driving maneuver. The on-board computing device113may cause the vehicle to perform a driving maneuver based on the severity of the respective collision for at least one object of the objects with the trajectories that indicate the respective collision. For example, on-board computing device113may cause the vehicle to brake, turn, accelerate, decelerate, and/or execute any other maneuver to avoid the respective collision when the severity of the respective collision satisfies a threshold. According to some aspects, the on-board computing device113may cause the vehicle to brake, turn, accelerate, decelerate, and/or execute any other maneuver to avoid the respective collision when the severity of the respective collision satisfies a threshold while also causing the vehicle to maneuver in a manner that avoids another collision.

Various embodiments can be implemented, for example, using one or more computer systems, such as computer system1400shown inFIG.14. Computer system1400can be any computer capable of performing the functions described herein.

Computer system1400can be any well-known computer capable of performing the functions described herein. According to some aspects, the on-board computing device113ofFIG.1(and/or any other device/component described herein) may be implemented using the computer system1400. According to some aspects, the computer system1400may be used and/or specifically configured to implement methods1200and1300.

Computer system1400includes one or more processors (also called central processing units, or CPUs), such as a processor1404. Processor1404is connected to a communication infrastructure or bus1406.

One or more processors1404may each be a graphics processing unit (GPU). In an embodiment, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

Computer system1400also includes user input/output device(s)1403, such as monitors, keyboards, pointing devices, etc., that communicate with communication infrastructure1406through user input/output interface(s)1402.

Computer system1400also includes a main or primary memory1408, such as random access memory (RAM). Main memory1408may include one or more levels of cache. Main memory1408has stored therein control logic (i.e., computer software) and/or data.

Computer system1400may also include one or more secondary storage devices or memory1410. Secondary memory1410may include, for example, a hard disk drive1412and/or a removable storage device or drive1414. Removable storage drive1414may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, a tape backup device, and/or any other storage device/drive.

Removable storage drive1414may interact with a removable storage unit1418. Removable storage unit1418includes a computer-usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit1418may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive1414reads from and/or writes to removable storage unit1418in a well-known manner.

According to an exemplary embodiment, secondary memory1410may include other means, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system1400. Such means, instrumentalities or other approaches may include, for example, a removable storage unit1422and an interface1420. Examples of the removable storage unit1422and the interface1420may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Computer system1400may further include a communication or network interface1424. Communication interface1424enables computer system1400to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (individually and collectively referenced by reference number1428). For example, communication interface1424may allow computer system1400to communicate with remote devices1428over communications path1426, which may be wired and/or wireless, and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system1400via communication path1426.

In an embodiment, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system1400, main memory1408, secondary memory1410, and removable storage units1418and1422, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system1400), causes such data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparent to persons skilled in the relevant art(s) how to make and use embodiments of this disclosure using data processing devices, computer systems, and/or computer architectures other than that shown inFIG.14. In particular, embodiments can operate with software, hardware, and/or operating system implementations other than those described herein.

It is to be appreciated that the Detailed Description section, and not any other section, is intended to be used to interpret the claims. Other sections can set forth one or more but not all exemplary embodiments as contemplated by the inventor(s), and thus, are not intended to limit this disclosure or the appended claims in any way.

While this disclosure describes exemplary embodiments for exemplary fields and applications, it should be understood that the disclosure is not limited thereto. Other embodiments and modifications thereto are possible, and are within the scope and spirit of this disclosure. For example, and without limiting the generality of this paragraph, embodiments are not limited to the software, hardware, firmware, and/or entities illustrated in the figures and/or described herein. Further, embodiments (whether or not explicitly described herein) have significant utility to fields and applications beyond the examples described herein.

Embodiments have been described herein with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined as long as the specified functions and relationships (or equivalents thereof) are appropriately performed. Also, alternative embodiments can perform functional blocks, steps, operations, methods, etc. using orderings different than those described herein.

References herein to “one embodiment,” “an embodiment,” “an example embodiment,” or similar phrases, indicate that the embodiment described can include a particular feature, structure, or characteristic, but every embodiment can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it would be within the knowledge of persons skilled in the relevant art(s) to incorporate such feature, structure, or characteristic into other embodiments whether or not explicitly mentioned or described herein. Additionally, some embodiments can be described using the expression “coupled” and “connected” along with their derivatives. These terms are not necessarily intended as synonyms for each other. For example, some embodiments can be described using the terms “connected” and/or “coupled” to indicate that two or more elements are in direct physical or electrical contact with each other. The term “coupled,” however, can also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other.

The breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.