DETERMINING PERCEPTION ZONES FOR OBJECT DETECTION IN AUTONOMOUS SYSTEMS AND APPLICATIONS

In various examples, techniques for determining perception zones for object detection are described. For instance, a system may use a dynamic model associated with an ego-machine, a dynamic model associated with an object, and one or more possible interactions between the ego-machine and the object to determine a perception zone. The system may then perform one or more processes using the perception zone. For instance, if the system is validating a perception system of the ego-machine, the system may determine whether a detection error associated with the object is a safety-critical error based on whether the object is located within the perception zone. Additionally, if the system is executing within the ego-machine, the system may determine whether the object is a safety-critical object based on whether the object is located within the perception zone.

BACKGROUND

Vehicles, such as autonomous vehicles or semi-autonomous vehicles, use perception systems to process sensor data from sensors of the vehicles in order to detect objects within environments for which the vehicles are navigating. Downstream systems of the vehicles, such as planning systems and/or control systems, then use the locations of the objects to determine control operations for the vehicles. As such, when errors occur with the perception systems, the downstream systems may be affected. As a first example, if a perception system of a vehicle does not detect an object that is located along a path of the vehicle, a planning system and/or a control system may cause the vehicle to continue navigating along the path until, for example, another system (e.g., a collision or obstacle avoidance system) is activated to come to a stop or avoid the object. This may cause extend travel times or result in sudden maneuvers that may be uncomfortable for passengers of the vehicle. As a second example, if a perception system of a vehicle inaccurately determines that an object is located along a path of the vehicle where no object is present, a planning system and/or a control system may cause the vehicle to navigate in reliance or with respect to the presence of the object—such as to slow down or come to a stop. As such, it is critical that the perception systems of the vehicles are reliable in detecting objects and, as a result, the perception systems of the vehicles must generally satisfy stringent safety requirements via rigorous verification and validation (V&V) regimes.

Various metrics have been created to evaluate the performance of perception systems. For example, evaluation metrics, such as Intersection over Union (IoU) and False Positive (FP) rates, are task-agnostic and provide comparability across a variety of benchmarks. For instance, these task-agnostic evaluation metrics determine the error rates of perception systems. However, such task-agnostic metrics do not adequately quantify how well a perception system will actually perform when integrated into a full autonomy stack and deployed into the real-world. This is because the type of misdetection may lead to very different behaviors in downstream tasks. For example, it has been shown that there is a linear degradation in the performance of task-aware evaluation metrics the further away an object is from a vehicle, therefore indicating that the task-aware evaluation metrics may not be sufficient or as effective as desired in validating safety.

Because of this, task-aware metrics have been used when evaluating perception systems. For instance, one example of a planning-aware metric (e.g., a task-aware metric) uses KL-divergence to compare how different the vehicle's plans are with noisy and with perfect detection. Additionally, another task-aware metric proposes combining scores measuring detection quality, collision potential, and time needed to make the detection. Furthermore, other task-aware metrics rank an object based on the object's perceived or simulated risks (imminent, potential, none) of a collision or other incident, as defined using a simplified forward reachable set of computations under an isotropic force assumption. While such task-aware metrics are useful in comparing the relative performance of a perception system over another perception system, the task-aware metrics are not useful in validating whether a perception system is sufficient in supporting safe vehicle operations.

Additionally, safety-critical perception error validation may include demonstrating that a perception system can operate within an acceptable risk level specified by the appropriate regulatory body or industry safety standard. The safety-critical perception error rate is the rate of the perception errors multiplied by the fraction of safety-critical perception errors. However, for safety-critical perception error validation, it is unclear and non-trivial how to compute a value for the fraction of the safety-critical errors. One conventional technique to determine such safety-critical errors is to determine that a perception error is unsafe when a simulated collision or other incident could have been prevented if the error had not occurred. However, precisely determining safety-critical errors using such a technique is challenging, because obtaining ground truth data is often difficult and/or time intensive.

Yet another conventional technique to determine safety-critical errors is to determine which objects in the environment are safety-critical and ensure that perception performance is high for those objects. For example, one approach includes determining that an object is safety-critical if the vehicle and the object would still collide before coming to a stop when braking. Another approach includes determining that an object is safety-critical if the vehicle and the object would collide when the vehicle and the object continue moving with a constant velocity. While these approaches consider the object's dynamics, they nonetheless make assumptions about the object's behavior, which the vehicle would not typically have control over. Furthermore, these approaches do not account for possible reactions of the vehicle and/or the object.

Other approaches to determine safety-critical objects include determining that all objects located within a radius around the vehicle are safety-critical objects. However, these approaches may still ignore safety-critical objects that are located outside of the radius, such as an object with a high velocity that poses a risk to the vehicle (e.g., is moving in a direction of the vehicle). Additionally, these approaches may determine that objects that provide little to no risk to the vehicle are safety-critical objects, such as an object that is located within the radius but moving in a high velocity away from the vehicle. Because these approaches do not account for the object's dynamics and/or the vehicle's dynamics, implementations according to these approaches can suffer from inaccuracies—some of which may be severe and/or safety critical.

SUMMARY

Embodiments of the present disclosure relate to techniques for determining perception zones for object detection. Systems and methods are disclosed that use a dynamic model of an ego-machine (e.g., the ego-machine's dynamics, the ego-machine's potential behavior, etc.), a dynamic model for an object (e.g., the object's dynamics, the object's potential behavior, etc.), and one or more possible interactions (e.g., all possible interactions) between the ego-machine and the object to determine a zone (also referred to as a “perception zone”) associated with the vehicle-object interaction. The techniques further include determining that the object is safety-critical (and/or an error is a safety-critical error) when the object is located within the perception zone or determining that the object is not safety-critical (and/or the error is not the safety-critical error) when the object is located outside of the perception zone. As such, the current techniques are able to generate perception zones that are sufficiently large to capture all safety-critical objects while still omitting objects that are not safety-critical to the ego-machine.

Because of this, the systems of the present disclosure provide improvements over conventional systems (such as those described above) that merely use an object's dynamics and/or an assumed behavior for the object. For example, by using the dynamic model for the ego-machine, the dynamic model for the object, and the one or more possible interactions (e.g., all possible interactions) between the ego-machine and the object, the described systems determine a perception zone that captures the object whenever the object is safety critical, unlike conventional systems that evaluate object behavior independent from and/or without consideration of the behavior of the ego-machine. Additionally, the current systems provide improvements over conventional systems that use a radius around the ego-machine to determine the safety-critical objects. For example, and as discussed above, such conventional systems may wrongly identify objects located within the radius as being safety-critical and/or may wrongly identify objects located outside of the radius as not being safety-critical. In contrast, the described systems determine a perception zone using the dynamic models for the ego-machine and the object, thus taking into consideration the dynamics and potential behaviors of the ego-machine and/or the object when determining whether the object is safety-critical with respect to the ego-machine.

DETAILED DESCRIPTION

Systems and methods are disclosed related to techniques for determining perception zones for object detection. Although the present disclosure may be described with respect to an example autonomous vehicle900(alternatively referred to herein as “vehicle900” or “ego-machine900,” an example of which is described with respect toFIGS.9A-9D), this is not intended to be limiting. For example, the systems and methods described herein may be used by, without limitation, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), piloted and un-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater craft, drones, and/or other vehicle types. In addition, although the present disclosure may be described with respect to object detection, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and/or any other technology spaces where object detection may be used.

For instance, a system (which may correspond to the ego-machine, may be included as part of a simulation and/or testing environment, etc.) may use a perception system of the ego-machine to process sensor data in order to detect an object within an environment. The system may then determine one or more parameters for the object. As described herein, the parameter(s) for the object may include, but is not limited to, a type (e.g., a vehicle, a pedestrian, a scooter, etc.) of the object, a location of the object, a velocity of the object (e.g., a current velocity, a maximum velocity, etc.), an acceleration of the object, a deceleration of the object, a size of the object, a direction of travel of the object, steering limits (e.g., a turning radius) for the object, and/or any other parameter. In some examples, the system determines one or more of the parameters based on further processing the sensor data. Additionally, or alternatively, in some examples, the system determines one or more of the parameters as a pre-programmed parameter(s). For example, the system may be pre-programmed with a parameter(s) for different types of objects, such as the acceleration, the steering limits, and/or the like.

The system may further determine one or more parameters associated with the ego-machine. As described herein, the parameter(s) for the ego-machine may include, but is not limited to, a location of the ego-machine, a velocity of the ego-machine (e.g., a current velocity), a deceleration of the ego-machine, a time period for the ego-machine to begin decelerating, a size of the ego-machine, a direction of travel of the ego-machine, steering limits (e.g., a turning radius) for the ego-machine, and/or any other parameter. In some examples, the system determines one or more of the parameters based on sensor data. Additionally, or alternatively, in some examples, the system determines one or more of the parameters as a pre-programmed parameter(s). For example, the system may be pre-programmed with the parameter(s) for the deceleration and/or the time period for the ego-machine to begin decelerating.

In order to be conservative and maximize the safety of the ego-machine and/or the object, the system may then use, in addition to the parameters, one or more assumptions when determining a perception zone for the object. For instance, in some examples, the system may use a first assumption that the ego-machine and the object will actively attempt to steer toward one another. For example, the system may determine that the ego-machine will turn in a direction(s) toward the object and that the object may turn in a direction(s) toward the ego-machine while, e.g., at the same time accelerating. In some examples, the system may use a second assumption that no obstacles are located between the ego-machine and the object and/or that the ego-machine and the object are navigating along a flat road. This way, the ego-machine and the object are able to navigate using the shortest path, which may increase the probability of a simulated collision.

The system may then use the parameter(s) for the ego-machine, the parameter(s) for the object, and the assumption(s) (which may together represent the dynamic models for the ego-machine and the object) to determine the perception zone for the object. In some examples, the system determines the perception zone using one or more reachability techniques, such as Hamilton-Jacobi (HJ) reachability, forward reachability, backward reachability, sampling-based reachability, a neural network(s), and/or the like. For an example of determining the perception zone, the system may determine a possible path(s) that the ego-vehicle may navigate in order to try and collide with the object (e.g., determine a path(s) for each interaction that may cause a collision between the ego-machine and the object). For instance, if the object is located along a direction of travel of the ego-machine, then the ego-machine may continue along the current path and/or change paths based on the steering radius of the ego-machine. This is because the ego-machine may still collide with the object using any of these paths since the object may continue along the object's current path towards the ego-machine and/or change paths to avoid the collision with the ego-machine. As such, by turning the ego-machine based on the steering radius, the ego-machine may still collide with the object even if the object attempts to navigate away from the ego-machine.

The system may also determine a distance(s) for the path(s) of the ego-machine. To determine the distance(s), the system may use the time period for the ego-machine to begin decelerating, the deceleration of the ego-machine, the velocity of the object, the acceleration of the object, and/or one or more other parameters. For instance, the system may assume that the ego-machine will attempt to immediately stop once the object is detected. As such, a distance of a path may be determined based at least on the current velocity of the ego-machine, the time period for the ego-machine to begin decelerating, and the deceleration of the ego-machine. The system may then use similar processes to determine a respective distance for each of the one or more paths. Additionally, the system may use the determined path(s), along with the distance(s) for the path(s), to determine the perception zone for the object. For example, the perception zone may include a region within the environment that includes one or more (e.g., all) of the possible paths that the ego-machine may navigate while attempting to collide with the object.

The perception zones may be modeled in this way—e.g., by simulating the worst-case scenarios using unrealistic assumptions (e.g., a flat ground assumption, absence of any intervening objects between the object and the ego-machine, ego-machine and object accelerate at one another as fast as possible, etc.)— in order to ensure that the perception zones account for all objects that, even if a real-world environment were to match these unrealistic assumptions used to model the simulation, the perception zone of the ego-machine would still account for these objects. In this way, the perception zone of the ego-machine may be robust to not only more likely real-world scenarios, but also to corner cases that are less likely to occur naturally in the real-world.

The system may then use the perception zone for performing various processes. For a first example, such as when the system is determining the performance of the perception system of the ego-machine, the system may use the perception zone to determine whether a detection error associated with the object is a safety-critical error. For instance, and as discussed above, when testing the perception system, the perception system may detect an object (which may be referred to as a “ghost object”) that is not located within the environment. As such, it is important to determine whether this detection error is safety-critical to the ego-machine. To determine whether the detection error is safety-critical, the system may determine whether the object is located within the perception zone. If the system determines that the object is located within the perception zone, then the system may determine that the detection error is a safety-critical error (e.g., a safety violation)—e.g., because a potential collision could occur before the ego-machine is able to come to a stop and/or an unintended deceleration associated with the vehicle because of the object. However, if the system determines that the object is located outside of the perception zone, then the system may determine that the detection error is not a safety-critical error—e.g., because there is no potential collision between the ego-machine and the object before the ego-machine is able to stop.

For a second example, such as when the system is deployed in an ego-machine that is navigating around a real-world environment, the system may use the perception zone to determine whether a detected object is a safety-critical object. For instance, while navigating, the ego-machine may detect the object using the perception system. The system may then perform the processes described herein to determine the perception zone for the object. Additionally, the system may determine whether the object is located within the perception zone. If the system determines that the object is located within the perception zone, then the system may determine that the object is a safety-critical object—e.g., that the object should be accounted for in making planning and/or control decisions. However, if the system determines that the object is located outside of the perception zone, then the system may determine that the object is not a safety-critical object—e.g., that the object may be ignored at least with respect to collision or obstacle avoidance, but may still be accounted for in planning and/or control decisions.

The systems and methods described herein may be used by, without limitation, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), piloted and un-piloted robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, underwater craft, drones, and/or other vehicle types. Further, the systems and methods described herein may be used for a variety of purposes, by way of example and without limitation, for machine control, machine locomotion, machine driving, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, real-time streaming, generating or presenting virtual reality (VR) content, generating or presenting augmented reality (AR) content, generating or presenting mixed reality (MR) content, autonomous or semi-autonomous machine applications, deep learning, environment simulation, object or actor simulation and/or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray-tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing and/or any other suitable applications.

Disclosed embodiments may be comprised in a variety of different systems such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, aerial systems, medial systems, boating systems, smart area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twin operations, systems for performing real-time streaming, systems for generating VR, AR, or MR content, systems for presenting VR, AR, or MR content, systems implemented using an edge device, systems incorporating one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems implemented at least partially using cloud computing resources, and/or other types of systems.

In some examples, the process100may be performed by one or more systems of an ego-machine. For example, and as discussed herein, the process100may be performed by the ego-machine during normal operation to determine whether an object(s) located within an environment is a safety-critical object(s). In some examples, the process100may be performed by one or more systems associated with a simulation and/or verification system. For example, the process100may be performed by a verification system in order to determine the performance of a perception system of the ego-machine. Both of these scenarios are described in more detail with respect to the output from an object-analysis component102ofFIG.1.

The process100may include a parameter detector104of the object-analysis component102that determines one or more parameters106associated with the ego-machine and/or one or more parameters106associated with an object. In some examples, the parameter detector104may determine a parameter(s)108(also referred to as a37detected parameter(s)108) associated with the ego-machine and/or a parameter(s)108associated with the object(s) using sensor data110. In embodiments where the sensor data110includes image data, the image data may include data representative of images depicting one or more fields of view of one or more cameras (e.g., image sensors) of ego-machine, such as stereo camera(s), wide-view camera(s) (e.g., fisheye cameras), infrared camera(s), surround camera(s) (e.g., 360 degree cameras), long-range and/or mid-range camera(s), and/or other camera type of the autonomous vehicle. In some examples, the image data may be captured in one format (e.g., RCCB, RCCC, RBGC, etc.), and then converted (e.g., during pre-processing of the image data) to another format. In some other examples, the image data may be provided as input to a sensor data pre-processor (not shown) to generate pre-processed image data (discussed herein). Many types of images or formats may be used as inputs; for example, compressed images such as in Joint Photographic Experts Group (JPEG), Red Green Blue (RGB), or Luminance/Chrominance (YUV) formats, compressed images as frames stemming from a compressed video format such as H.264/Advanced Video Coding (AVC) or H.265/High Efficiency Video Coding (HEVC), raw images such as originating from Red Clear Blue (RCCB), Red Clear (RCCC) or other type of imaging sensor.

In some examples, before processing the sensor data110, a sensor data pre-processor may use image data representative of one or more images (or other data representations) and load the sensor data110into memory in the form of a multi-dimensional array/matrix (alternatively referred to as tensor, or more specifically an input tensor, in some examples). The array size may be computed and/or represented as W×H×C, where W stands for the image width in pixels, H stands for the height in pixels, and C stands for the number of color channels. Without loss of generality, other types and orderings of input image components are also possible. Additionally, the batch size B may be used as a dimension (e.g., an additional fourth dimension) when batching is used. Batching may be used for training and/or for inference. Thus, the input tensor may represent an array of dimension W×H×C×B. Any ordering of the dimensions may be possible, which may depend on the particular hardware and software used to implement the sensor data pre-processor. This ordering may be chosen to maximize training and/or inference performance of the parameter detector104.

In some embodiments, a pre-processing image pipeline may be employed by the sensor data pre-processor to process a raw image(s) acquired by a sensor(s) (e.g., image sensor(s)) and included in the image data to produce pre-processed image data which may represent an input image(s) to the input layer(s) of the parameter detector104. An example of a suitable pre-processing image pipeline may use a raw RCCB Bayer (e.g., 1-channel) type of image from the sensor102and convert that image to a RCB (e.g., 3-channel) planar image stored in Fixed Precision (e.g., 16-bit-per-channel) format. The pre-processing image pipeline may include decompanding, noise reduction, demosaicing, white balancing, histogram computing, and/or adaptive global tone mapping (e.g., in that order, or in an alternative order).

Where noise reduction is employed by the sensor data pre-processor, it may include bilateral denoising in the Bayer domain. Where demosaicing is employed by the sensor data pre-processor, it may include bilinear interpolation. Where histogram computing is employed by the sensor data pre-processor, it may involve computing a histogram for the C channel, and may be merged with the decompanding or noise reduction in some examples. Where adaptive global tone mapping is employed by the sensor data pre-processor, it may include performing an adaptive gamma-log transform. This may include calculating a histogram, getting a mid-tone level, and/or estimating a maximum luminance with the mid-tone level.

As described herein, the parameter(s)106for the object may include, but is not limited to, a type (e.g., a vehicle, a pedestrian, a scooter, etc.) of the object, a location of the object, a velocity of the object (e.g., a current velocity, a maximum velocity, etc.), an acceleration of the object, a deceleration of the object, a size of the object, a direction of travel of the object, steering limits (e.g., a turning radius) for the object, and/or any other parameter106. Additionally, as described herein, the parameter(s)106for the ego-machine may include, but is not limited to, a location of the ego-machine, a velocity of the ego-machine (e.g., a current velocity), a deceleration of the ego-machine, a time period for the ego-machine to begin decelerating, a size of the ego-machine, a front-rear axle distance of the ego-machine, a direction of travel of the ego-machine, steering limits (e.g., a turning radius) for the ego-machine, and/or any other parameter106.

The process100may also include the parameter detector104receiving and/or generating a parameter(s)112(also referred to as a “pre-programmed parameter(s)112”) associated with the ego-machine and/or a parameter(s)112associated with the object. For instance, the parameters112for the object may include the acceleration of the object, the deceleration of the object, the maximum velocity of the object, and/or any other of the parameters106. For example, the parameters112may indicate that the deceleration for the object is −4.5 ms−2, the acceleration for the object is 4.5 ms−2, and the maximum velocity for the object is 20 ms−1, although other decelerations, accelerations, and/or maximum velocities may be used. In some examples, the parameter(s)112for the object may be based on the type of object. For example, the parameters112may include a first acceleration and/or a first maximum velocity for vehicles and a second acceleration and/or a second maximum velocity for pedestrians.

Additionally, with regard to the ego-machine, the parameter(s)112may include the deceleration of the ego-machine, the time period for the ego-machine to begin decelerating, the steering limits of the ego-machine, the size of the ego-machine, and/or the front-rear axle distance of the ego-machine. For example, the parameter(s)112for the ego-machine may indicate that the deceleration is −3.5 ms−2, the ego-machine steering limits is a range between −10° and 10°, the ego-machine size is 4.5 m×2.5 m, the front-rear axle distance is 3 m, and the time period for the ego-machine to begin decelerating is 0.5 seconds, although other decelerations, steering limits, ego-machine sizes, front-rear axle distances, and/or time periods for the ego-machine to begin decelerating may be used.

The process100may also include receiving and/or generating one or more assumptions114associated with the ego-machine and/or the object. For instance, and as described herein, in order to be conservative and maximize the safety of the ego-machine and/or the object, the object-analysis component102may use, in addition to the parameters106, the assumption(s)114when determining a perception zone116for the object. For instance, in some examples, a first assumption114may indicate that the ego-machine and the object will actively attempt to steer toward one another. For example, based on this first assumption114, the object-analysis component102may determine that the ego-machine will turn in a direction(s) toward the object and that the object may turn in a direction(s) toward the ego-machine while at the same time accelerating. In some examples, a second assumption114may indicate that no obstacles are located between the ego-machine and the object and/or that the ego-machine and the object are navigating along a flat road. This way, the ego-machine and the object are able to navigate directly at one another using the shortest path, which may increase the simulated probability of collision.

For instance,FIG.2Aillustrates an example of parameters106associated with an ego-machine202and objects204(1)-(2) (also referred to singularly as “object204” or in plural as “objects204”) located within an environment206, in accordance with some embodiments of the present disclosure. In the example ofFIG.2A, the environment206may include a real-world environment for which the ego-machine202is navigating or a simulated environment for which the ego-machine202is navigating. Additionally, while the example ofFIG.2Aillustrates the objects204as including vehicles, in other examples, one or more of the objects204may include a different type of object (e.g., a person, a motorcycle, an animal, a building, a sign, etc.).

With reference toFIG.1, a perception system118of the ego-machine202may analyze sensor data110to detect the objects204within the environment206. In some examples, such as when the ego-machine202is navigating within a real-world environment, the perception system118may be included within and/or communicate with the object-analysis component102. In other examples, such as when the ego-machine202is navigating within a simulated environment, the perception system118may be separate from the object-analysis component102. For instance, in such examples, a system may be testing the perception system118in order to determine a performance of the perception system118. In either of the examples, the perception system118may output data120associated with the objects204. For example, and for an object204, the data120may indicate a type of the object204, a location of the object204, and/or any other information associated with the object204.

The parameter detector104may determine the parameters106associated with the ego-machine202. The parameters106may include at least a location208of the ego-machine202(as indicated by the dashed lines), a velocity of the ego-machine202(e.g., a current velocity), a deceleration of the ego-machine202, a time period for the ego-machine202to begin decelerating, a size of the ego-machine202, front-rear axle distance of the ego-machine202, a direction of travel210of the ego-machine202, steering limits212(e.g., a turning radius) for the ego-machine202, and/or any other parameter106. As described herein, in some examples, one or more of the parameters106may include a detected parameter(s)108and/or one or more of the parameters106may include a pre-programmed parameter(s)112.

FIG.2Billustrates an example of assumptions114associated with the ego-machine202and the objects204located within the environment206, in accordance with some embodiments of the present disclosure. As described herein, in some examples, a first assumption114may indicate that the ego-machine202and the objects204will actively attempt to steer toward one another. As such,FIG.2Billustrates that the ego-machine202may take paths220(1) (which are represented by the large-dashed lines) and the object204(1) may take paths220(2) (which are also represented by the large-dashed lines) to actively attempt to steer toward one another. In some examples, the paths220(1) are based on the location208, the direction of travel210, the steering limits212, and/or the velocity of the ego-machine202. Additionally, the paths220(2) are based on the location214(1), the driving direction216(1), the steering limits218(1), and/or the velocity of the object204(1).

Additionally,FIG.2Billustrates that the ego-machine202may take paths222(1) (which are represented by the short-dashed lines) and the object204(2) may take paths222(2) (which are also represented by the short-dashed lines) to actively attempt to steer toward one another. In some examples, the paths222(1) are based on the location208, the direction of travel210, the steering limits212, and/or the velocity of the ego-machine202. Additionally, the paths222(2) are based on the location214(2), the driving direction216(2), the steering limits218(2), and/or the velocity of the object204(2).

As further described herein, a second assumption114may indicate that no obstacles are located between the ego-machine202and the objects204and/or that the ego-machine202and the object204are navigating along a flat road. As such, in the example ofFIG.2B, because of the second assumption114, the ego-machine202may continue along the paths220(1) and the object204(1) may continue along the paths220(2) as if obstacles224(1)-(3) are not located within the environment206, which is illustrated by the light shading of the obstacle224(1) and the dotted lines of the obstacles224(2)-(3). Additionally, because of the second assumption114, the ego-machine202may continue along the paths222(1) and the object204(2) may continue along the paths222(2) as if an obstacle226is not located within the environment206, which is illustrated by the light shading of the obstacle226. As described herein, the assumptions114may be used in order to be conservative and maximize the safety of the ego-machine202and/or the object204.

Referring back toFIG.1, a zone generator122of the object-analysis component102may use at least the parameters106and the assumption(s)114to determine a perception zone(s)116for an object(s) within the environment. In some examples, the zone generator122determines the perception zone(s)116using one or more reachability techniques, such as Hamilton-Jacobi (HJ) reachability. For an example of determining a perception zone116, and for the object204(1) ofFIGS.2A-2B, the zone generator122may determine the possible paths220(1) that the ego-machine202may navigate in order to try and steer toward the object204(1). As described herein, since the zone generator122may use one or more possible interactions (e.g., all possible interactions) between the ego-machine202and the object204(1), the zone generator122may determine the paths220(1) of the ego-machine202based on the paths220(2) of the object204(2).

For instance, and as shown by the example ofFIG.2B, the paths220(1) of the ego-machine202may include a first path220(1) where the ego-machine202continues along a straight path (e.g., along the direction of travel210), a second path220(1) where the ego-machine202turns left according to the maximum steering limit212of the ego-machine202, and multiple other paths220(1) that are between the first path220(1) and the second path220(1). This is because the paths220(2) of the object204(1) include a first path220(2) where the object204(1) continues along a straight path (e.g., along the direction of travel216(1)), a second path220(2) where the object204(1) turns right according to the maximum steering limit218(1) of the object204(1), and multiple other paths220(2) that are between the first path220(2) and the second path220(2). As such, the ego-machine202is able to steer toward the object204(1) (and thus simulate an intent to collide with the object204(1)) when taking one or more (e.g., each of) the paths220(1).

The zone generator122may also determine distances for the paths220(1) of the ego-machine202. To determine the distances, the zone generator122may use the time period for the ego-machine202to begin decelerating, the deceleration of the ego-machine202, the velocity of the object204(1), the acceleration of the object204(1), and/or one or more additional parameters106. For instance, the zone generator122may assume that the ego-machine202will attempt to immediately stop once the object204(1) is detected. As such, a distance along a path220(1) may be determined based at least on the current velocity of the ego-machine202, the time period for the ego-machine202to begin decelerating, the deceleration of the ego-machine202, the velocity of the object204(1), and the acceleration of the object204(1). The zone generator122may then use similar processes to determine a respective distance for one or more other paths220(1) (e.g., each of the paths220(1)).

The zone generator122may use one or more of the determined paths220(1), along with the distance(s) for the determined path(s)220(1), to determine the perception zone116for the object204(1). For instance,FIG.3illustrates an example of perception zones302(1)-(2) (also referred to singularly as “perception zone302” or in plural as “perception zones302”) for the objects204, in accordance with some embodiments of the present disclosure. As shown, the perception zone302(1) (which may represent, and/or include, one of the perception zone(s)116) may include a region within the environment206that includes one or more of the paths220(1) (e.g., all of the paths220(1)) that the ego-machine202may navigate while attempting to collide with the object204(1).

The zone generator122may also determine distances for the paths222(1) of the ego-machine202. To determine the distances, the zone generator122may use the time period for the ego-machine202to begin decelerating, the deceleration of the ego-machine202, the velocity of the object204(2), the acceleration of the object204(2), and/or one or more additional parameters106. For instance, the zone generator122may assume that the ego-machine202will attempt to immediately stop (e.g., taking into account a delay in performing the actuation) once the object204(2) is detected. As such, a distance along a path222(1) may be determined based at least on the current velocity of the ego-machine202, the time period for the ego-machine202to begin decelerating, the deceleration of the ego-machine202, the velocity of the object204(2), and the acceleration of the object204(2). The zone generator122may then use similar processes to determine a respective distance for one or more other paths222(1) (e.g., each of the paths222(1)).

The zone generator122may use one or more of the determined paths222(1), along with the distance(s) for the determined path(s)222(1), to determine the perception zone302(2) (which may represent, and/or include, one of the perception zone(s)116) for the object204(2). For instance, and as shown, the perception zone302(2) may include a region within the environment206that includes one or more of the paths222(1) (e.g., all of the paths222(1)) that the ego-machine202may navigate while attempting to collide with the object204(2). While the example ofFIG.3illustrates the zone generator122as determining the perception zones302for the objects204, in other examples, the zone generator122may determine one or more additional perception zones116for one or more other objects located within the environment206.

While the examples above describe the zone generator122as generating the perception zones using one or more reachability techniques, in other examples, the zone generator122may generate the perception zones using one or more additional and/or alternative techniques. For instance, the zone generator122may generate the perception zones using a dynamic programming and/or partial differential equation technique(s) (e.g., the one or more reachability techniques, one or more reinforcement learning techniques, etc.), a data-driven technique(s) (e.g., using one or more neural networks, which are described in more detail with regard toFIGS.8A-8B), a sampling-based technique(s) (e.g., one or more Monte-Carlo Simulations, etc.), a geometric reachable set approximation technique(s) (e.g., one or more Zonotopes, etc.), and/or any other technique.

Referring back toFIG.1, the process100may include a zone analyzer124of the object-analysis component102determining whether an object(s) associated with a perception zone(s)116is a safety-critical object(s). In some examples, the zone analyzer124may determine that an object is a safety-critical object based on the object being located within the perception zone116and determine that the object is not a safety-critical based on the object being located outside of the perception zone116. In such examples, the zone analyzer124uses such a technique based on an assumption114that the ego-machine is not at fault for a simulated collision at a time that the ego-machine stops.

For instance, and referring back toFIG.3, the zone analyzer124may determine that the current location214(1) of the object204(1) is within the perception zone302(1). As such, the zone analyzer124may determine that the object204(1) is a safety-critical object. This is because, in the simulation, a collision is possible between the ego-machine202and the object204(1), e.g., before the ego-machine202is able to stop. The zone analyzer124may also determine that the current location214(2) of the object204(2) is outside of the perception zone302(2). As such, the zone analyzer124may determine that the object204(2) is not a safety-critical object—e.g., because there is no possibility of collision between the ego-machine202and the object204(2) before the ego-machine202is able to stop.

As another illustration of using reachability to determine whether the objects204are safety-critical objects,FIG.4illustrates an example of the ego-machine202and the objects204attempting to steer toward one another while the ego-machine202is attempting to stop, in accordance with examples of the present disclosure. As shown,FIG.4illustrates a stopping zone402(1) associated with the object204(1) and a stopping zone402(2) associated with the object204(2). In some examples, the stopping zones402(1) represents the region that the ego-machine202will travel along the paths220(1) and the stopping zone402(2) represents the region that the ego-machine202will travel along the path222(1) based on the velocity of the ego-machine202, the deceleration of the ego-machine202, and the time period for the ego-machine202to begin decelerating.

As shown, there is a possibility of collision between the ego-machine202and the object204(1) before the time that the ego-machine202stops. This is based on a possible path404(1) (which may represent, and/or include, one of the paths220(2)) of the object204(1). As shown, based on the possible path404(1), a future location406(1) of the object204(1) at a time before the ego-machine202stops is within the stopping zone402(1). As such, within this simulation, there is a possibility of collision between the ego-machine202and the object204(1) while the ego-machine202is stopping.

As also shown, there is no probability of collision between the ego-machine202and the object204(2) before the time that the ego-machine202stops. This is based on a possible path404(2) (which may represent, and/or include, one of the paths222(2)) of the object204(2). As shown, based on the possible path404(2), a future location406(2) of the object204(2) at a time that the ego-machine202stops is outside of the stopping zone402(2). As such, there is no possibility of collision between the ego-machine202and the object204(2) while the ego-machine202is stopping.

As described herein, the object-analysis component102may use reachability, such as HJ reachability, to determine a perception zone(s)116and/or whether an object(s) is a safety-critical object(s). For example, the object-analysis component102may set target states L which the ego-machine and an object will seek or avoid within a time horizon T. For example, L may correspond to the set of joint system states where the ego-machine E and the object C are in collision. The output of the reachability computation is then a set of initial states (termed the backwards reachable tube (BRT)) where membership denotes that it is possible for a simulated collision between the ego-machine E and the object C at some point in the future while following their respective optimal control policies.

In some examples, when calculating the perception zone(s)116, the object-analysis component102may not consider an adversarial setting where the ego-machine E is trying to avoid the object C while subject to the worst-case (e.g., collision-seeking) actions of the object C. Instead, the object-analysis component102may make an even more conservative assumption that the ego-machine E may be in a situation where the ego-machine's E preferred actions are to steer toward the object C (e.g., similar to the assumption(s)114). With join dynamics ż=f (z, uE, uC), where z∈Z∈ndenotes the joint state of the ego-machine E and the object C, and uEand uCare the controls of the ego-machine E and the object C, respectively, the object-analysis component102then defines:

In other words, S(t) denotes a set of joint initial states where there exists a policy uE(·) and uC(·) where the ego-machine E and the object C may enter L (e.g., collide if L represents collision states) within a time horizon |t| in the future. The set S(t) may be computed as the zero sublevel set of a value function V(z, t) which may obey a Hamilton-Jacobi-Bellman partial differential equation (PDE):

The boundary condition for this PDE is defined by the function: Z→whose zero sublevel set encodes L. Additionally, equation (2) accounts for closed-loop control policies of both the ego-machine E and the object C because at any point in time and at any state, equation (2) considers uEand uCthat minimizes the value function.

By solving equation (2) backwards in time over a time horizon of T, the object-analysis component102may obtain the value function V(z, t) for t∈[−T, 0]. For a starting state (e.g., any starting state), z∈Z, V(s, t) corresponds to the lowest value of(·) along a system trajectory within |t| seconds if both the ego-machine E and the object C act optimally, by the following equation:

Thus, the set states from which collision may be reached within |t| seconds is determined by the following equation:

The object-analysis component102may numerically solve the PDE and store the value function over a n-dimensional grid in state space where z∈Z∈n.

In some examples, by approaching the perception zone116problem using reachability, the object-analysis component102may compute the set of joint states S(t) where membership of the set indicates that the entry into L (e.g., the safety requirement is violated) is possible within |t| seconds when considering a set of possible closed-loop control policies that the ego-machine E and the object C may take.

In some examples, the one or more of the elements (e.g., the parameters) of equation (2) may be updated to reflect one or more of the assumption(s)114. For example, the object-analysis component102may modify the information pattern and/or if the ego-machine E and the object C are minimizers or maximizers. For instance, as discussed herein, the object-analysis component102considers the minimum and maximum formulation (e.g., the ego-machine E and the object C are attempting to collide) while another approach may include assuming that the ego-machine E is collision-avoiding which corresponds to the minimum and maximum formulation.

Additionally, the discussion above described that the control sets (uEand UC) represent one or more (e.g., all) dynamic feasible controls of the ego-machine E and the object C. However, the object-analysis component102may restrict the control set to reflect assumptions about how the ego-machine E and/or the object C may behave in safety-critical scenarios. Furthermore, the object-analysis component102may define(·) as long as its zero sublevel set equals L. However, the object-analysis component102may design or learn alternative functions that capture more nuanced notations for safety. For example, by shapingto penalize more dangerous orientations (e.g., a T-bone collision), the object-analysis component102may encode collision severity or collision responsibility.

In some examples, the object-analysis component102may select a dynamics model for the ego-machine E. A higher fidelity model may better represent the true system, but may also make reachability computation quickly intractable based on the dimensionality. As such, the object-analysis component102may relax the perception zone soundness requirement while maintaining completeness. Essentially, the object-analysis component102may elect to use a lower fidelity dynamics model (e.g., ignore higher order derivatives such as jerk and steering rate) to keep the reachability computation tractable at the cost of being slightly over conservative.

In some examples, the dynamics model considered may depend on the type of obstacle detected (e.g., a vehicle, a pedestrian, a motorcycle, etc.). The object-analysis component102may assume that the ego-machine E and the object C will obey the dynamically extended simple model described by the following:

In equation (5), (x. y) is the position of the center of the rear axle in a fixed world frame, φ is the heading angle, v is the velocity, δ is the steering input, a is the acceleration input, and d is the distance between the front axle and real axle. Because of this, the object-analysis component102may define the relative state of z=[xR, yR, φR, vE, vC] and associated dynamics as:

The object-analysis component102may then use the relative dynamics from equation (6) to solve the PDE.

In some examples, the terminal value function(·) may be designed to reflect different safety requirements such as front or rear end collisions within a pre-specified velocity range. For simplicity, the object-analysis component102may consider any type of collision at any velocity as unsafe. Therefore, the object-analysis component102may define(·):=SD(·) to describe the signed distance between the ego-machine E and the object C as two rectangular rigid objects.

As described herein, in some examples, the object-analysis component102may assume that the ego-machine E will perform a hard braking maneuver and come to a complete stop. For instance, the object-analysis component102may use a reaction time Δtreactbefore the ego-machine E starts to brake at a fixed deceleration abrake. The reaction time allows for any possible latency that the ego-machine E may experience before the braking is initiated. To compute the reachability function according to those two phases, noting that the reachability function is computed backward in time, the object-analysis component102may compute the braking phase Vbrake(z, t) by usingSD(·) as the terminal value function. Then, for the reaction phase, the object-analysis component102may set

as the terminal value function where

is the time taken for the ego-machine E to come to a complete stop when applying constant deceleration abrake. In some examples, since the object-analysis component102may consider the stopping time in the braking phase, and the reaction time is over a fixed time interval, the object-analysis component102may simply consider the last time slice from the resulting value function.

In some examples, such as when a lower fidelity model is used as compared to a higher fidelity model with higher-order integrator states (e.g., jerk, steering rate, etc.), the object-analysis component102may be more conservative in the perception zone116computation. As such, completeness may still be preservice with a relaxed soundness requirement. For instance, extended dynamics may include:

In equation (7), integrators may be added to the controls to reflect realistic slew rates (as opposed to instantaneous changes) in acceleration, steering, and/or one or more other controls. The perception zone Sextcorresponding to the extended dynamics is a subset of S, the perception zone corresponding to the lower fidelity dynamics f. As such, the completeness may still be preserved by computing with the lower fidelity dynamics. In some examples, if there are controls

in the higher fidelity dynamics that lead to collision, then the object-analysis component102may consider the corresponding lower fidelity controls uE(·), uC(·) integrated from the higher fidelity controls as an existence of proof for membership for S. In other words, ignoring slew-rates may increase the control authority of the ego-machine E and the object C that are seeking collision with one another.

In some examples, the object-analysis component102may generate more than one perception zone116for more than one object within the environment. In such examples, the object-analysis component102may still consider the ego-machine E with each object C pairwise in order to preserve the completeness. For instance, given a safety-critical joint state zmulti∈ Smulti(e.g., there exist joint controls such that at least one contender K collides with the ego-machine E) the object-analysis component102may still consider the restriction of the joint controls for the ego-machine E and the contender K. This multi-agent configuration may still be regarded as safety-critical for the (E,K) pair, such that the object-analysis component102may still be sufficiently considering the safety-critical detection(s) (e.g., each false detection) individually.

In some examples, the perception zone(s)116may correspond to a region(s) within position space where the reachability value is negative. The stopping rate of the ego-vehicle E may be determined by the following:

In equation (8), rstopis the stopping distance for the ego-machine, abrakeis the deceleration of the ego-machine, Δtreactis the reaction time for the ego-machine to begin decelerating, L is the length of the ego-machine, and W is the width of the ego-machine.

Referring back toFIG.1, the object-analysis component102may generate data126indicating whether an object(s) is a safety-critical object( ) For example, and using the example ofFIG.3, the data126may indicate that the object204(1) is a safety-critical object and that the object204(2) is not a safety-critical object. In some examples, one or more systems may then perform one or more processes based on the data126.

For instance, in some examples, if a system is performing a verification test on the perception system118of the ego-machine, then the system may determine whether an error(s) from the perception system118is a safety-critical error(s) or not a safety-critical error(s). For a first example, if the perception system118detects an object that is not located within an environment (e.g., detects a “ghost object”), then the system may determine that the error is a safety-critical error when the data126indicates that the object is a safety-critical object or determine that the error is not a safety-critical error when the data126indicates that the object is not a safety-critical object. For a second example, if the perception system118does not detect an object that is located within an environment, then the system may again determine that the error is a safety-critical error when the data126indicates that the object is a safety-critical object or determine that the error is not a safety-critical error when the data126indicates that the object is not a safety-critical object.

In some examples, the system may be executing on the ego-machine while the ego-machine is navigating in the real-world. In such examples, the ego-machine may perform one or more operations based on the data126output from the object-analysis component102. For a first example, if the data126indicates that a detected object is a safety-critical object, then one or more other systems of the ego-machine may use data associated with the object when determining an operation(s) for the ego-machine to perform. For a second example, if the data126indicates that a detected object is not a safety-critical object, then the one or more other systems of the ego-machine may not use data associated with the object when determining an operation(s) for the ego-machine to perform.

As further illustrated in the example ofFIG.1, in some examples, the process100may include the object-analysis component102generating a lookup table(s)128— either online or offline (e.g., prior to deployment). In such examples, the lookup table(s)128may indicate the parameters106that caused objects to include safety-critical objects and/or the parameters106that caused objects to not include safety-critical objects. This way, the ego-machine may later use the lookup table(s)128to determine whether an object in the real-world is a safety-critical object without performing the full analysis using the object-analysis component102. For example, when the ego-machine detects an object, the ego-machine may determine the parameters106associated with the object and the parameters associated with the ego-machine. The ego-machine may then compare the determined parameters of the ego-machine and/or the object to the stored parameters within the lookup table(s)128in order to identify a match between the determined parameters and the stored parameters (and/or a substantial match, such as when the determined parameters are within one or more thresholds to the stored parameters). Based on the match (and/or the substantial match), the ego-machine may then use the lookup table(s)128to determine whether the object is a safety-critical object This way, the ego-machine is able to quickly determine whether objects within the environment are safety-critical objects or non-safety-critical objects. As such, by modeling the perception zones using a conservative approach (e.g., ego-machine and/or object steer toward one another, flat ground, no other objects between ego-machine the object, etc.), the object classifications of safety-critical or not safety-critical determined using the perception zones may help to ensure that the ego-machine accounts for any and all objects that may affect the planning and control of the ego-machine.

As described herein, the systems may provide improvement over conventional systems by better determining when objects are safety-critical objects and when objects are not safety-critical objects. As such,FIG.5Aillustrates a first example comparison between a reachability perception zone502(which may represent, and/or include, one of the perception zone(s)116) and a circular baseline perception zone504, in accordance with some embodiment of the present disclosure. As shown, an ego-machine506may detect an object508within an environment510(e.g., a real-world environment, a simulated environment, etc.). In the example ofFIG.5A, the ego-machine506may be navigating along a first lane512(1), as represented by an arrow514(1), and the object508may be navigating along a second lane512(2), as represented by an arrow514(2). The ego-machine506and/or the object-analysis component102may then determine whether the object508is safety-critical. As shown, using the circular baseline technique, the ego-machine506and/or another conventional system may determine that the object508is not safety-critical since the object508is located outside of the circular baseline perception zone502. However, using the reachability technique described herein, the ego-machine506and/or the object-analysis component102may determine that the object508is safety-critical since the object508is located within the reachability perception zone502.

In some examples, the circular baseline technique may determine that the object508is not safety-critical since the object508is located a far distance from the ego-machine506(e.g., a distance that is greater than a radius of the circular baseline perception zone504circle). However, since the reachability technique described herein uses dynamic models for the ego-machine506and the object508, as well as one or more possible interactions (e.g., all possible interactions) between the ego-machine506and the object508, the reachability technique may use the velocity of the object508when determining the reachability perception zone504. As such, even though the object508is located a far distance from the ego-machine506, the object508may still be safety-critical since the object508is moving at a high velocity towards the ego-machine506.

FIG.5Billustrates a second example comparison between a reachability perception zone516(which may represent, and/or include, one of the perception zone(s)116) and a circular baseline perception zone518, in accordance with some embodiment of the present disclosure. As shown, the ego-machine506may detect an object520within an environment522(e.g., a real-world environment, a simulated environment, etc.). In the example ofFIG.5B, the ego-machine506may be navigating along lane524, as represented by an arrow526, and the object520may be parked on another road. The ego-machine506and/or the object-analysis component102may then determine whether the object520is safety-critical. As shown, using the circular baseline technique, the ego-machine506and/or another conventional system may determine that the object520is safety-critical since the object520is located within the circular baseline perception zone518. However, using the reachability technique described herein, the ego-machine506and/or the object-analysis component102may determine that the object520is not safety-critical since the object520is located outside of the reachability perception zone516.

In some examples, the circular baseline technique may determine that the object520is safety-critical since the object520is located near the ego-machine506(e.g., a distance that is within a radius of the circular baseline perception zone518circle). However, since the reachability technique described herein uses dynamic models for the ego-machine506and the object520, as well as one or more possible interactions (e.g., all possible interactions) between the ego-machine506and the object520, the reachability technique may use the velocity of the object520when determining the reachability perception zone516. As such, even though the object520is located near the ego-machine506, the object520may still not be safety-critical since the object520is directed away from the ego-machine506and the object520is stopped.

Now referring toFIGS.6and7, each block of methods600and700, described herein, comprises a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods600and700may also be embodied as computer-usable instructions stored on computer storage media. The methods600and700may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, methods600and700are described, by way of example, with respect toFIG.1. However, these methods600and700may additionally or alternatively be executed by any one system, or any combination of systems, including, but not limited to, those described herein.

FIG.6is a flow diagram showing a method600for determining a perception zone and then using the perception zone to determine whether an object is a safety-critical object, in accordance with some embodiments of the present disclosure. The method600, at block B602, may include determining a location of an object within an environment. For instance, the ego-machine may be navigating within an environment, such as a real-world environment or a virtual environment. While navigating, the ego-machine may use one or more sensors to generate sensor data110. The object-analysis component102may then analyze the sensor data110to determine the location of the object (or may use information available from a simulation system to determine the location of the object). In some examples, the object-analysis component102may determine an additional parameter(s)106associated with the object. As described herein, the parameter(s)106for the object may include, but are not limited to, a type (e.g., a vehicle, a pedestrian, a scooter, etc.) of the object, a location of the object, a velocity of the object (e.g., a current velocity, a maximum velocity, etc.), an acceleration of the object, a deceleration of the object, a size of the object, a direction of travel of the object, steering limits (e.g., a turning radius) for the object, and/or any other parameter106.

The method600, at block B604, may include determining a velocity of an ego-machine. For instance, the object-analysis component102may determine the velocity associated with the ego-machine. In some examples, the object-analysis component102may determine an additional parameter(s)106associated with the ego-machine. As described herein, the parameter(s)106for the ego-machine may include, but are not limited to, a location of the ego-machine, a velocity of the ego-machine (e.g., a current velocity), a deceleration of the ego-machine, a time period for the ego-machine to begin decelerating, a size of the ego-machine, a front-rear axle distance of the ego-machine, a direction of travel of the ego-machine, steering limits (e.g., a turning radius) for the ego-machine, and/or any other parameter106.

The method600, at block B606, may include determining a perception zone based at least in part on the location of the object and the velocity of the ego-machine. For instance, the object-analysis component102may determine the perception zone116associated with the object using the location of the object and the velocity of the ego-machine. In some examples, the object-analysis component102may determine the perception zone116using an additional parameter(s)106associated with the ego-machine and/or an additional parameter(s)106associated with the object. Additionally, in some examples, the object-analysis component102may determine the perception zone116using one or more assumptions114.

The method600, at block B608, may include determining a type associated with the object based at least in part on the perception zone. For instance, the object-analysis component10may determine whether the object is a first type of object, such as a safety-critical object, or a second type of object, such as a non-safety-critical object, based on the perception zone116. As described herein, the object-analysis component102may determine that the object is the first type of object when the object is located within the perception zone116and determine that the object is the second type of object when the object is located outside of the perception zone116.

FIG.7is a flow diagram showing a method700for determining a perception zone associated with an object, in accordance with some embodiments of the present disclosure. The method700, at block B702, may include determining one or more first parameters associated with an ego-machine. For instance, the object-analysis component102may determine the parameter(s)106associated with the ego-machine. As described herein, the parameter(s)106for the ego-machine may include, but are not limited to, a location of the ego-machine, a velocity of the ego-machine (e.g., a current velocity), a deceleration of the ego-machine, a time period for the ego-machine to begin decelerating, a size of the ego-machine, a front-rear axle distance of the ego-machine, a direction of travel of the ego-machine, steering limits (e.g., a turning radius) for the ego-machine, and/or any other parameter106. In some examples, the parameter(s)106may include a detected parameter(s)108. In some examples, the parameter(s)106may include a pre-programmed parameter(s)112.

The method700, at block B704, may include determining one or more second parameters associated with an object. For instance, the object-analysis component102may determine the parameter(s)106associated with the object. As described herein, the parameter(s)106for the object may include, but are not limited to, a type (e.g., a vehicle, a pedestrian, a scooter, etc.) of the object, a location of the object, a velocity of the object (e.g., a current velocity, a maximum velocity, etc.), an acceleration of the object, a deceleration of the object, a size of the object, a direction of travel of the object, steering limits (e.g., a turning radius) for the object, and/or any other parameter106. In some examples, the parameter(s)106may include a detected parameter(s)108. In some examples, the parameter(s)106may include a pre-programmed parameter(s)112.

The method700, at block B706, may include determining one or more possible interactions between the ego-machine and the object. For instance, the object-analysis component102may determine the interaction(s) between the ego-machine and the object. In some examples, the object-analysis component102determines the interaction(s) using an assumption(s)114. For example, the object-analysis component102may determine the interaction(s) based on the assumption114that the ego-machine and the object will actively attempt to steer toward one another. Additionally, in some examples, the object-analysis component102may determine the interaction(s) based on the assumption114that no other obstacles are located between the ego-machine and the object. As such, based on the assumption(s)114, the interaction(s) may be associated with the ego-machine and the object taking one or more paths in order to steer toward one another.

The method700, at block B708, may include determining a perception zone associated with the object based at least in part on the one or more first parameters, the one or more second parameters, and the one or more possible interactions. For instance, the object-analysis component102may determine the perception zone116using the parameters(s)106for the ego-machine, the parameter(s)106for the object, and the possible interaction(s). In some examples, the object-analysis component102may then use the perception zone116to determine whether the object is a type of object, such as a safety-critical object, using one or more of the processes described herein. In some examples, the object-analysis component102may use the perception zone116to determine whether a perception error associated with the object is a safety-critical error, using one or more of the processes described herein.

The method700, at block B710, may include evaluating perception information corresponding to the object in view of the perception information. For instance, the object-analysis component102may evaluate the perception information (e.g., the location of the object, the type of the object, etc.) with respect to the perception zone.

The method700, at block B712, may include determining whether an error is present in the perception information. For instance, the object-analysis component102may determine whether there is an error in the perception information. In some examples, the object-analysis component102may determine that there is the error when the location of the object is within the perception zone116. Additionally, the object-analysis component102may determine that there is not the error when the location of the object is outside of the perception zone116

As described herein, in some examples, the object-analysis component102may include a neural network(s) that is configured to determine a perception zone(s), determine a type of object, and/or determine whether an error is a safety-critical error. As such,FIG.8Ais an illustration of an example object-analysis component802(which may represent, and/or include, the object-analysis component102), in accordance with some embodiments of the present disclosure. The object-analysis component802may be one example of a machine learning model that may be used to perform one or more of the processes described herein. The object-analysis component802may include one or more neural networks, such as convolutional neural networks (alternatively referred to herein as convolutional neural network802, convolutional network802, or CNN802).

As described herein, the object-analysis component802may use the sensor data110(with or without pre-processing) as an input. The sensor data110may represent images generated by one or more cameras, depth data generated by one or more depth sensors, RADAR data, LIDAR data, and/or any other type of sensor data. More specifically, the sensor data110may include individual images generated by the camera(s), where image data representative of one or more of the individual images may be input into the object-analysis component802at each iteration of the object-analysis component802. The sensor data110may be input as a single image, or may be input using batching, such as mini-batching. For example, two or more images may be used as inputs together (e.g., at the same time). The two or more images may be from two or more sensors that captured the images at the same time. In addition to sensor data110, other data may also be provided as input to the DNN(s), such as parameter information corresponding to the parameters described herein (e.g., velocity, acceleration, etc., corresponding to the objects and/or the ego-machine).

The sensor data110(and/or other data) may be input into a feature extractor layer(s)804of the object-analysis component802. The feature extractor layer(s)804may include any number of layers804, such as the layers804A-804C. One or more of the layers804may include an input layer. The input layer may hold values associated with the sensor data110. For example, when the sensor data110is an image(s), the input layer may hold values representative of the raw pixel values of the image(s) as a volume (e.g., a width, W, a height, H, and color channels, C (e.g., RGB), such as 32.times.32.times.3), and/or a batch size, B (e.g., where batching is used).

One or more layers804may include convolutional layers. The convolutional layers may compute the output of neurons that are connected to local regions in an input layer (e.g., the input layer), each neuron computing a dot product between their weights and a small region they are connected to in the input volume. A result of a convolutional layer may be another volume, with one of the dimensions based on the number of filters applied (e.g., the width, the height, and the number of filters, such as 32.times.32.times.12, if 12 were the number of filters).

One or more of the layers804may include a rectified linear unit (ReLU) layer. The ReLU layer(s) may apply an elementwise activation function, such as the max (0, x), thresholding at zero, for example. The resulting volume of a ReLU layer may be the same as the volume of the input of the ReLU layer.

One or more of the layers804may include a pooling layer. The pooling layer may perform a down-sampling operation along the spatial dimensions (e.g., the height and the width), which may result in a smaller volume than the input of the pooling layer (e.g., 16.times.16.times.12 from the 32.times.32.times.12 input volume). In some examples, the object-analysis component102may not include any pooling layers. In such examples, other types of convolution layers may be used in place of pooling layers. In some examples, the feature extractor layer(s)804may include alternating convolutional layers and pooling layers.

One or more of the layers804may include a fully connected layer. Each neuron in the fully connected layer(s) may be connected to each of the neurons in the previous volume. The fully connected layer may compute class scores, and the resulting volume may be 1.times. 1.times.number of classes. In some examples, the feature extractor layer(s)804may include a fully connected layer, while in other examples, the fully connected layer of the object-analysis component802may be the fully connected layer separate from the feature extractor layer(s)804. In some examples, no fully connected layers may be used by the feature extractor layer(s)804and/or the object-analysis component802as a whole, in an effort to increase processing times and reduce computing resource requirements. In such examples, where no fully connected layers are used, the object-analysis component802may be referred to as a fully convolutional network.

One or more of the layers804may, in some examples, include deconvolutional layer(s). However, the use of the term deconvolutional may be misleading and is not intended to be limiting. For example, the deconvolutional layer(s) may alternatively be referred to as transposed convolutional layers or fractionally strided convolutional layers. The deconvolutional layer(s) may be used to perform up-sampling on the output of a prior layer. For example, the deconvolutional layer(s) may be used to up-sample to a spatial resolution that is equal to the spatial resolution of the input images (e.g., the sensor data110) to the object-analysis component802, or used to up-sample to the input spatial resolution of a next layer.

Although input layers, convolutional layers, pooling layers, ReLU layers, deconvolutional layers, and fully connected layers are discussed herein with respect to the feature extractor layer(s)804, this is not intended to be limiting. For example, additional or alternative layers804may be used in the feature extractor layer(s)804, such as normalization layers, SoftMax layers, and/or other layer types.

The output of the feature extractor layer(s)804may be an input to a perception zone layer(s)806. The perception zone layer(s)806A-C may use one or more of the layer types described herein with respect to the feature extractor layer(s)804. As described herein, the perception zone layer(s)806may not include any fully connected layers, in some examples, to reduce processing speeds and decrease computing resource requirements. In such examples, the perception zone layers806may be referred to as fully convolutional layers.

Different orders and numbers of the layers804and806of the object-analysis component802may be used, depending on the embodiment. For example, where two or more cameras or other sensor types are used to generate inputs, there may be a different order and number of layers804and806for one or more of the sensors. As another example, different ordering and numbering of layers may be used depending on the type of sensor used to generate the sensor data110, or the type of the sensor data110(e.g., RGB, YUV, etc.). In other words, the order and number of layers804and806of the object-analysis component802is not limited to any one architecture.

In addition, some of the layers804and806may include parameters (e.g., weights and/or biases)˜such as the feature extractor layer(s)804and/or the perception zone layer(s)806— while others may not, such as the ReLU layers and pooling layers, for example. In some examples, the parameters may be learned by the object-analysis component802during training. Further, some of the layers804and806may include additional hyper-parameters (e.g., learning rate, stride, epochs, kernel size, number of filters, type of pooling for pooling layers, etc.)— such as the convolutional layer(s), the deconvolutional layer(s), and the pooling layer(s)—while other layers may not, such as the ReLU layer(s). Various activation functions may be used, including but not limited to, ReLU, leaky ReLU, sigmoid, hyperbolic tangent (tan h), exponential linear unit (ELU), etc. The parameters, hyper-parameters, and/or activation functions are not to be limited and may differ depending on the embodiment.

Now referring toFIG.8B,FIG.8Bis a data flow diagram illustrating a process810for training the object-analysis component802for determining and using a perception zone(s), in accordance with some embodiments of the present disclosure. As shown, the object-analysis component802may be trained using sensor data812(e.g., training sensor data). The sensor data812used for training may include original images (e.g., as captured by one or more image sensors), down-sampled images, up-sampled images, cropped or region of interest (ROI) images, otherwise augmented images, and/or a combination thereof. The sensor data812may represent images or other sensor data representations (e.g., point clouds, projection images, etc.) captured by one or more sensors (e.g., cameras, depth sensors, etc.), and/or may be images and/or other sensor data representations captured from within a virtual environment used for testing and/or generating training sensor data (e.g., a virtual camera of a virtual machine within a virtual or simulated environment). In some examples, the sensor data812may represent images and/or other sensor data representations from a data store or repository of training sensor data (e.g., images of driving surfaces, RADAR data from an automatic teller machine (ATM), images from a surveillance system, etc.).

The object-analysis component802may be trained using the training sensor data812as well as corresponding ground truth data814. The ground truth data814may include annotations, labels, masks, indicated perception zones, and/or the like. For example, in some embodiments, the ground truth data814may include perception zones816(e.g., indicating where perception zones are present in the particular sensor data instance) and/or safety-critical information818(e.g., mask, tablet, etc.). The ground truth data814may be generated within a drawing program (e.g., an annotation program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating the ground truth data814, and/or may be hand drawn, in some examples. In any example, the ground truth data814may be synthetically produced (e.g., generated from computer models or renderings), real produced (e.g., designed and produced from real-world data), machine-automated (e.g., using feature analysis and learning to extract features from data and then generate labels), human annotated (e.g., labeler, or annotation expert, defines the location of the labels), and/or a combination thereof (e.g., human identifies vertices of polylines, machine generates polygons using polygon rasterizer). In some examples, for each input image, there may be corresponding ground truth data814.

A training engine820may use one or more loss functions that measure loss (e.g., error) in outputs822(which may represent, and/or include, the safety-critical data126and/or the output808) as compared to the ground truth data814. Any type of loss function may be used, such as cross entropy loss, mean squared error, mean absolute error, mean bias error, and/or other loss function types. In some embodiments, different outputs822may have different loss functions. For example, the perception zone outputs may have a first loss function and the safety-critical information may have a second loss function. In such examples, the loss functions may be combined to form a total loss, and the total loss may be used to train (e.g., update the parameters of) the object-analysis component802. In any example, backward pass computations may be performed to recursively compute gradients of the loss function(s) with respect to training parameters. In some examples, weight and biases of the object-analysis component802may be used to compute these gradients.

Example Autonomous Vehicle

The vehicle900may include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle900may include a propulsion system950, such as an internal combustion engine, hybrid electric power plant, an all-electric engine, and/or another propulsion system type. The propulsion system950may be connected to a drive train of the vehicle900, which may include a transmission, to enable the propulsion of the vehicle900. The propulsion system950may be controlled in response to receiving signals from the throttle/accelerator952.

A steering system954, which may include a steering wheel, may be used to steer the vehicle900(e.g., along a desired path or route) when the propulsion system950is operating (e.g., when the vehicle is in motion). The steering system954may receive signals from a steering actuator956. The steering wheel may be optional for full automation (Level 5) functionality.

The brake sensor system946may be used to operate the vehicle brakes in response to receiving signals from the brake actuators948and/or brake sensors.

Controller(s)936, which may include one or more system on chips (SoCs)904(FIG.9C) and/or GPU(s), may provide signals (e.g., representative of commands) to one or more components and/or systems of the vehicle900. For example, the controller(s) may send signals to operate the vehicle brakes via one or more brake actuators948, to operate the steering system954via one or more steering actuators956, to operate the propulsion system950via one or more throttle/accelerators952. The controller(s)936may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals, and output operation commands (e.g., signals representing commands) to enable autonomous driving and/or to assist a human driver in driving the vehicle900. The controller(s)936may include a first controller936for autonomous driving functions, a second controller936for functional safety functions, a third controller936for artificial intelligence functionality (e.g., computer vision), a fourth controller936for infotainment functionality, a fifth controller936for redundancy in emergency conditions, and/or other controllers. In some examples, a single controller936may handle two or more of the above functionalities, two or more controllers936may handle a single functionality, and/or any combination thereof.

The controller(s)936may provide the signals for controlling one or more components and/or systems of the vehicle900in response to sensor data received from one or more sensors (e.g., sensor inputs). The sensor data may be received from, for example and without limitation, global navigation satellite systems (“GNSS”) sensor(s)958(e.g., Global Positioning System sensor(s)), RADAR sensor(s)960, ultrasonic sensor(s)962, LIDAR sensor(s)964, inertial measurement unit (IMU) sensor(s)966(e.g., accelerometer(s), gyroscope(s), magnetic compass(es), magnetometer(s), etc.), microphone(s)996, stereo camera(s)968, wide-view camera(s)970(e.g., fisheye cameras), infrared camera(s)972, surround camera(s)974(e.g., 360 degree cameras), long-range and/or mid-range camera(s)998, speed sensor(s)944(e.g., for measuring the speed of the vehicle900), vibration sensor(s)942, steering sensor(s)940, brake sensor(s) (e.g., as part of the brake sensor system946), and/or other sensor types.

One or more of the controller(s)936may receive inputs (e.g., represented by input data) from an instrument cluster932of the vehicle900and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display934, an audible annunciator, a loudspeaker, and/or via other components of the vehicle900. The outputs may include information such as vehicle velocity, speed, time, map data (e.g., the High Definition (“HD”) map922ofFIG.9C), location data (e.g., the vehicle's900location, such as on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s)936, etc. For example, the HMI display934may display information about the presence of one or more objects (e.g., a street sign, caution sign, traffic light changing, etc.), and/or information about driving maneuvers the vehicle has made, is making, or will make (e.g., changing lanes now, taking exit34B in two miles, etc.).

The vehicle900further includes a network interface924which may use one or more wireless antenna(s)926and/or modem(s) to communicate over one or more networks. For example, the network interface924may be capable of communication over Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile communication (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. The wireless antenna(s)926may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.), using local area network(s), such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and/or low power wide-area network(s) (“LPWANs”), such as LoRaWAN, SigFox, etc.

FIG.9Bis an example of camera locations and fields of view for the example autonomous vehicle900ofFIG.9A, in accordance with some embodiments of the present disclosure. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and/or alternative cameras may be included and/or the cameras may be located at different locations on the vehicle900.

A variety of cameras may be used in a front-facing configuration, including, for example, a monocular camera platform that includes a complementary metal oxide semiconductor (“CMOS”) color imager. Another example may be a wide-view camera(s)970that may be used to perceive objects coming into view from the periphery (e.g., pedestrians, crossing traffic or bicycles). Although only one wide-view camera is illustrated inFIG.9B, there may be any number (including zero) of wide-view cameras970on the vehicle900. In addition, any number of long-range camera(s)998(e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The long-range camera(s)998may also be used for object detection and classification, as well as basic object tracking.

Any number of stereo cameras968may also be included in a front-facing configuration. In at least one embodiment, one or more of stereo camera(s)968may include an integrated control unit comprising a scalable processing unit, which may provide a programmable logic (“FPGA”) and a multi-core micro-processor with an integrated Controller Area Network (“CAN”) or Ethernet interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including a distance estimate for all the points in the image. An alternative stereo camera(s)968may include a compact stereo vision sensor(s) that may include two camera lenses (one each on the left and right) and an image processing chip that may measure the distance from the vehicle to the target object and use the generated information (e.g., metadata) to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s)968may be used in addition to, or alternatively from, those described herein.

Cameras with a field of view that include portions of the environment to the side of the vehicle900(e.g., side-view cameras) may be used for surround view, providing information used to create and update the occupancy grid, as well as to generate side impact collision warnings. For example, surround camera(s)974(e.g., four surround cameras974as illustrated inFIG.9B) may be positioned to on the vehicle900. The surround camera(s)974may include wide-view camera(s)970, fisheye camera(s), 360 degree camera(s), and/or the like. Four example, four fisheye cameras may be positioned on the vehicle's front, rear, and sides. In an alternative arrangement, the vehicle may use three surround camera(s)974(e.g., left, right, and rear), and may leverage one or more other camera(s) (e.g., a forward-facing camera) as a fourth surround view camera.

Cameras with a field of view that include portions of the environment to the rear of the vehicle900(e.g., rear-view cameras) may be used for park assistance, surround view, rear collision warnings, and creating and updating the occupancy grid. A wide variety of cameras may be used including, but not limited to, cameras that are also suitable as a front-facing camera(s) (e.g., long-range and/or mid-range camera(s)998, stereo camera(s)968), infrared camera(s)972, etc.), as described herein.

Each of the components, features, and systems of the vehicle900inFIG.9Care illustrated as being connected via bus902. The bus902may include a Controller Area Network (CAN) data interface (alternatively referred to herein as a “CAN bus”). A CAN may be a network inside the vehicle900used to aid in control of various features and functionality of the vehicle900, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus may be read to find steering wheel angle, ground speed, engine revolutions per minute (RPMs), button positions, and/or other vehicle status indicators. The CAN bus may be ASIL B compliant.

Although the bus902is described herein as being a CAN bus, this is not intended to be limiting. For example, in addition to, or alternatively from, the CAN bus, FlexRay and/or Ethernet may be used. Additionally, although a single line is used to represent the bus902, this is not intended to be limiting. For example, there may be any number of busses902, which may include one or more CAN busses, one or more FlexRay busses, one or more Ethernet busses, and/or one or more other types of busses using a different protocol. In some examples, two or more busses902may be used to perform different functions, and/or may be used for redundancy. For example, a first bus902may be used for collision avoidance functionality and a second bus902may be used for actuation control. In any example, each bus902may communicate with any of the components of the vehicle900, and two or more busses902may communicate with the same components. In some examples, each SoC904, each controller936, and/or each computer within the vehicle may have access to the same input data (e.g., inputs from sensors of the vehicle900), and may be connected to a common bus, such the CAN bus.

The vehicle900may include one or more controller(s)936, such as those described herein with respect toFIG.9A. The controller(s)936may be used for a variety of functions. The controller(s)936may be coupled to any of the various other components and systems of the vehicle900, and may be used for control of the vehicle900, artificial intelligence of the vehicle900, infotainment for the vehicle900, and/or the like.

The vehicle900may include a system(s) on a chip (SoC)904. The SoC904may include CPU(s)906, GPU(s)908, processor(s)910, cache(s)912, accelerator(s)914, data store(s)916, and/or other components and features not illustrated. The SoC(s)904may be used to control the vehicle900in a variety of platforms and systems. For example, the SoC(s)904may be combined in a system (e.g., the system of the vehicle900) with an HD map922which may obtain map refreshes and/or updates via a network interface924from one or more servers (e.g., server(s)978ofFIG.9D).

The CPU(s)906may include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU(s)906may include multiple cores and/or L2 caches. For example, in some embodiments, the CPU(s)906may include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU(s)906may include four dual-core clusters where each cluster has a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s)906(e.g., the CCPLEX) may be configured to support simultaneous cluster operation enabling any combination of the clusters of the CPU(s)906to be active at any given time.

The GPU(s)908may include an integrated GPU (alternatively referred to herein as an “iGPU”). The GPU(s)908may be programmable and may be efficient for parallel workloads. The GPU(s)908, in some examples, may use an enhanced tensor instruction set. The GPU(s)908may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96 KB storage capacity), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a 512 KB storage capacity). In some embodiments, the GPU(s)908may include at least eight streaming microprocessors. The GPU(s)908may use compute application programming interface(s) (API(s)). In addition, the GPU(s)908may use one or more parallel computing platforms and/or programming models (e.g., NVIDIA's CUDA).

The GPU(s)908may include unified memory technology including access counters to allow for more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory ranges shared between processors. In some examples, address translation services (ATS) support may be used to allow the GPU(s)908to access the CPU(s)906page tables directly. In such examples, when the GPU(s)908memory management unit (MMU) experiences a miss, an address translation request may be transmitted to the CPU(s)906. In response, the CPU(s)906may look in its page tables for the virtual-to-physical mapping for the address and transmits the translation back to the GPU(s)908. As such, unified memory technology may allow a single unified virtual address space for memory of both the CPU(s)906and the GPU(s)908, thereby simplifying the GPU(s)908programming and porting of applications to the GPU(s)908.

In addition, the GPU(s)908may include an access counter that may keep track of the frequency of access of the GPU(s)908to memory of other processors. The access counter may help ensure that memory pages are moved to the physical memory of the processor that is accessing the pages most frequently.

The SoC(s)904may include any number of cache(s)912, including those described herein. For example, the cache(s)912may include an L3 cache that is available to both the CPU(s)906and the GPU(s)908(e.g., that is connected both the CPU(s)906and the GPU(s)908). The cache(s)912may include a write-back cache that may keep track of states of lines, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

The SoC(s)904may include an arithmetic logic unit(s) (ALU(s)) which may be leveraged in performing processing with respect to any of the variety of tasks or operations of the vehicle900—such as processing DNNs. In addition, the SoC(s)904may include a floating point unit(s) (FPU(s))— or other math coprocessor or numeric coprocessor types—for performing mathematical operations within the system. For example, the SoC(s)104may include one or more FPUs integrated as execution units within a CPU(s)906and/or GPU(s)908.

The SoC(s)904may include one or more accelerators914(e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s)904may include a hardware acceleration cluster that may include optimized hardware accelerators and/or large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM), may enable the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster may be used to complement the GPU(s)908and to off-load some of the tasks of the GPU(s)908(e.g., to free up more cycles of the GPU(s)908for performing other tasks). As an example, the accelerator(s)914may be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be amenable to acceleration. The term “CNN,” as used herein, may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Fast RCNNs (e.g., as used for object detection).

The DLA(s) may perform any function of the GPU(s)908, and by using an inference accelerator, for example, a designer may target either the DLA(s) or the GPU(s)908for any function. For example, the designer may focus processing of CNNs and floating point operations on the DLA(s) and leave other functions to the GPU(s)908and/or other accelerator(s)914.

The SoC(s)904may include data store(s)916(e.g., memory). The data store(s)916may be on-chip memory of the SoC(s)904, which may store neural networks to be executed on the GPU and/or the DLA. In some examples, the data store(s)916may be large enough in capacity to store multiple instances of neural networks for redundancy and safety. The data store(s)912may comprise L2 or L3 cache(s)912. Reference to the data store(s)916may include reference to the memory associated with the PVA, DLA, and/or other accelerator(s)914, as described herein.

The SoC(s)904may include one or more processor(s)910(e.g., embedded processors). The processor(s)910may include a boot and power management processor that may be a dedicated processor and subsystem to handle boot power and management functions and related security enforcement. The boot and power management processor may be a part of the SoC(s)904boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance in system low power state transitions, management of SoC(s)904thermals and temperature sensors, and/or management of the SoC(s)904power states. Each temperature sensor may be implemented as a ring-oscillator whose output frequency is proportional to temperature, and the SoC(s)904may use the ring-oscillators to detect temperatures of the CPU(s)906, GPU(s)908, and/or accelerator(s)914. If temperatures are determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine and put the SoC(s)904into a lower power state and/or put the vehicle900into a chauffeur to safe stop mode (e.g., bring the vehicle900to a safe stop).

The processor(s)910may further include a real-time camera engine that may include a dedicated processor subsystem for handling real-time camera management.

The processor(s)910may further include a high-dynamic range signal processor that may include an image signal processor that is a hardware engine that is part of the camera processing pipeline.

The video image compositor may also be configured to perform stereo rectification on input stereo lens frames. The video image compositor may further be used for user interface composition when the operating system desktop is in use, and the GPU(s)908is not required to continuously render new surfaces. Even when the GPU(s)908is powered on and active doing 3D rendering, the video image compositor may be used to offload the GPU(s)908to improve performance and responsiveness.

The SoC(s)904may further include a broad range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and/or other devices. The SoC(s)904may be used to process data from cameras (e.g., connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor(s)964, RADAR sensor(s)960, etc. that may be connected over Ethernet), data from bus902(e.g., speed of vehicle900, steering wheel position, etc.), data from GNSS sensor(s)958(e.g., connected over Ethernet or CAN bus). The SoC(s)904may further include dedicated high-performance mass storage controllers that may include their own DMA engines, and that may be used to free the CPU(s)906from routine data management tasks.

The SoC(s)904may be an end-to-end platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and makes efficient use of computer vision and ADAS techniques for diversity and redundancy, provides a platform for a flexible, reliable driving software stack, along with deep learning tools. The SoC(s)904may be faster, more reliable, and even more energy-efficient and space-efficient than conventional systems. For example, the accelerator(s)914, when combined with the CPU(s)906, the GPU(s)908, and the data store(s)916, may provide for a fast, efficient platform for level 3-5 autonomous vehicles.

In some examples, a CNN for facial recognition and vehicle owner identification may use data from camera sensors to identify the presence of an authorized driver and/or owner of the vehicle900. The always on sensor processing engine may be used to unlock the vehicle when the owner approaches the driver door and turn on the lights, and, in security mode, to disable the vehicle when the owner leaves the vehicle. In this way, the SoC(s)904provide for security against theft and/or carjacking.

The vehicle may include a CPU(s)918(e.g., discrete CPU(s), or dCPU(s)), that may be coupled to the SoC(s)904via a high-speed interconnect (e.g., PCIe). The CPU(s)918may include an X86 processor, for example. The CPU(s)918may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between ADAS sensors and the SoC(s)904, and/or monitoring the status and health of the controller(s)936and/or infotainment SoC930, for example.

The vehicle900may include a GPU(s)920(e.g., discrete GPU(s), or dGPU(s)), that may be coupled to the SoC(s)904via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU(s)920may provide additional artificial intelligence functionality, such as by executing redundant and/or different neural networks, and may be used to train and/or update neural networks based on input (e.g., sensor data) from sensors of the vehicle900.

The vehicle900may further include the network interface924which may include one or more wireless antennas926(e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface924may be used to enable wireless connectivity over the Internet with the cloud (e.g., with the server(s)978and/or other network devices), with other vehicles, and/or with computing devices (e.g., client devices of passengers). To communicate with other vehicles, a direct link may be established between the two vehicles and/or an indirect link may be established (e.g., across networks and over the Internet). Direct links may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link may provide the vehicle900information about vehicles in proximity to the vehicle900(e.g., vehicles in front of, on the side of, and/or behind the vehicle900). This functionality may be part of a cooperative adaptive cruise control functionality of the vehicle900.

The vehicle900may further include data store(s)928which may include off-chip (e.g., off the SoC(s)904) storage. The data store(s)928may include one or more storage elements including RAM, SRAM, DRAM, VRAM, Flash, hard disks, and/or other components and/or devices that may store at least one bit of data.

The vehicle900may further include GNSS sensor(s)958. The GNSS sensor(s)958(e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.), to assist in mapping, perception, occupancy grid generation, and/or path planning functions. Any number of GNSS sensor(s)958may be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet to Serial (RS-232) bridge.

The vehicle900may further include RADAR sensor(s)960. The RADAR sensor(s)960may be used by the vehicle900for long-range vehicle detection, even in darkness and/or severe weather conditions. RADAR functional safety levels may be ASIL B. The RADAR sensor(s)960may use the CAN and/or the bus902(e.g., to transmit data generated by the RADAR sensor(s)960) for control and to access object tracking data, with access to Ethernet to access raw data in some examples. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor(s)960may be suitable for front, rear, and side RADAR use. In some example, Pulse Doppler RADAR sensor(s) are used.

The vehicle900may further include ultrasonic sensor(s)962. The ultrasonic sensor(s)962, which may be positioned at the front, back, and/or the sides of the vehicle900, may be used for park assist and/or to create and update an occupancy grid. A wide variety of ultrasonic sensor(s)962may be used, and different ultrasonic sensor(s)962may be used for different ranges of detection (e.g., 2.5 m, 4 m). The ultrasonic sensor(s)962may operate at functional safety levels of ASIL B.

The vehicle900may include LIDAR sensor(s)964. The LIDAR sensor(s)964may be used for object and pedestrian detection, emergency braking, collision avoidance, and/or other functions. The LIDAR sensor(s)964may be functional safety level ASIL B. In some examples, the vehicle900may include multiple LIDAR sensors964(e.g., two, four, six, etc.) that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

In some examples, the LIDAR sensor(s)964may be capable of providing a list of objects and their distances for a 360-degree field of view. Commercially available LIDAR sensor(s)964may have an advertised range of approximately 900 m, with an accuracy of 2 cm-3 cm, and with support for a 900 Mbps Ethernet connection, for example. In some examples, one or more non-protruding LIDAR sensors964may be used. In such examples, the LIDAR sensor(s)964may be implemented as a small device that may be embedded into the front, rear, sides, and/or corners of the vehicle900. The LIDAR sensor(s)964, in such examples, may provide up to a 120-degree horizontal and 35-degree vertical field-of-view, with a 200 m range even for low-reflectivity objects. Front-mounted LIDAR sensor(s)964may be configured for a horizontal field of view between 45 degrees and 135 degrees.

The vehicle may further include IMU sensor(s)966. The IMU sensor(s)966may be located at a center of the rear axle of the vehicle900, in some examples. The IMU sensor(s)966may include, for example and without limitation, an accelerometer(s), a magnetometer(s), a gyroscope(s), a magnetic compass(es), and/or other sensor types. In some examples, such as in six-axis applications, the IMU sensor(s)966may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s)966may include accelerometers, gyroscopes, and magnetometers.

In some embodiments, the IMU sensor(s)966may be implemented as a miniature, high performance GPS-Aided Inertial Navigation System (GPS/INS) that combines micro-electro-mechanical systems (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor(s)966may enable the vehicle900to estimate heading without requiring input from a magnetic sensor by directly observing and correlating the changes in velocity from GPS to the IMU sensor(s)966. In some examples, the IMU sensor(s)966and the GNSS sensor(s)958may be combined in a single integrated unit.

The vehicle may include microphone(s)996placed in and/or around the vehicle900. The microphone(s)996may be used for emergency vehicle detection and identification, among other things.

The vehicle may further include any number of camera types, including stereo camera(s)968, wide-view camera(s)970, infrared camera(s)972, surround camera(s)974, long-range and/or mid-range camera(s)998, and/or other camera types. The cameras may be used to capture image data around an entire periphery of the vehicle900. The types of cameras used depends on the embodiments and requirements for the vehicle900, and any combination of camera types may be used to provide the necessary coverage around the vehicle900. In addition, the number of cameras may differ depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and/or another number of cameras. The cameras may support, as an example and without limitation, Gigabit Multimedia Serial Link (GMSL) and/or Gigabit Ethernet. Each of the camera(s) is described with more detail herein with respect toFIG.9AandFIG.9B.

The vehicle900may further include vibration sensor(s)942. The vibration sensor(s)942may measure vibrations of components of the vehicle, such as the axle(s). For example, changes in vibrations may indicate a change in road surfaces. In another example, when two or more vibration sensors942are used, the differences between the vibrations may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a power-driven axle and a freely rotating axle).

The vehicle900may include an ADAS system938. The ADAS system938may include a SoC, in some examples. The ADAS system938may include autonomous/adaptive/automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warnings (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and/or other features and functionality.

The ACC systems may use RADAR sensor(s)960, LIDAR sensor(s)964, and/or a camera(s). The ACC systems may include longitudinal ACC and/or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle900and automatically adjust the vehicle speed to maintain a safe distance from vehicles ahead. Lateral ACC performs distance keeping, and advises the vehicle900to change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

LKA systems are a variation of LDW systems. LKA systems provide steering input or braking to correct the vehicle900if the vehicle900starts to exit the lane.

The vehicle900may further include the infotainment SoC930(e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system may not be a SoC, and may include two or more discrete components. The infotainment SoC930may include a combination of hardware and software that may be used to provide audio (e.g., music, a personal digital assistant, navigational instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and/or information services (e.g., navigation systems, rear-parking assistance, a radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, door open/close, air filter information, etc.) to the vehicle900. For example, the infotainment SoC930may radios, disk players, navigation systems, video players, USB and Bluetooth connectivity, carputers, in-car entertainment, Wi-Fi, steering wheel audio controls, hands free voice control, a heads-up display (HUD), an HMI display934, a telematics device, a control panel (e.g., for controlling and/or interacting with various components, features, and/or systems), and/or other components. The infotainment SoC930may further be used to provide information (e.g., visual and/or audible) to a user(s) of the vehicle, such as information from the ADAS system938, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and/or other information.

The infotainment SoC930may include GPU functionality. The infotainment SoC930may communicate over the bus902(e.g., CAN bus, Ethernet, etc.) with other devices, systems, and/or components of the vehicle900. In some examples, the infotainment SoC930may be coupled to a supervisory MCU such that the GPU of the infotainment system may perform some self-driving functions in the event that the primary controller(s)936(e.g., the primary and/or backup computers of the vehicle900) fail. In such an example, the infotainment SoC930may put the vehicle900into a chauffeur to safe stop mode, as described herein.

The vehicle900may further include an instrument cluster932(e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster932may include a controller and/or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster932may include a set of instrumentation such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicators, gearshift position indicator, seat belt warning light(s), parking-brake warning light(s), engine-malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and/or shared among the infotainment SoC930and the instrument cluster932. In other words, the instrument cluster932may be included as part of the infotainment SoC930, or vice versa.

FIG.9Dis a system diagram for communication between cloud-based server(s) and the example autonomous vehicle900ofFIG.9A, in accordance with some embodiments of the present disclosure. The system976may include server(s)978, network(s)990, and vehicles, including the vehicle900. The server(s)978may include a plurality of GPUs984(A)-984(H) (collectively referred to herein as GPUs984), PCIe switches982(A)-982(H) (collectively referred to herein as PCIe switches982), and/or CPUs980(A)-980(B) (collectively referred to herein as CPUs980). The GPUs984, the CPUs980, and the PCIe switches may be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces988developed by NVIDIA and/or PCIe connections986. In some examples, the GPUs984are connected via NVLink and/or NVSwitch SoC and the GPUs984and the PCIe switches982are connected via PCIe interconnects. Although eight GPUs984, two CPUs980, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the server(s)978may include any number of GPUs984, CPUs980, and/or PCIe switches. For example, the server(s)978may each include eight, sixteen, thirty-two, and/or more GPUs984.

The server(s)978may receive, over the network(s)990and from the vehicles, image data representative of images showing unexpected or changed road conditions, such as recently commenced road-work. The server(s)978may transmit, over the network(s)990and to the vehicles, neural networks992, updated neural networks992, and/or map information994, including information regarding traffic and road conditions. The updates to the map information994may include updates for the HD map922, such as information regarding construction sites, potholes, detours, flooding, and/or other obstructions. In some examples, the neural networks992, the updated neural networks992, and/or the map information994may have resulted from new training and/or experiences represented in data received from any number of vehicles in the environment, and/or based on training performed at a datacenter (e.g., using the server(s)978and/or other servers).

In some examples, the server(s)978may receive data from the vehicles and apply the data to up-to-date real-time neural networks for real-time intelligent inferencing. The server(s)978may include deep-learning supercomputers and/or dedicated AI computers powered by GPU(s)984, such as a DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server(s)978may include deep learning infrastructure that use only CPU-powered datacenters.

The deep-learning infrastructure of the server(s)978may be capable of fast, real-time inferencing, and may use that capability to evaluate and verify the health of the processors, software, and/or associated hardware in the vehicle900. For example, the deep-learning infrastructure may receive periodic updates from the vehicle900, such as a sequence of images and/or objects that the vehicle900has located in that sequence of images (e.g., via computer vision and/or other machine learning object classification techniques). The deep-learning infrastructure may run its own neural network to identify the objects and compare them with the objects identified by the vehicle900and, if the results do not match and the infrastructure concludes that the AI in the vehicle900is malfunctioning, the server(s)978may transmit a signal to the vehicle900instructing a fail-safe computer of the vehicle900to assume control, notify the passengers, and complete a safe parking maneuver.

For inferencing, the server(s)978may include the GPU(s)984and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration may make real-time responsiveness possible. In other examples, such as where performance is less critical, servers powered by CPUs, FPGAs, and other processors may be used for inferencing.

Example Computing Device

FIG.10is a block diagram of an example computing device(s)1000suitable for use in implementing some embodiments of the present disclosure. Computing device1000may include an interconnect system1002that directly or indirectly couples the following devices: memory1004, one or more central processing units (CPUs)1006, one or more graphics processing units (GPUs)1008, a communication interface1010, input/output (I/O) ports1012, input/output components1014, a power supply1016, one or more presentation components1018(e.g., display(s)), and one or more logic units1020. In at least one embodiment, the computing device(s)1000may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs1008may comprise one or more vGPUs, one or more of the CPUs1006may comprise one or more vCPUs, and/or one or more of the logic units1020may comprise one or more virtual logic units. As such, a computing device(s)1000may include discrete components (e.g., a full GPU dedicated to the computing device1000), virtual components (e.g., a portion of a GPU dedicated to the computing device1000), or a combination thereof.

Although the various blocks ofFIG.10are shown as connected via the interconnect system1002with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component1018, such as a display device, may be considered an I/O component1014(e.g., if the display is a touch screen). As another example, the CPUs1006and/or GPUs1008may include memory (e.g., the memory1004may be representative of a storage device in addition to the memory of the GPUs1008, the CPUs1006, and/or other components). In other words, the computing device ofFIG.10is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device ofFIG.10.

The interconnect system1002may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system1002may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU1006may be directly connected to the memory1004. Further, the CPU1006may be directly connected to the GPU1008. Where there is direct, or point-to-point connection between components, the interconnect system1002may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device1000.

The memory1004may include any of a variety of computer-readable media. The computer-readable media may be any available media that may be accessed by the computing device1000. The computer-readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, the computer-readable media may comprise computer-storage media and communication media.

The CPU(s)1006may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device1000to perform one or more of the methods and/or processes described herein. The CPU(s)1006may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s)1006may include any type of processor, and may include different types of processors depending on the type of computing device1000implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device1000, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device1000may include one or more CPUs1006in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

In addition to or alternatively from the CPU(s)1006, the GPU(s)1008may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device1000to perform one or more of the methods and/or processes described herein. One or more of the GPU(s)1008may be an integrated GPU (e.g., with one or more of the CPU(s)1006and/or one or more of the GPU(s)1008may be a discrete GPU. In embodiments, one or more of the GPU(s)1008may be a coprocessor of one or more of the CPU(s)1006. The GPU(s)1008may be used by the computing device1000to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s)1008may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s)1008may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s)1008may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s)1006received via a host interface). The GPU(s)1008may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory1004. The GPU(s)1008may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU1008may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In addition to or alternatively from the CPU(s)1006and/or the GPU(s)1008, the logic unit(s)1020may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device1000to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s)1006, the GPU(s)1008, and/or the logic unit(s)1020may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units1020may be part of and/or integrated in one or more of the CPU(s)1006and/or the GPU(s)1008and/or one or more of the logic units1020may be discrete components or otherwise external to the CPU(s)1006and/or the GPU(s)1008. In embodiments, one or more of the logic units1020may be a coprocessor of one or more of the CPU(s)1006and/or one or more of the GPU(s)1008.

The communication interface1010may include one or more receivers, transmitters, and/or transceivers that enable the computing device1000to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface1010may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet. In one or more embodiments, logic unit(s)1020and/or communication interface1010may include one or more data processing units (DPUs) to transmit data received over a network and/or through interconnect system1002directly to (e.g., a memory of) one or more GPU(s)1008.

The I/O ports1012may enable the computing device1000to be logically coupled to other devices including the I/O components1014, the presentation component(s)1018, and/or other components, some of which may be built in to (e.g., integrated in) the computing device1000. Illustrative I/O components1014include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components1014may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device1000. The computing device1000may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device1000may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device1000to render immersive augmented reality or virtual reality.

The power supply1016may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply1016may provide power to the computing device1000to enable the components of the computing device1000to operate.

The presentation component(s)1018may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s)1018may receive data from other components (e.g., the GPU(s)1008, the CPU(s)1006, DPUs, etc.), and output the data (e.g., as an image, video, sound, etc.).

Example Data Center

FIG.11illustrates an example data center1100that may be used in at least one embodiments of the present disclosure. The data center1100may include a data center infrastructure layer1110, a framework layer1120, a software layer1130, and/or an application layer1140.

As shown inFIG.11, the data center infrastructure layer1110may include a resource orchestrator1112, grouped computing resources1114, and node computing resources (“node C.R.s”)1116(1)-1116(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s1116(1)-1116(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (NW I/O) devices, network switches, virtual machines (VMs), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s1116(1)-1116(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s1116(1)-11161(N) may include one or more virtual components, such as vGPUs, vCPUs, and/or the like, and/or one or more of the node C.R.s1116(1)-1116(N) may correspond to a virtual machine (VM).

In at least one embodiment, grouped computing resources1114may include separate groupings of node C.R.s1116housed within one or more racks (not shown), or many racks housed in data centers at various geographical locations (also not shown). Separate groupings of node C.R.s1116within grouped computing resources1114may include grouped compute, network, memory or storage resources that may be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s1116including CPUs, GPUs, DPUs, and/or other processors may be grouped within one or more racks to provide compute resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and/or network switches, in any combination.

The resource orchestrator1112may configure or otherwise control one or more node C.R.s1116(1)-1116(N) and/or grouped computing resources1114. In at least one embodiment, resource orchestrator1112may include a software design infrastructure (SDI) management entity for the data center1100. The resource orchestrator1112may include hardware, software, or some combination thereof.

In at least one embodiment, as shown inFIG.11, framework layer1120may include a job scheduler1133, a configuration manager1134, a resource manager1136, and/or a distributed file system1138. The framework layer1120may include a framework to support software1132of software layer1130and/or one or more application(s)1142of application layer1140. The software1132or application(s)1142may respectively include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud and Microsoft Azure. The framework layer1120may be, but is not limited to, a type of free and open-source software web application framework such as Apache Spark™ (hereinafter “Spark”) that may utilize distributed file system1138for large-scale data processing (e.g., “big data”). In at least one embodiment, job scheduler1133may include a Spark driver to facilitate scheduling of workloads supported by various layers of data center1100. The configuration manager1134may be capable of configuring different layers such as software layer1130and framework layer1120including Spark and distributed file system1138for supporting large-scale data processing. The resource manager1136may be capable of managing clustered or grouped computing resources mapped to or allocated for support of distributed file system1138and job scheduler1133. In at least one embodiment, clustered or grouped computing resources may include grouped computing resource1114at data center infrastructure layer1110. The resource manager1136may coordinate with resource orchestrator1112to manage these mapped or allocated computing resources.

In at least one embodiment, software1132included in software layer1130may include software used by at least portions of node C.R.s1116(1)-1116(N), grouped computing resources1114, and/or distributed file system1138of framework layer1120. One or more types of software may include, but are not limited to, Internet web page search software, e-mail virus scan software, database software, and streaming video content software.

In at least one embodiment, application(s)1142included in application layer1140may include one or more types of applications used by at least portions of node C.R.s1116(1)-1116(N), grouped computing resources1114, and/or distributed file system1138of framework layer1120. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive compute, and a machine learning application, including training or inferencing software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and/or other machine learning applications used in conjunction with one or more embodiments.

In at least one embodiment, any of configuration manager1134, resource manager1136, and resource orchestrator1112may implement any number and type of self-modifying actions based on any amount and type of data acquired in any technically feasible fashion. Self-modifying actions may relieve a data center operator of data center1100from making possibly bad configuration decisions and possibly avoiding underutilized and/or poor performing portions of a data center.

Example Network Environments

Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network attached storage (NAS), other backend devices, and/or other device types. The client devices, servers, and/or other device types (e.g., each device) may be implemented on one or more instances of the computing device(s)1000ofFIG.10—e.g., each device may include similar components, features, and/or functionality of the computing device(s)1000. In addition, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices may be included as part of a data center1100, an example of which is described in more detail herein with respect toFIG.11.