Patent Description:
The output of each hidden layer is used as input to or more other layers in the network, i.e., one or more other hidden layers, the output layer, or both.

CHARLES R QI ET AL: "Frustrum Pointnets for 3D Object Detection from RGB-D Data" discloses extracting the 3D bounding frustum of an object by extruding 2D bounding boxes from image detectors. Then, within the 3D space trimmed by each of the 3D frustums, consecutively performing 3D object instance segmentation and amodal 3D bounding box regression using two variants of Point-Net. The segmentation network predicts the 3D mask of the object of interest (i.e. instance segmentation); and the regression network estimates the amodal 3D bounding box(covering the entire object even if only part of it is visible). VoxelNet: End-to-End Learning for Point Cloud-Based 3D Object Detection (Zhou & Tuzel, <NUM>) introduces a method to transform sparse point clouds into a structured representation by voxelizing the 3D space and applying a Voxel Feature Encoding (VFE) layer to learn rich local features. The model then processes these voxel features using a 3D convolutional neural network (CNN) for object detection. This approach eliminates the need for handcrafted feature engineering and significantly improves 3D object detection accuracy for autonomous driving applications.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that processes point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene.

According to a first aspect, there is provided a system for detecting objects within point clouds. The system obtains point cloud data representing a sensor measurement of a scene captured by one or more sensors and including three-dimensional points in the scene, and then determines multiple two-dimensional proposal locations based on the three-dimensional points in the scene. The system generates, for each two-dimensional proposal location, a feature representation from three-dimensional points in the point cloud data that are near the two-dimensional proposal location. The system then processes the feature representations of the two-dimensional proposal locations using an object detection neural network that is configured to generate an object detection output that identifies objects in the scene.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

The system described in this specification can process point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. The one or more sensors can be sensors of an autonomous vehicle (e.g., LIDAR sensors), the scene can be a scene that is in the vicinity of the autonomous vehicle, and the object detection output can be used to make autonomous driving decisions for the vehicle, to display information to operators or passengers of the vehicle, or both. The system implements a non-convolutional point-based network designed specifically for point cloud data that can generate accurate object detection outputs with minimal latency and at a relatively low computational cost. In contrast, many conventional approaches to classifying or detecting objects within point cloud data involve projecting point clouds into 2D planar images and processing such point clouds as if they are camera images using computationally-heavy convolutional image processing techniques to detect objects in the resulting images. By employing processing techniques initially developed to address challenges unique to camera-based 2D imagery, such approaches fail to take full advantage of some of the features of point clouds, such as their relatively high sparsity in 3D space. Moreover, because such approaches employ convolutional image processing techniques in which all spatial positions in an image are treated equally, by virtue of employing non-convolutional techniques for object detection, the system described in this specification also has the advantage of being capable of selectively directing computation to different spatial regions in the scene. In some examples, the system is configured to leverage this capability to adapt the amount of computation that is dedicated to each spatial region in the scene to system priorities, resource availability, or both. By manipulating the sampling procedure at inference time, the system may dynamically alter the computational demand by tuning the number of proposals that are determined without having to alter or retrain the system's point-based network. This framework not only allows the system to be flexibly targeted across a range of computational priorities, but also enables the system to generate object proposals in a manner geared to maximize spatial coverage or match the density of point clouds. Given the need for accurate real-time information in autonomous vehicles and the nature of their surroundings, the system described in this specification may better fit the requirements of autonomous vehicle-based perception systems.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that processes point cloud data representing a sensor measurement of a scene captured by one or more sensors to generate an object detection output that identifies locations of one or more objects in the scene. For example, the one or more sensors can be sensors of an autonomous vehicle, e.g., a land, air, or sea vehicle, and the scene can be a scene that is in the vicinity of the autonomous vehicle. The object detection output can then be used to make autonomous driving decisions for the vehicle, to display information to operators or passengers of the vehicle, or both.

In particular, the system receives point cloud data representing a sensor measurement of a scene captured by one or more sensors. The point cloud data includes a set of three-dimensional points, i.e., a set of points corresponding to reflections identified by one or more scans of the scene by the one or more sensors, and optionally features generated by the one or more sensors for the three-dimensional points, e.g., LiDAR features. Each three-dimensional point generally has x, y, and z coordinates (or three different coordinates in a different coordinate system).

The system determines, based on the three-dimensional points in the scene, a plurality of two-dimensional proposal locations. In particular, the system samples a fixed number of two-dimensional locations from the locations of the three-dimensional points. In other words, the system designates a pair of coordinates, e.g., (x,y), from the three coordinates representing the three-dimensional points and then samples a fixed number of two-dimensional proposal locations from among the designated coordinates, e.g., the (x,y) coordinates, of the three-dimensional points in the scene.

The system can sample the fixed number of two-dimensional proposal locations in any of a variety of data dependent but computationally efficient ways. As one example, the system can sample the fixed number of two-dimensional proposal locations using farthest point sampling, in which individual points are selected sequentially such that the next point selected is maximally far away from all previous points selected. As another example, the system can sample the fixed number of two-dimensional proposal locations using random uniform sampling, in which each two-dimensional proposal location is randomly sampled from a uniform distribution over the three-dimensional points, i.e., the (x,y) coordinates of each three-dimensional point are equally likely to be sampled.

The system generates, for each two-dimensional proposal location, a feature representation from three-dimensional points in the point cloud data that are near the two-dimensional proposal location. In some implementations, the system can modify this phase of the object detection process based on the amount of computational resources available for the process or the latency requirements for the object detection process. In particular, the system can adjust how many points are used for each two-dimensional proposal location to satisfy the resource or latency requirements, i.e., the system can adapt the object detector to different computational settings without needing to re-train any of the neural network layers that are used by the object detector. When fewer points need to be used to satisfy the requirements, the system can prioritize the points that have higher predictive priorities or that are in spatial regions that are likely to be relevant. For example, in the case of a self-driving vehicle, the system can prioritize points that are likely to be relevant to operation of the vehicle.

The system then processes the feature representations of the two-dimensional proposal locations using an object detection neural network that is configured to generate an object detection output that identifies objects in the scene.

These features and other features are described in more detail below.

<FIG> is a block diagram of an example on-board system <NUM>. The on-board system <NUM> is physically located on-board a vehicle <NUM>. The vehicle <NUM> in <FIG> is illustrated as an automobile, but the on-board system <NUM> can be located on-board any appropriate vehicle type. The vehicle <NUM> can be a fully autonomous vehicle that makes fully-autonomous driving decisions or a semi-autonomous vehicle that aids a human operator. For example, the vehicle <NUM> can autonomously apply the brakes if a full-vehicle prediction indicates that a human driver is about to collide with a detected object, e.g., a pedestrian, a cyclist, another vehicle. While the vehicle <NUM> is illustrated in <FIG> as being an automobile, the vehicle <NUM> can be any appropriate vehicle that uses sensor data to make fully-autonomous or semi-autonomous operation decisions. For example, the vehicle <NUM> can be a watercraft or an aircraft. Moreover, the on-board system <NUM> can include components additional to those depicted in <FIG> (e.g., a control subsystem or a user interface subsystem).

The on-board system <NUM> includes a sensor subsystem <NUM> which enables the on-board system <NUM> to "see" the environment in a vicinity of the vehicle <NUM>. The sensor subsystem <NUM> includes one or more sensors, some of which are configured to receive reflections of electromagnetic radiation from the environment in the vicinity of the vehicle <NUM>. For example, the sensor subsystem <NUM> can include one or more laser sensors (e.g., LIDAR sensors) that are configured to detect reflections of laser light. As another example, the sensor subsystem <NUM> can include one or more radar sensors that are configured to detect reflections of radio waves. As another example, the sensor subsystem <NUM> can include one or more camera sensors that are configured to detect reflections of visible light.

The sensor subsystem <NUM> repeatedly (i.e., at each of multiple time points) uses raw sensor measurements, data derived from raw sensor measurements, or both to generate sensor data <NUM>. The raw sensor measurements indicate the directions, intensities, and distances travelled by reflected radiation. For example, a sensor in the sensor subsystem <NUM> can transmit one or more pulses of electromagnetic radiation in a particular direction and can measure the intensity of any reflections as well as the time that the reflection was received. A distance can be computed by determining the time which elapses between transmitting a pulse and receiving its reflection. Each sensor can continually sweep a particular space in angle, azimuth, or both. Sweeping in azimuth, for example, can allow a sensor to detect multiple objects along the same line of sight.

In particular, the sensor data <NUM> includes point cloud data that characterizes the latest state of an environment (i.e., an environment at the current time point) in the vicinity of the vehicle <NUM>. A point cloud is a collection of data points defined by a given coordinate system. For example, in a three-dimensional coordinate system, a point cloud can define the shape of some real or synthetic physical system, where each point in the point cloud is defined by three values representing respective coordinates in the coordinate system, e.g., (x, y, z) coordinates. As another example, in a three-dimensional coordinate system, each point in the point cloud can be defined by more than three values, wherein three values represent coordinates in the coordinate system and the additional values each represent a property of the point of the point cloud, e.g., an intensity of the point in the point cloud. Point cloud data can be generated, for example, by using LIDAR sensors or depth camera sensors that are on-board the vehicle <NUM>. For example, each point in the point cloud can correspond to a reflection of laser light or other radiation transmitted in a particular direction by a sensor on-board the vehicle <NUM>.

The on-board system <NUM> can provide the sensor data <NUM> generated by the sensor subsystem <NUM> to a perception subsystem <NUM> for use in generating perception outputs <NUM>.

The perception subsystem <NUM> implements components that identify objects within a vicinity of the vehicle. The components typically include one or more fully-learned machine learning models. A machine learning model is said to be "fully-leamed" if the model has been trained to compute a desired prediction when performing a perception task. In other words, a fully-learned model generates a perception output based solely on being trained on training data rather than on human-programmed decisions. For example, the perception output <NUM> may be a classification output that includes a respective object score corresponding to each of one or more object categories, each object score representing a likelihood that the input sensor data characterizes an object belonging to the corresponding object category. As another example, the perception output <NUM> can include data defining one or more bounding boxes in the sensor data <NUM>, and optionally, for each of the one or more bounding boxes, a respective confidence score that represents a likelihood that an object belonging to an object category from a set of one or more object categories is present in the region of the environment shown in the bounding box. Examples of object categories include pedestrians, cyclists, or other vehicles near the vicinity of the vehicle <NUM> as it travels on a road.

The on-board system <NUM> can provide the perception outputs <NUM> to a planning subsystem <NUM>. When the planning subsystem <NUM> receives the perception outputs <NUM>, the planning subsystem <NUM> can use the perception outputs <NUM> to generate planning decisions which plan the future trajectory of the vehicle <NUM>. The planning decisions generated by the planning subsystem <NUM> can include, for example: yielding (e.g., to pedestrians identified in the perception outputs <NUM>), stopping (e.g., at a "Stop" sign identified in the perception outputs <NUM>), passing other vehicles identified in the perception outputs <NUM>, adjusting vehicle lane position to accommodate a bicyclist identified in the perception outputs <NUM>, slowing down in a school or construction zone, merging (e.g., onto a highway), and parking. The planning decisions generated by the planning subsystem <NUM> can be provided to a control system of the vehicle <NUM>. The control system of the vehicle can control some or all of the operations of the vehicle by implementing the planning decisions generated by the planning system. For example, in response to receiving a planning decision to apply the brakes of the vehicle, the control system of the vehicle <NUM> may transmit an electronic signal to a braking control unit of the vehicle. In response to receiving the electronic signal, the braking control unit can mechanically apply the brakes of the vehicle.

In order for the planning subsystem <NUM> to generate planning decisions which cause the vehicle <NUM> to travel along a safe and comfortable trajectory, the on-board system <NUM> must provide the planning subsystem <NUM> with high quality perception outputs <NUM>. Many approaches to classifying or detecting objects within point cloud data involve projecting point clouds into 2D planar images and processing such point clouds as if they are camera images, e.g., using image processing techniques such as those involving the use of convolutional neural network (CNN) architectures or convolutional operations, to detect objects in the resulting images. However, such approaches are oftentimes quite computationally demanding, such that running inference on high resolution images is often not feasible in many systems. Given that predictions must be made by the perception subsystem <NUM> with minimal latency in order to ensure that accurate planning decisions can be made by the planning subsystem <NUM>, and further that computational resources within vehicle <NUM> must travel with the vehicle and thus may be limited, the on-board system <NUM> may be seen as an example of one such system.

Thus, to generate perception outputs with sufficient accuracy and at a relatively low computational cost, the perception subsystem <NUM> may implement a non-convolutional object detector designed specifically for point cloud data that may better fit the requirements of autonomous vehicles. The architecture and functionality of such an object detector is described in further detail below with reference to <FIG>.

<FIG> is a block diagram of an example perception subsystem <NUM>. The perception subsystem <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. In some implementations, the perception subsystem <NUM> of <FIG> may correspond to the perception subsystem <NUM> as described above with reference to <FIG>. As depicted in <FIG>, the perception subsystem <NUM> includes a proposal location determination engine <NUM>, a featurizer <NUM>, and an object detection neural network <NUM>. Briefly, and as described in further detail below, given a location in a point cloud, the perception subsystem <NUM> determines or selects a subset of neighboring points in the point cloud, featurizes these points, and regresses these points to object class and bounding box parameters. Importantly, the object location is predicted relative to the selected location and does not employ any global information, i.e., information for points that are outside the subset of neighboring points in the point cloud. This setup ensures that each spatial location may be processed by the perception subsystem <NUM> independently, which may enable computation of each location by the perception subsystem <NUM> to be parallelized to decrease inference latency.

The proposal location determination engine <NUM> receives sensor data <NUM> as input and determines, based at least in part on sensor data <NUM>, a set of proposal locations <NUM>. The sensor data <NUM> includes point cloud data representing a sensor measurement of a scene captured by one or more sensors and including multiple three-dimensional points in the scene, and the proposal locations <NUM> that are determined by the proposal location determination engine <NUM> include multiple two-dimensional proposal locations. In some examples, the sensor data <NUM> may correspond to sensor data <NUM> as described above with reference to <FIG>.

More specifically, in some implementations, each of the three-dimensional points in the scene has respective (x, y) coordinates, and the two-dimensional proposal locations <NUM> that are determined by the proposal location determination engine <NUM> correspond to the (x, y) coordinates where individual points reside in the point cloud. As such, in some implementations, to determine the proposal locations <NUM>, the proposal location determination engine <NUM> may determine or sample a fixed number of two-dimensional proposal locations from among the (x, y) coordinates of the three-dimensional points in the scene.

In these implementations, the proposal location determination engine <NUM> may any employ of a variety of different techniques to determine or sample two-dimensional proposal locations from among the (x, y) coordinates of the three-dimensional points in the scene. As one example, the proposal location determination engine <NUM> may sample a fixed number of two-dimensional proposal locations from among the (x, y) coordinates of the three-dimensional points in the scene using random uniform sampling, in which each two-dimensional proposal location is randomly sampled from a uniform distribution over the three-dimensional points, i.e., the (x,y) coordinates of each three-dimensional point are equally likely to be sampled.

As another example, the proposal location determination engine <NUM> may sample a fixed number of two-dimensional proposal locations from among the (x, y) coordinates of the three-dimensional points in the scene using farthest point sampling (FPS), in which individual points are selected sequentially such that the next point selected is maximally far away from all previous points selected. The proposal locations <NUM> that are determined by the proposal location determination engine <NUM> are provided to the featurizer <NUM>.

The featurizer <NUM> receives proposal locations <NUM> as input from the proposal location determination engine <NUM> and generates, based at least in part on proposal locations <NUM>, a set of feature representations <NUM>.

More specifically, for each two-dimensional proposal location included in the proposal locations <NUM>, the featurizer <NUM> generates a feature representation from three-dimensional points in the point cloud data that are near the two-dimensional proposal location. As such, in some examples, the featurizer <NUM> generates feature representations <NUM> based on proposal locations <NUM> and based further on at least a portion of sensor data <NUM> or an abstraction thereof.

In some implementations, the featurizer <NUM> determines, for each two-dimensional proposal location included in the proposal locations <NUM>, a fixed number of points that have (x, y) coordinates that are within a threshold radius of the proposal location and generates the feature representation for the proposal location from the fixed number of points. For example, the fixed number of determined points may include between <NUM> and <NUM> points, and the threshold radius may be set to a value between <NUM> and <NUM> meters. Other configurations are possible. In these implementations, the featurizer <NUM> may generate the feature representation for each two-dimensional proposal location included in the proposal locations <NUM> from the sampled fixed number of points that have (x, y) coordinates that are within the threshold radius of the respective proposal location. As described in further detail below, in some examples, the featurizer <NUM> may include a featurizer neural network that may be leveraged to generate the feature representation for each two-dimensional proposal location included in the proposal locations <NUM>.

In some examples, the perception subsystem <NUM> can modify processes performed by the featurizer <NUM> based on the amount of computational resources available for the process or the latency requirements for the object detection process. In particular, the perception subsystem <NUM> can adjust how many points are determined or selected and used for each two-dimensional proposal location to satisfy the resource or latency requirements, i.e., the perception subsystem <NUM> can adapt to different computational settings without needing to re-train any of the neural network layers that are employed in the perception subsystem <NUM>. In some of these examples, the perception subsystem <NUM> determines how many points can be sampled for each proposal location while still satisfying latency or resource requirements, and then uses no more than the determined number of points when generating the feature representation. For instance, the perception subsystem <NUM> may determine how many points can be sampled for each proposal location while still satisfying latency or resource requirements based on how much time and/or compute is required to identify the points and generate the feature representations for the points in the current computational environment. When fewer points need to be used to satisfy the requirements, the perception subsystem <NUM> can prioritize the points that have higher predictive priorities or that are in spatial regions that are likely to be relevant. For example, in the case of a self-driving vehicle, the perception subsystem <NUM> can prioritize points that are likely to be relevant to operation of the vehicle.

In at least some of the aforementioned implementations, the featurizer <NUM> may further receive or otherwise access contextual data <NUM> and determine or select the fixed number of points for each two-dimensional proposal location included in the proposal locations <NUM> based on their distance from the proposal location and based further on the contextual data <NUM>.

For example, contextual data <NUM> may include data from one or more sensors of a self-driving vehicle, and the featurizer <NUM> may select a fixed number of points that have (x, y) coordinates that are (i) determined to be within a first threshold radius of the proposal location and (ii) determined to be within a second threshold radius of the self-driving vehicle based on contextual data <NUM>.

In some examples, the featurizer <NUM> may initially sample a larger number of points from the points that have (x,y) coordinates that are within the threshold radius, i.e., a larger number than the fixed number that will be used to generate the feature representation and then rank these points based on a relative importance to operation of the self-driving vehicle of each point based on the contextual data <NUM>. The featurizer <NUM> may then select, as the determined fixed number of points, a subset of the points that have (x, y) coordinates that are within the threshold radius of the proposal location based at least in part on the ranking. For example, the featurizer <NUM> may rank the points based on distance from the vehicle or based on other information in the contextual data <NUM>. In this way, the featurizer <NUM> may be able to prioritize points that are likely to be relevant to operation of the vehicle, and thus achieve computational savings. Other pieces of information that may be included in contextual data <NUM> and leveraged by the perception subsystem <NUM> to determine or select points for each proposal location include data indicative one or more computational loads that are currently placed or will be placed on the perception subsystem <NUM> or one or more other systems of a vehicle, data indicative of a level of confidence in perception output <NUM>, GPS coordinates or other data indicative of a current location of a vehicle, semantic or road map information that may be indicative of regions of pedestrian and/or vehicular traffic, temporal information, data indicative of current or future traffic or weather conditions within the vicinity of a vehicle, historical driving data, data indicative of the current speed or acceleration of a vehicle, data indicative of a vehicle's fuel and/or battery levels, satellite imagery, data communicated from and/or about other nearby vehicles, user preferences, and the like. As such, data that is included or represented in contextual data <NUM> may originate from a variety of different sources including one or more sensors onboard a vehicle, internet resources, computing devices in communication with the perception subsystem <NUM>, and so on.

In some implementations, contextual data <NUM> may include data obtained or generated by the perception subsystem <NUM> for one or more previous frames, including sensor data <NUM> from one or more previous frames, proposal locations <NUM> from one or more previous frames, feature representations <NUM> from one or more previous frames, and/or perception output <NUM> from one or more previous frames. Such data from previous frames may serve to provide the perception subsystem <NUM> with a relatively reliable estimate of where objects may be expected to be located. In this way, the perception subsystem <NUM> may be able to allocate more computational resources to the regions in the scene in which objects are more likely to be located in the current frame and/or allocate fewer computational resources to the regions in the scene in which objects are less likely to be located in the current frame.

In some examples, once the points for a given proposal location have been determined, the featurizer <NUM> may further re-center each determined point to an origin for the given proposal location, i.e., represent the determined points in a coordinate system in which the given proposal location is located at the origin and then use the re-centered points to generate the feature representation for the determined point.

In at least some of the aforementioned implementations, the featurizer <NUM> includes a featurizer neural network that may be leveraged to generate feature representations <NUM>. More specifically, for a given proposal location, the featurizer <NUM> may process a featurizer input for the given proposal location using the featurizer neural network to generate a feature representation for the given proposal location. For example, the featurizer input that is applied to the featurizer neural network may include data indicating a fixed number of points that are determined or selected for the given proposal location. For examples in which the featurizer <NUM> re-centers each determined point to an origin, the featurizer input that is applied to the featurizer neural network may include data indicating the re-centered points. Furthermore, in some examples, the featurizer input that is applied to the featurizer neural network may include data indicating sensor features for each of the determined points. In some implementations, the featurizer neural network that is included in the featurizer <NUM> may correspond to featurizer neural network <NUM>, as described in further detail below with reference to <FIG>.

<FIG> is a block diagram of an example featurizer neural network <NUM>. The featurizer neural network <NUM> receives data <NUM> as input and generates, based at least in part on data <NUM>, a set of feature representations <NUM>. As mentioned above, in some implementations, the featurizer neural network <NUM> may be implemented as part of the featurizer <NUM> of the perception subsystem <NUM> as described herein with reference to <FIG>. In these implementations, data <NUM> and feature representations <NUM> may correspond to the featurizer input and feature representations <NUM> as described above with reference to <FIG>, respectively. In the example of <FIG>, the featurizer neural network <NUM> includes multiple layers 361A-361E (e.g., <NUM> layers). Although <FIG> only includes a detailed diagram of layer 361B, it is to be understood that the architectures of layers 361A and 361C-361E may be similar or equivalent to that of layer 361B. Each one of layers 361A-361E receives a set of points as input, where each point has an associated feature vector. As shown in the detailed diagram of layer 361B, when processing an input for a given point, each one of layers 361A-361E may first compute aggregate statistics across the set of points, e.g., by computing the maximum (max) of each feature across the feature vectors for the set of points, and then concatenate the resulting global statistics back to the given point's feature to generate a concatenated input for the given point. Each one of layers 361A-361E may apply two fully-connected layers, each of which is composed of batch normalization (BN), linear projection, and ReLU activation to the concatenated input for the given point. The featurizer neural network <NUM> performs a readout of each of layers 361A-361E using aggregation, i.e., aggregates the outputs of each of layers 361A-361E for the set of points to generate a single feature vector, e.g., by computing the means of the outputs of the layer for the individual points in the set of points, and then concatenates the readouts together, i.e., concatenates the means of the layer outputs together, to form the featurization for the proposal location, e.g., feature representations <NUM>. By design, because the outputs of each layer are aggregated before being used in the feature representations, the featurizer neural network <NUM> can be used with varying numbers of input points, giving it a large degree of flexibility. As such, the number of points that are determined or selected and applied as input to the featurizer neural network <NUM> may be adjusted to adapt to different computational settings without issue, e.g., without needing to re-train any of layers 361A-361E of the featurizer neural network <NUM>. Although the featurizer neural network <NUM> is depicted in <FIG> as having <NUM> layers, e.g., layers 361A-361E, it is to be understood that the featurizer neural network <NUM> may be implemented with greater or fewer than <NUM> layers.

Referring once again to <FIG>, the feature representations <NUM> that are determined by the featurizer <NUM>, which may correspond to the feature representations <NUM> that are generated by the featurizer neural network <NUM> as described above with reference to <FIG>, are provided to the object detection neural network <NUM> for regression and classification. The perception subsystem <NUM> processes feature representations <NUM> using the object detection neural network <NUM> to generate perception output <NUM>. The perception output <NUM> that the object detection neural network <NUM> is configured to generate is an object detection output that identifies objects in the scene. In some examples, the perception output <NUM> may correspond to perception output <NUM> as described above with reference to <FIG>.

In some implementations, to generate the perception output <NUM>, the object detection neural network <NUM> projects each feature representation included in the feature representations <NUM> to generate multiple feature vectors for multiple anchor offsets, respectively, and processes the multiple feature vectors to generate an object detection output for each of the multiple anchor offsets. That is, for each proposal location, the neural network <NUM> generates a respective feature vector for each anchor offset and then processes the feature vector for the anchor offset to generate the object detection output for the anchor offset. In these implementations, each object detection output included in the perception output <NUM> corresponds to one of the proposal locations and one of the anchor offsets and identifies (i) a location of a possible object relative to a region of the scene that corresponds to the proposal location offset by the anchor offset and (ii) a likelihood that an object is located at the identified location. In at least some of these implementations, different anchor offsets are associated with different projection weights, and to generate a respective feature vector for each anchor offset, the object detection neural network <NUM> projects each feature representation included in the feature representations <NUM> in accordance with projection weights associated with the anchor offset. In at least some implementations, the object detection outputs also include a classification output for each of the identified locations and, to generate an object detection output for each of the multiple anchor offsets, the object detection neural network <NUM> uses the feature vector for the anchor offset to predict classification logits using a classification head and bounding box regression logits for the offset using a regression head, where each head includes one or more fully-connected or convolutional neural network layers. Such predictions may be included or represented in the perception output <NUM>. Furthermore, in some examples, the object detection neural network <NUM> employs non-maximal suppression (NMS) or at least one technique similar to NMS to remove predictions of the same class that heavily overlap with one another. In these examples, the remaining predictions may be included or represented in the perception output <NUM>.

The perception output <NUM> that is generated by way of the object detection neural network <NUM> of the perception subsystem <NUM> may be provided to one or more systems and used to make autonomous driving decisions for the vehicle, to display information to operators or passengers of the vehicle, or both. For example, the perception subsystem <NUM> may provide perception output <NUM> to one or more systems or subsystems that are similar to equivalent to one or more of those described above with reference to <FIG>, including the planning subsystem <NUM>, a control subsystem, and a user interface subsystem. Other configurations are possible.

The featurizer neural network of the featurizer <NUM> and the object detection neural network <NUM> may be trained jointly on ground truth object detection outputs for point clouds in a set of training data. As mentioned above, in some implementations, the featurizer neural network of the featurizer <NUM> may correspond to the featurizer neural network <NUM> as described with reference to <FIG>. The loss function used for the training of these neural networks can be an object detection loss that measures the quality of object detection outputs generated by the these neural networks relative to the ground truth object detection outputs, e.g., smoothed L1 losses for regressed values and cross entropy losses for classification outputs.

In some implementations, the perception subsystem <NUM> is further configured to remove points that are likely associated with ground reflections from obtained point cloud data. In at least some of these implementations, operations associated with this process may be carried out by the proposal location determination engine <NUM>. For example, the perception subsystem <NUM> may remove points with positions in the z-dimension that fail to satisfy one or more thresholds. In some examples, the perception subsystem <NUM> may remove points that are positioned outside of one or more specified ranges of positions in the z-dimension. Doing so may potentially yield computational savings and/or allow the system to focus computational resources on points that are more likely to be associated with a pedestrian, vehicle, or other object of interest.

Although described as distinct modules or components, it is to be understood that some or all of the functionality of each the proposal location determination engine <NUM>, featurizer <NUM>, and object detection neural network <NUM> may be provided by way of other modules or components of the perception subsystem <NUM> or in communication with the perception subsystem <NUM>. As one example, in some implementations, one or more of the operations as described above as being performed by the featurizer <NUM> may be performed by the proposal location determination engine <NUM>, such that the output that is provided to the featurizer <NUM> by the proposal location determination engine <NUM> may be similar or equivalent to data <NUM> as described above with reference to <FIG>. In such implementations, contextual data <NUM> may be provided to the proposal location determination engine <NUM>. In some examples, the proposal location determination engine <NUM> may utilize contextual data <NUM> to determine proposal locations <NUM>. Other configurations are possible.

<FIG> is a flow diagram of an example process <NUM> for detecting objects within point clouds. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an on-board system, e.g., the on-board system <NUM> of <FIG>, or subsystems thereof, e.g., the perception subsystem <NUM> of <FIG> or the perception subsystem <NUM> of <FIG>, appropriately programmed in accordance with this specification, can perform the process <NUM>. However, process <NUM> may be performed by other systems or system configurations.

The system obtains point cloud data (<NUM>) representing a sensor measurement of a scene captured by one or more sensors. For example, this may correspond to the perception subsystem <NUM> of the on-board system <NUM> of <FIG> obtaining sensor data <NUM> from the sensor subsystem of the on-board system <NUM>. Similarly, this may also correspond to the proposal location determination engine <NUM> of the perception subsystem <NUM> of <FIG> obtaining sensor data <NUM>. The point cloud data that is obtained by the system includes a plurality of three-dimensional points in the scene and, in some implementations, each three-dimensional point in the scene has respective (x, y) coordinates. In some examples, the one or more sensors are sensors of a self-driving vehicle, e.g., one or more LIDAR sensors or depth camera sensors. In some examples, the point cloud data includes sensor features generated by the one or more sensors for each of the three-dimensional points.

The system determines a plurality of two-dimensional proposal locations (<NUM>) based on the three-dimensional points in the scene. For example, this may correspond to the proposal location determination engine <NUM> of the perception subsystem <NUM> of <FIG> determining proposal locations <NUM> based on sensor data <NUM>. As mentioned above, in some implementations, each three-dimensional point in the scene has respective (x, y) coordinates. In at least some of these implementations, to determine a plurality of two-dimensional proposal locations based on the three-dimensional points in the scene, the system samples a fixed number of two-dimensional proposal locations from among the (x, y) coordinates of the three-dimensional points in the scene. In some examples, the system may use a sampling technique such as farthest point sampling (FPS) and/or random uniform sampling to sample the fixed number of two-dimensional proposal locations.

The system generates a feature representation for each two-dimensional proposal location (<NUM>) from three-dimensional points in the point cloud data that are near the two-dimensional proposal location. For example, this may correspond to the featurizer <NUM> of the perception subsystem <NUM> of <FIG> generating feature representations <NUM> based on proposal locations <NUM>. In some implementations, to generate a feature representation for each two-dimensional proposal location, the system determines a fixed number of points that have (x,y) coordinates that are within a threshold radius of the proposal location, and generates the feature representation for the two-dimensional proposal location from the sampled fixed number of points that have (x,y) coordinates that are within the threshold radius of the proposal location. As mentioned above, in some examples, the one or more sensors are sensors of a self-driving vehicle. In some of these examples, to determine the fixed number of points that have (x,y) coordinates that are within the threshold radius of the proposal location, in at least some of the aforementioned implementations, the system (i) samples a second, larger number of points from the points that have (x,y) coordinates that are within the threshold radius, (ii) ranks the second number of points based on a relative importance to operation of the self-driving vehicle, and (iii) selects, as the determined fixed number of points, a subset of the sampled second number of points based on the ranking. For instance, the system may rank the second number of points based on a distance from the self-driving vehicle. This may correspond to the featurizer <NUM> of the perception subsystem <NUM> of <FIG> ranking points within the vicinity of a proposal location, e.g., based at least in part on contextual data <NUM>.

In at least some of the aforementioned implementations, to generate the feature representation for the two-dimensional proposal location, the system (i) generates, for each determined point, a re-centered point that is centered at the two-dimensional proposal location, and (ii) processes a featurizer input including the re-centered points using a featurizer neural network to generate the feature representation. The featurizer neural network may be configured to process a variable number of input points. For example, the featurizer neural network may correspond to the featurizer neural network <NUM> of <FIG>. As mentioned above, in some examples, the point cloud data includes sensor features generated by the one or more sensors for each of the three-dimensional points. In at least some of these examples, the featurizer input that the system processes using the featurizer neural network includes the sensor features for each of the determined points.

The system processes the feature representations using an object detection neural network (<NUM>) that is configured to generate an object detection output that identifies objects in the scene. For example, this may correspond to the perception subsystem <NUM> of <FIG> processing feature representations <NUM> using the object detection neural network to generate perception output <NUM>. In some implementations, to process the feature representations of the two-dimensional proposal locations using an object detection neural network, for each proposal location, the system projects the feature representation to generate a respective feature vector for each of a plurality of anchor offsets, and processes the feature vectors to generate, for each of the plurality of anchor offsets, an object detection output that identifies (i) a location of a possible object relative to a region of the scene that corresponds to the proposal location offset by the anchor offset and (ii) a likelihood that an object is located at the identified location. In some examples, different anchor offsets have different projection weights. In at least some of these examples, to generate a respective feature vector for each anchor offset, the system projects the feature representation in accordance with projection weights for the anchor offset.

In some implementations, the process <NUM> includes one or more additional operations in which the system removes points that are likely associated with ground reflections from obtained point cloud data. For example, the system may remove points that are positioned outside of one or more specified ranges of positions in the z-dimension. As mentioned above with reference to <FIG>, doing so may potentially yield computational savings and/or allow the system to focus computational resources on points that are more likely to be associated with a pedestrian, vehicle, or other object of interest. In some examples, such one or more additional operations may be performed after the system obtains point cloud data (<NUM>), but before the system determines a plurality of two-dimensional proposal locations (<NUM>).

Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

Claim 1:
A method comprising:
obtaining point cloud data representing a sensor measurement of a scene captured by one or more sensors, the point cloud data comprising a plurality of three-dimensional points in the scene;
determining, based on the three-dimensional points in the scene, a plurality of two-dimensional proposal locations (<NUM>);
generating, for each two-dimensional proposal location, a feature representation (<NUM>) from three-dimensional points in the point cloud data that are within a threshold radius of the two-dimensional proposal location; and
processing the feature representations of the two-dimensional proposal locations using an object detection neural network (<NUM>) that is configured to generate an object detection output that identifies objects in the scene.