AUTONOMOUS VEHICLE SENSOR SELF-HIT DATA FILTERING

The present disclosure generally relates to detecting and filtering self-hit data from a sensor mounted to an autonomous vehicle and, more specifically, to identifying self-hit sensor data and generating and applying an image mask to filter out the self-hit sensor data. In some aspects, a method of the disclosed technology includes steps for collecting first sensor data for an environment around an autonomous vehicle (AV); identifying one or more data points, from the collected first sensor data, that correspond with a surface of the AV; generating an image mask representing the one or more data points that correspond with the surface of the AV; collecting second sensor data for the environment around the AV; and applying the image mask to the collected second sensor data. Systems and machine-readable media are also provided.

BACKGROUND

1. Technical Field

The present disclosure generally relates to detecting and filtering self-hit data from a sensor mounted to an autonomous vehicle and, more specifically, to identifying self-hit sensor data and generating and applying a mask to filter out the self-hit sensor data.

An autonomous vehicle is a motorized vehicle that can navigate without a human driver. An exemplary AV can include various image sensors, such as a camera sensor, a time-of-flight (TOF) sensor, a light detection and ranging (LiDAR) sensor, a radio detection and ranging (RADAR) sensor, and an ultrasonic sensor, amongst others. The sensors collect data and measurements that the autonomous vehicle can use for operations such as navigation. The sensors can provide the data and measurements to an internal computing system of the autonomous vehicle, which can use the data and measurements to control a mechanical system of the autonomous vehicle, such as a vehicle propulsion system, a braking system, or a steering system. Typically, the sensors are mounted at fixed locations on the autonomous vehicles.

DETAILED DESCRIPTION

Some aspect of the present technology may relate to the gathering and use of data available from various sources to improve safety, quality, and experience. The present disclosure contemplates that in some instances, this gathered data may include personal information. The present disclosure contemplates that the entities involved with such personal information respect and value privacy policies and practices.

An autonomous vehicle (AV) is a motorized vehicle that can navigate roadways without a human driver. Collection of environmental sensor data, and accurately identifying objects around the AV based on the collected environmental sensor data, can be important for safe and efficient navigation of the AV. Environmental data collection can be performed using sensors disposed on the AV, such as using time-of-flight (TOF) sensors, light detection and ranging (LIDAR) sensors, or camera sensors mounted about the surface of the AV. A TOF sensor (also referred to as a TOF camera sensor or TOF camera) is a range imaging camera sensor system that measures the time it takes for a signal (e.g., a light pulse or electromagnetic continuous wave (CW)) to travel from the sensor to an object and back again. By measuring the time of flight of the signal, the TOF sensor can determine the distance between the sensor and the object, which allows the sensor to create a three-dimensional (3D) image of the scene or object being measured. As such, frames captured by TOF sensors can be used to estimate depth information of targets in a scene. For example, an internal computing system of an AV can use a TOF sensor to measure a distance of each pixel in a frame captured by the TOF sensor relative to the TOF sensor. The distance information can be used to obtain a representation of the spatial structure, distance, and/or geometry of a scene and/or a target in the scene.

However, one problem encountered in performing effective data capture is that the sensors can record unwanted sensor data received from the reflection of the AV itself, rather than from the surrounding environment. This unwanted sensor data (e.g., self-hit data) can deceive the AV into believing that one or more objects exist in the AV environment that do not really exist. This can cause confusion regarding the status and location of objects proximate the AV as the AV attempts to navigate the environment. As used herein, self-hit or self-hit data refers to data associated with measurements located on, or reflected from, a surface of the AV. Aspects of the disclosed technology provide solutions for identifying self-hit data, and generating and applying a mask filter to the frames captured by the sensor to filter out the self-hit data. For example, one or more TOF sensors can be configured to collect sensor data of the surrounding environment. It is understood that although a TOF sensor is provided as an example of an AV sensor herein, any number of sensors, or different sensor types, may be implemented, without departing from the disclosed technology.

In the case where TOF sensors are used, sensor data can comprise data indicating a distance between the sensor and an object based on the time it takes for a signal to travel from the sensor to the object and back again. To help distinguish between valid environmental sensor measurements (e.g., detecting objects located in the AV environment), and those measurements located on, or reflected from, a surface of AV (e.g., self-hit data), machine learning models can be used to identify AV boundaries. For example, machine learning models can be trained to identify self-hit data, and generate and apply a mask to the sensor frames to filter out unwanted self-hit data.

One or more sensors can be mounted to the exterior of the AV at fixed locations and in fixed positions. In some examples, a portion of the sensor field of view can face inward toward the body of the AV, thereby unintentionally detecting the body of the AV during each scan. Light waves that are reflected off the body of the AV itself can be considered self-hit and can be undesirable. For example, due to the highly reflective surface of AVs, light waves reflected off the AV itself (e.g., self-hit) can cause multipath reflections leading to odd and unusual artifacts in the processed frame data. The computing system of the AV can misinterpret these unwanted artifacts as physical objects located in the environment of the AV. Therefore, it can be important to identify self-hit data and filter out this unwanted self-hit data from the frames. Due to the fact that the sensors (such as, for example, TOF or LIDAR sensors) can be positioned in fixed locations about the body of the AV, the self-hit data generated by a given sensor can appear in the same location on each frame, each time the sensor completes a scan. Therefore, a machine learning model can be trained to identify artifacts within frames that appear in the same location in each and every frame as self-hit data. Machine learning models can determine the number of frames comprising an identical, static artifact that is sufficient to distinguish between a real stationary object detected in the AV environment and an artifact of self-hit data. Once the machine learning model has identified self-hit data for a particular sensor, the machine learning model can generate a fixed binary mask effectively blocking the self-hit data and subsequently apply the generated mask to the received frames to filter out the unwanted self-hit data. The process of training the machine learning model can include first providing labeled data frames to the machine learning model that include ground-truth boundaries that separate the AV from the environment. Then, subsequently updating the machine learning model based on a loss function (for example, the difference between the machine learning model's predicted boundary and the boundary indicated by the ground-truth data).

FIG.1Aillustrates an example frame100generated from a scan using a sensor mounted to the exterior of an AV. Frame100includes self-hit data101(e.g., the darkened corner) and self-hit data102(e.g., the elongated feature). That is, each time the scanner scans the location illustrated in frame100it will detect objects in that field of view, as well as a portion of the AV itself (e.g., self-hit data101and self-hit data102). In some examples, frame100is merely a single frame of hundreds (or thousands, or more) that have been produced based on sensor data received from the sensor positioned at that location of the AV. Because of the fixed nature of the location of the sensor, each and every frame of the field of view shown in frame100will show the same portion of the AV in the same location of the frame (e.g., self-hit data101and self-hit data102). A machine learning model can therefore be trained to identify these recurring objects located in the same location in every frame of this field of view (e.g., self-hit data101and self-hit data102) as undesirable self-hit data. Once the machine learning model has determined the location of self-hit data for a particular sensor, a machine learning model can generate and apply a binary mask to the frames (such as frame100, for example) to filter out the self-hit data, as shown inFIG.1B.

FIG.1Billustrates an example mask110that has been generated to filter out the self-hit data (e.g., self-hit data101and self-hit data102) identified in frame100. In some examples, a machine learning model can be trained to generate a binary mask based on the identified self-hit data. For example, mask110shows mask portion111and mask portion112that, when overlayed onto frame100, will block the unwanted self-hit data (e.g., self-hit data101and self-hit data102). The machine learning model can be trained to identify a boundary between the body of the AV and the environment, and that boundary can be used to generate a mask (e.g., mask110) that can be applied to subsequently collected sensor frames. In view of the static location of each sensor positioned about the AV, a mask can be generated for a particular sensor and the same mask can be applied to each and every frame of that sensor until the sensor itself is physically moved. Any method can be used to apply the generated mask110to the frames (e.g., frame100) such as, for example, multiplying the mask matrix against the collected field to zero-out the unwanted self-hit data. In some examples, the mask can be applied during operation of the AV stack, as discussed in more detail below. Another aspect of the static location of each sensor on the AV, is that a mask can be generated and maintained for each sensor mounted on the AV. For example, each sensor can have a mask associated with that specific sensor (which is different than the mask for each other sensor of the AV).

FIG.2Aillustrates an example frame200before a mask has been applied to filter out the unwanted self-hit data. For example, frame200illustrates AV201and sensor203and the surrounding point cloud data (e.g., portion210, portion211, and portion220) generated from sensor data. For example, point cloud data portion210can comprise point cloud data representing the area around the AV201, point cloud data portion211can comprise point cloud data representing reflections off the hood of AV201, and point cloud data portion220can comprise point cloud data representing an unwanted artifact that exists due the self-hit reflected off the hood of AV201. This unwanted artifact220can confuse the AV into believing that objects exist in the environment that are not really there.FIG.2Billustrates an example frame250after a mask has been applied to filter out the unwanted self-hit data. As shown inFIG.2B, the unwanted artifact (e.g., point cloud data portion220) is no longer present in the frame. The lack of this unwanted artifact (e.g., point cloud data portion220) can be important for the AV to accurately understand the environment around it when navigating the environment.

In some examples, it can be helpful to perform a mask dilation after the mask has been generated to account for slight errors in sensor calibration and/or slight unintended movement of the sensor as it travels through an environment. In some cases, mask dilation can be the process of increasing the size of the mask (by a few extra pixels) within the frame mask to account for these errors and/or unintended movement of the sensor. For example, as the AV travels along a roadway it can hit a large pothole that can bump the location of one or more sensors positioned on the AV. Dilating the mask can account for these types of minor adjustments to the sensors. In some scenarios, however, a sensor can be moved far enough from its original location that even a dilated mask may not account for all of the self-hit data that the sensor is receiving due to the movement of the sensor. In a situation where the sensor is receiving self-hit data even after applying a dilated mask, a machine learning model can detect that the sensor is receiving unwanted self-hit data by detecting a static portion of the frame that does not move within the frame (even as the AV continues to traverse the environment). In this case, the machine learning model can detect the self-hit data on the fly, and subsequently generate and apply a new binary mask based on the detection of the new self-hit data.

FIG.3illustrates a flow diagram of an example method300for detecting and filtering self-hit data of a sensor using a mask. At block302, the process300can include using a sensor to collect road data. For example, one or more TOF sensors can be configured to collect sensor data of the surrounding environment. It is understood that although a TOF sensor is provided as an example of an AV sensor herein, any number of sensors, or different sensor types, may be implemented, without departing from the disclosed technology (such as, for example, LIDAR sensors). In the case where TOF sensors are used, sensor data can comprise data indicating a distance between the sensor and an object based on the time it takes for a signal to travel from the sensor to the object and back again.

At block304, the process300can include identifying self-hit data within frames captured by the sensor. For example, a machine learning model can be trained to identify artifacts within a frame that appear in the same location in each and every other frame of the same field of view of the sensor as self-hit data. Due to the fact that the sensors are generally positioned in fixed locations about the body of the AV, the self-hit data generated by a given sensor can appear in the same location on each frame, each time the sensor completes a scan. Machine learning models can determine the number of frames comprising an identical, static artifact that is sufficient to distinguish between a real stationary object detected in the AV environment and an artifact of self-hit data.

At block306, the process300can include generating a mask to filter out the unwanted self-hit data. For example, once the machine learning model has identified self-hit data for a particular sensor (e.g., block304), the machine learning model can generate a fixed binary mask effectively blocking the self-hit data. In some examples, a machine learning model can be trained to generate a binary mask based on the identified self-hit data. For example, as discussed with reference toFIG.1B, a mask can be generated for a sensor and the same mask can be applied to each and every frame of that sensor until the sensor itself is physically moved or it is determined that the mask needs to be recomputed, for example, due to changes in sensor calibration.

At block308, the process300can include dilating the mask generated at block306. As explained above, it can be helpful to perform a mask dilation after the mask has been generated to account for slight errors in sensor calibration and/or slight unintended movement of the sensor as it travels through an environment. In some cases, mask dilation can be the process of increasing the size of the mask (by a few extra pixels) within the frame mask to account for these errors and/or unintended movement of the sensor. For example, as the AV travels along a roadway it can hit a large pothole that can bump the location of one or more sensors positioned on the AV. Dilating the mask can account for these types of minor adjustments to the sensors.

At block310, the process300can include applying the mask generated at block306(and, in some examples, dilated at block308) to the frames captured by the sensor. Any method can be used to apply the generated mask to the frames such as, for example, multiplying the mask matrix against the collected field to zero-out the unwanted self-hit data. In some examples, the mask can be applied during operation of the AV stack, as discussed in more detail below. Applying the mask will filter out the unwanted self-hit data as described with reference toFIGS.2A and2B. As shown inFIG.2B, the unwanted artifact220is no longer present in the frame. The lack of this unwanted artifact220can be important for the AV to accurately understand the environment around it when navigating the environment. As explained with regard to block308, in some examples, the mask can be dilated prior to being applied to the frames.

At block312, the process300can include monitoring the frames received based on the sensor data to determine if any persistent artifacts exist in the frames that can be classified as self-hit data. Similar to identifying self-hit data (e.g., block304), this monitoring can identify self-hit data on-the-fly, as the AV traverses the environment. In some examples, an AV can be bumped (e.g., hit a pothole, collision with another object, etc.) and the mounted sensor's position can be altered. In this scenario, the self-hit data for that sensor will need to be identified again. Machine learning models can be implemented to monitor the frames to determine if any persistent artifacts exist in the frames that can be classified as self-hit data. If persistent artifacts exist (e.g., self-hit data), the process300can return to block306to generate a mask based on the newly identified self-hit data, the mask can be dilated (e.g., block308), the mask can be applied (e.g., block308), and the process can continue in the manner illustrated inFIG.3. In the scenario where no persistent artifacts are found in the frames (at block312), the process can continue monitoring the frames received to determine if any persistent artifacts are detected in the frames that can be classified as self-hit data to account for any inadvertent movement of the sensor.

FIG.4illustrates a process400for identifying self-hit sensor data and generating a mask to filter out the self-hit sensor data. At block402, the process400can include collecting first sensor data for an environment around an AV. For example, one or more TOF sensors can be configured to collect sensor data of the environment surrounding the AV. As discussed above, any number of sensors, or different sensor types, may be implemented, without departing from the disclosed technology. In the case where TOF sensors are used, sensor data can comprise data indicating a distance between the sensor and an object based on the time it takes for a signal to travel from the sensor to the object and back again.

At block404, the process400can include identifying one or more data points, from the collected first sensor data, that correspond with a surface of the AV. In some examples, the one or more data points that correspond with a surface of the AV can be self-hit data. As discussed above, self-hit data can refer to data associated with measurements located on, or reflected from, a surface of the AV. To help distinguish between valid environmental sensor measurements (e.g., point clouds detecting objects location in the AV environment), and the one or more data points that correspond with a surface of the AV (e.g., self-hit data), machine learning models can be trained. For example, machine learning models can be trained to identify self-hit data. Since the sensors are positioned in fixed locations about the body of the AV, the self-hit data generated by a given sensor can appear in the same location on each frame, each time the sensor completes a scan. Therefore, in some scenarios, a machine learning model can be trained to identify artifacts within frames that appear in the same location in each and every frame as self-hit data.

At block406, the process400can include generating a mask representing the one or more data points that correspond with the surface of the AV. Once the machine learning model has determined the location of self-hit data for a sensor, a machine learning model can generate a binary mask to the frames (such as frame100illustrated inFIG.1A, for example) to filter out the self-hit data. For example, as illustrated inFIG.1B, an example mask110can be generated to filter out the self-hit data (e.g., self-hit data101and self-hit data102) identified in frame100. In some examples, a machine learning model can be trained to generate a binary mask based on the identified self-hit data. For example, mask110shows mask portion111and mask portion112that, when overlayed onto frame100, will block the unwanted self-hit data (e.g., self-hit data101and self-hit data102). In view of the static location of each sensor on the AV, a mask can be generated for a sensor and the same mask can be applied to each and every frame of that sensor until the sensor itself is physically moved.

At block408, the process400can include collecting second sensor data for the environment around the AV. The sensors collect data and measurements that the AV can use for operations such as navigation. The sensors can provide the data and measurements to an internal computing system of the AV, which can use the data and measurements to control a mechanical system of the autonomous vehicle, such as a vehicle propulsion system, a braking system, or a steering system. At block410, the process400can include applying the mask to the collected second sensor data. For example, machine learning models can be trained to identify self-hit data, and generate and apply a mask to the sensor frames to filter out unwanted self-hit data. Any method can be used to apply the generated mask to newly captured frames (e.g., second sensor data for the environment around the AV) such as, for example, pixel-wise multiplication to remove self-hit artifacts from the collected frame. In some examples, the mask can be applied during operation of the AV stack, as discussed in more detail below. The application of the mask to the collected second sensor data can result in filtering out the unwanted identified one or more data points that correspond with a surface of the AV (e.g., self-hit data).

InFIG.5, the disclosure now turns to a further discussion of models that can be used through the environments and techniques described herein.FIG.5is an example of a deep learning neural network500that can be used to implement all or a portion of the systems and techniques described herein (e.g., neural network500can be used to implement a model for detecting and filtering self-hit data of a sensor using a mask as discussed above). An input layer520can be configured to receive sensor data and/or data relating to an environment surrounding an AV. Neural network500includes multiple hidden layers522a,522b, through522n. The hidden layers522a,522b, through522ninclude “n” number of hidden layers, where “n” is an integer greater than or equal to one. The number of hidden layers can be made to include as many layers as needed for the given application. Neural network500further includes an output layer521that provides an output resulting from the processing performed by the hidden layers522a,522b, through522n.

Information can be exchanged between nodes through node-to-node interconnections between the various layers. Nodes of the input layer520can activate a set of nodes in the first hidden layer522a. For example, as shown, each of the input nodes of the input layer520is connected to each of the nodes of the first hidden layer522a. The nodes of the first hidden layer522acan transform the information of each input node by applying activation functions to the input node information. The information derived from the transformation can then be passed to and can activate the nodes of the next hidden layer522b, which can perform their own designated functions. Example functions include convolutional, up-sampling, data transformation, and/or any other suitable functions. The output of the hidden layer522bcan then activate nodes of the next hidden layer, and so on. The output of the last hidden layer522ncan activate one or more nodes of the output layer521, at which an output is provided. In some cases, while nodes in the neural network500are shown as having multiple output lines, a node can have a single output and all lines shown as being output from a node represent the same output value.

In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network500. Once the neural network500is trained, it can be referred to as a trained neural network, which can be used to classify one or more activities. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable numeric weight that can be tuned (e.g., based on a training dataset), allowing the neural network500to be adaptive to inputs and able to learn as more and more data is processed.

The neural network500is pre-trained to process the features from the data in the input layer520using the different hidden layers522a,522b, through522nin order to provide the output through the output layer521.

In some cases, the neural network500can adjust the weights of the nodes using a training process called backpropagation. A backpropagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter/weight update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training data until the neural network500is trained well enough so that the weights of the layers are accurately tuned.

To perform training, a loss function can be used to analyze error in the output. Any suitable loss function definition can be used, such as a Cross-Entropy loss. Another example of a loss function includes the mean squared error (MSE), defined as E_total=Σ(½ (target-output){circumflex over ( )}2). The loss can be set to be equal to the value of E_total.

The loss (or error) will be high for the initial training data since the actual values will be much different than the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training output. The neural network500can perform a backward pass by determining which inputs (weights) most contributed to the loss of the network, and can adjust the weights so that the loss decreases and is eventually minimized.

As understood by those of skill in the art, machine-learning based classification techniques can vary depending on the desired implementation. For example, machine-learning classification schemes can utilize one or more of the following, alone or in combination: hidden Markov models; RNNs; CNNs; deep learning; Bayesian symbolic methods; Generative Adversarial Networks (GANs); support vector machines; image registration methods; and applicable rule-based systems. Where regression algorithms are used, they may include but are not limited to: a Stochastic Gradient Descent Regressor, a Passive Aggressive Regressor, etc.

In this example, the AV environment600includes an AV602, a data center650, and a client computing device670. The AV602, the data center650, and the client computing device670can communicate with one another over one or more networks (not shown), such as a public network (e.g., the Internet, an Infrastructure as a Service (IaaS) network, a Platform as a Service (PaaS) network, a Software as a Service (SaaS) network, other Cloud Service Provider (CSP) network, etc.), a private network (e.g., a Local Area Network (LAN), a private cloud, a Virtual Private Network (VPN), etc.), and/or a hybrid network (e.g., a multi-cloud or hybrid cloud network, etc.).

The AV602can navigate roadways without a human driver based on sensor signals generated by multiple sensor systems604,606, and608. The sensor systems604-608can include one or more types of sensors and can be arranged about the AV602. For instance, the sensor systems604-608can include Inertial Measurement Units (IMUs), cameras (e.g., still image cameras, video cameras, etc.), light sensors (e.g., LIDAR systems, ambient light sensors, infrared sensors, etc.), RADAR systems, GPS receivers, audio sensors (e.g., microphones, Sound Navigation and Ranging (SONAR) systems, ultrasonic sensors, etc.), engine sensors, speedometers, tachometers, odometers, altimeters, tilt sensors, impact sensors, airbag sensors, seat occupancy sensors, open/closed door sensors, tire pressure sensors, rain sensors, and so forth. For example, the sensor system604can be a camera system, the sensor system606can be a LIDAR system, and the sensor system608can be a RADAR system. Other examples may include any other number and type of sensors.

The AV602can also include several mechanical systems that can be used to maneuver or operate the AV602. For instance, the mechanical systems can include a vehicle propulsion system630, a braking system632, a steering system634, a safety system636, and a cabin system638, among other systems. The vehicle propulsion system630can include an electric motor, an internal combustion engine, or both. The braking system632can include an engine brake, brake pads, actuators, and/or any other suitable componentry configured to assist in decelerating the AV602. The steering system634can include suitable componentry configured to control the direction of movement of the AV602during navigation. The safety system636can include lights and signal indicators, a parking brake, airbags, and so forth. The cabin system638can include cabin temperature control systems, in-cabin entertainment systems, and so forth. In some examples, the AV602might not include human driver actuators (e.g., steering wheel, handbrake, foot brake pedal, foot accelerator pedal, turn signal lever, window wipers, etc.) for controlling the AV602. Instead, the cabin system638can include one or more client interfaces (e.g., Graphical User Interfaces (GUIs), Voice User Interfaces (VUIs), etc.) for controlling certain aspects of the mechanical systems630-638.

The AV602can include a local computing device610that is in communication with the sensor systems604-608, the mechanical systems630-638, the data center650, and the client computing device670, among other systems. The local computing device610can include one or more processors and memory, including instructions that can be executed by the one or more processors. The instructions can make up one or more software stacks or components responsible for controlling the AV602; communicating with the data center650, the client computing device670, and other systems; receiving inputs from riders, passengers, and other entities within the AV's environment; logging metrics collected by the sensor systems604-608; and so forth. In this example, the local computing device610includes a perception stack612, a localization stack614, a prediction stack616, a planning stack618, a communications stack620, a control stack622, an AV operational database624, and an HD geospatial database626, among other stacks and systems.

Perception stack612can enable the AV602to “see” (e.g., via cameras, LIDAR sensors, infrared sensors, etc.), “hear” (e.g., via microphones, ultrasonic sensors, RADAR, etc.), and “feel” (e.g., pressure sensors, force sensors, impact sensors, etc.) its environment using information from the sensor systems604-608, the localization stack614, the HD geospatial database626, other components of the AV, and other data sources (e.g., the data center650, the client computing device670, third party data sources, etc.). The perception stack612can detect and classify objects and determine their current locations, speeds, directions, and the like. In addition, the perception stack612can determine the free space around the AV602(e.g., to maintain a safe distance from other objects, change lanes, park the AV, etc.). The perception stack612can identify environmental uncertainties, such as where to look for moving objects, flag areas that may be obscured or blocked from view, and so forth. In some examples, an output of the perception stack612can be a bounding area around a perceived object that can be associated with a semantic label that identifies the type of object that is within the bounding area, the kinematic of the object (information about its movement), a tracked path of the object, and a description of the pose of the object (its orientation or heading, etc.).

Localization stack614can determine the AV's position and orientation (pose) using different methods from multiple systems (e.g., GPS, IMUs, cameras, LIDAR, RADAR, ultrasonic sensors, the HD geospatial database626, etc.). For example, in some cases, the AV602can compare sensor data captured in real-time by the sensor systems604-608to data in the HD geospatial database626to determine its precise (e.g., accurate to the order of a few centimeters or less) position and orientation. The AV602can focus its search based on sensor data from one or more first sensor systems (e.g., GPS) by matching sensor data from one or more second sensor systems (e.g., LIDAR). If the mapping and localization information from one system is unavailable, the AV602can use mapping and localization information from a redundant system and/or from remote data sources.

Prediction stack616can receive information from the localization stack614and objects identified by the perception stack612and predict a future path for the objects. In some examples, the prediction stack616can output several likely paths that an object is predicted to take along with a probability associated with each path. For each predicted path, the prediction stack616can also output a range of points along the path corresponding to a predicted location of the object along the path at future time intervals along with an expected error value for each of the points that indicates a probabilistic deviation from that point.

Planning stack618can determine how to maneuver or operate the AV602safely and efficiently in its environment. For example, the planning stack618can receive the location, speed, and direction of the AV602, geospatial data, data regarding objects sharing the road with the AV602(e.g., pedestrians, bicycles, vehicles, ambulances, buses, cable cars, trains, traffic lights, lanes, road markings, etc.) or certain events occurring during a trip (e.g., emergency vehicle blaring a siren, intersections, occluded areas, street closures for construction or street repairs, double-parked cars, etc.), traffic rules and other safety standards or practices for the road, user input, and other relevant data for directing the AV602from one point to another and outputs from the perception stack612, localization stack614, and prediction stack616. The planning stack618can determine multiple sets of one or more mechanical operations that the AV602can perform (e.g., go straight at a specified rate of acceleration, including maintaining the same speed or decelerating; turn on the left blinker, decelerate if the AV is above a threshold range for turning, and turn left; turn on the right blinker, accelerate if the AV is stopped or below the threshold range for turning, and turn right; decelerate until completely stopped and reverse; etc.), and select the best one to meet changing road conditions and events. If something unexpected happens, the planning stack618can select from multiple backup plans to carry out. For example, while preparing to change lanes to turn right at an intersection, another vehicle may aggressively cut into the destination lane, making the lane change unsafe. The planning stack618could have already determined an alternative plan for such an event. Upon its occurrence, it could help direct the AV602to go around the block instead of blocking a current lane while waiting for an opening to change lanes.

Control stack622can manage the operation of the vehicle propulsion system630, the braking system632, the steering system634, the safety system636, and the cabin system638. The control stack622can receive sensor signals from the sensor systems604-608as well as communicate with other stacks or components of the local computing device610or a remote system (e.g., the data center650) to effectuate operation of the AV602. For example, the control stack622can implement the final path or actions from the multiple paths or actions provided by the planning stack618. This can involve turning the routes and decisions from the planning stack618into commands for the actuators that control the AV's steering, throttle, brake, and drive unit.

Communications stack620can transmit and receive signals between the various stacks and other components of the AV602and between the AV602, the data center650, the client computing device670, and other remote systems. The communications stack620can enable the local computing device610to exchange information remotely over a network, such as through an antenna array or interface that can provide a metropolitan WIFI network connection, a mobile or cellular network connection (e.g., Third Generation (3G), Fourth Generation (4G), Long-Term Evolution (LTE), 5th Generation (5G), etc.), and/or other wireless network connection (e.g., License Assisted Access (LAA), Citizens Broadband Radio Service (CBRS), MULTEFIRE, etc.). Communications stack620can also facilitate the local exchange of information, such as through a wired connection (e.g., a user's mobile computing device docked in an in-car docking station or connected via Universal Serial Bus (USB), etc.) or a local wireless connection (e.g., Wireless Local Area Network (WLAN), Low Power Wide Area Network (LPWAN), Bluetooth®, infrared, etc.).

AV operational database624can store raw AV data generated by the sensor systems604-608, stacks612-622, and other components of the AV602and/or data received by the AV602from remote systems (e.g., the data center650, the client computing device670, etc.). In some examples, the raw AV data can include HD LIDAR point cloud data, image data, RADAR data, GPS data, and other sensor data that the data center650can use for creating or updating AV geospatial data or for creating simulations of situations encountered by AV602for future testing or training of various machine learning algorithms that are incorporated in the local computing device610.

Data center650can include a private cloud (e.g., an enterprise network, a co-location provider network, etc.), a public cloud (e.g., an Infrastructure as a Service (IaaS) network, a Platform as a Service (PaaS) network, a Software as a Service (SaaS) network, or other Cloud Service Provider (CSP) network), a hybrid cloud, a multi-cloud, and/or any other network. The data center650can include one or more computing devices remote to the local computing device610for managing a fleet of AVs and AV-related services. For example, in addition to managing the AV602, the data center650may also support a ride-hailing service (e.g., a ridesharing service), a delivery service, a remote/roadside assistance service, street services (e.g., street mapping, street patrol, street cleaning, street metering, parking reservation, etc.), and the like.

Data center650can send and receive various signals to and from the AV602and the client computing device670. These signals can include sensor data captured by the sensor systems604-608, roadside assistance requests, software updates, ride-hailing/ridesharing pick-up and drop-off instructions, and so forth. In this example, the data center650includes a data management platform652, an Artificial Intelligence/Machine Learning (AI/ML) platform654, a simulation platform656, a remote assistance platform658, and a ride-hailing platform660, and a map management platform662, among other systems.

The AI/ML platform654can provide the infrastructure for training and evaluating machine learning algorithms for operating the AV602, the simulation platform656, the remote assistance platform658, the ride-hailing platform660, the map management platform662, and other platforms and systems. Using the AI/ML platform654, data scientists can prepare data sets from the data management platform652; select, design, and train machine learning models; evaluate, refine, and deploy the models; maintain, monitor, and retrain the models; and so on.

Simulation platform656can enable testing and validation of the algorithms, machine learning models, neural networks, and other development efforts for the AV602, the remote assistance platform658, the ride-hailing platform660, the map management platform662, and other platforms and systems. Simulation platform656can replicate a variety of driving environments and/or reproduce real-world scenarios from data captured by the AV602, including rendering geospatial information and road infrastructure (e.g., streets, lanes, crosswalks, traffic lights, stop signs, etc.) obtained from a cartography platform (e.g., map management platform662); modeling the behavior of other vehicles, bicycles, pedestrians, and other dynamic elements; simulating inclement weather conditions, different traffic scenarios; and so on.

Remote assistance platform658can generate and transmit instructions regarding the operation of the AV602. For example, in response to an output of the AI/ML platform654or other system of the data center650, the remote assistance platform658can prepare instructions for one or more stacks or other components of the AV602.

Ride-hailing platform660can interact with a customer of a ride-hailing service via a ride-hailing application672executing on the client computing device670. The client computing device670can be any type of computing system such as, for example and without limitation, a server, desktop computer, laptop computer, tablet computer, smartphone, smart wearable device (e.g., smartwatch, smart eyeglasses or other Head-Mounted Display (HMD), smart ear pods, or other smart in-ear, on-ear, or over-ear device, etc.), gaming system, or any other computing device for accessing the ride-hailing application672. The client computing device670can be a customer's mobile computing device or a computing device integrated with the AV602(e.g., the local computing device610). The ride-hailing platform660can receive requests to pick up or drop off from the ride-hailing application672and dispatch the AV602for the trip.

In some embodiments, the map viewing services of map management platform662can be modularized and deployed as part of one or more of the platforms and systems of the data center650. For example, the AI/ML platform654may incorporate the map viewing services for visualizing the effectiveness of various object detection or object classification models, the simulation platform656may incorporate the map viewing services for recreating and visualizing certain driving scenarios, the remote assistance platform658may incorporate the map viewing services for replaying traffic incidents to facilitate and coordinate aid, the ride-hailing platform660may incorporate the map viewing services into the client application672to enable passengers to view the AV602in transit en route to a pick-up or drop-off location, and so on.

While the autonomous vehicle602, the local computing device610, and the autonomous vehicle environment600are shown to include certain systems and components, one of ordinary skill will appreciate that the autonomous vehicle602, the local computing device610, and/or the autonomous vehicle environment600can include more or fewer systems and/or components than those shown inFIG.6. For example, the autonomous vehicle602can include other services than those shown inFIG.6and the local computing device610can also include, in some instances, one or more memory devices (e.g., RAM, ROM, cache, and/or the like), one or more network interfaces (e.g., wired and/or wireless communications interfaces and the like), and/or other hardware or processing devices that are not shown inFIG.6. An illustrative example of a computing device and hardware components that can be implemented with the local computing device610is described below with respect toFIG.7.

FIG.7illustrates an example processor-based system with which some aspects of the subject technology can be implemented. For example, processor-based system700can be any computing device making up, or any component thereof in which the components of the system are in communication with each other using connection705. Connection705can be a physical connection via a bus, or a direct connection into processor710, such as in a chipset architecture. Connection705can also be a virtual connection, networked connection, or logical connection.

Example system700includes at least one processing unit (Central Processing Unit (CPU) or processor)710and connection705that couples various system components including system memory715, such as Read-Only Memory (ROM)720and Random-Access Memory (RAM)725to processor710. Computing system700can include a cache of high-speed memory712connected directly with, in close proximity to, or integrated as part of processor710.

Processor710can include any general-purpose processor and a hardware service or software service, such as services732,734, and736stored in storage device730, configured to control processor710as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor710may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

Storage device730can include software services, servers, services, etc., that when the code that defines such software is executed by the processor710, it causes the system700to perform a function. In some embodiments, a hardware service that performs a particular function can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor710, connection705, output device735, etc., to carry out the function.

Illustrative Examples of the Disclosure Include

Aspect 1. A system comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to: collect first sensor data for an environment around an autonomous vehicle (AV); identify one or more data points, from the collected first sensor data, that correspond with a surface of the AV; generate an image mask representing the one or more data points that correspond with the surface of the AV; collect second sensor data for the environment around the AV; and apply the image mask to the collected second sensor data.

Aspect 2. The system of Aspect 1, further comprising: identifying the one or more data points from among the collected first sensor data as self-hit data points.

Aspect 3. The system of Aspect 1 or 2, wherein a machine learning model identifies the one or more data points that correspond with a surface of the AV.

Aspect 4. The system of any of Aspects 1 to 3, wherein the image mask is generated using a machine learning model.

Aspect 5. The system of any of Aspects 1 to 4, wherein the image mask is dilated before it is applied to the collected second sensor data.

Aspect 6. The system of any of Aspects 1 to 5, wherein the sensor data comprises time of flight (TOF) data.

Aspect 7. The system of any of Aspects 1 to 6, wherein a machine learning model determines a within a frame between the AV and the environment.

Aspect 8. A method comprising: collecting first sensor data for an environment around an autonomous vehicle (AV); identifying one or more data points, from the collected first sensor data, that correspond with a surface of the AV; generating an image mask representing the one or more data points that correspond with the surface of the AV; collecting second sensor data for the environment around the AV; and applying the image mask to the collected second sensor data.

Aspect 9. The method of Aspect 8, further comprising: identifying the one or more data points from among the collected first sensor data as self-hit data points.

Aspect 10. The method of Aspect 8 or 9, wherein a machine learning model identifies the one or more data points that correspond with a surface of the AV.

Aspect 11. The method of any of Aspects 8 to 10, wherein the image mask is generated using a machine learning model.

Aspect 12. The method of any of Aspects 8 to 11, wherein the image mask is dilated before it is applied to the collected second sensor data.

Aspect 13. The method of any of Aspects 8 to 12, wherein the sensor data comprises time of flight (TOF) data.

Aspect 14. The method of any of Aspects 8 to 13, wherein a machine learning model determines a within a frame between the AV and the environment.

Aspect 15. A non-transitory computer-readable storage medium comprising at least one instruction for causing a computer or processor to: collect first sensor data for an environment around an autonomous vehicle (AV); identify one or more data points, from the collected first sensor data, that correspond with a surface of the AV; generate an image mask representing the one or more data points that correspond with the surface of the AV; collect second sensor data for the environment around the AV; and apply the image mask to the collected second sensor data.

Aspect 16. The non-transitory computer-readable storage medium of Aspect 15, further comprising: identifying the one or more data points from among the collected first sensor data as self-hit data points.

Aspect 17. The non-transitory computer-readable storage medium of Aspect 15 or 16, wherein a machine learning model identifies the one or more data points that correspond with a surface of the AV.

Aspect 18. The non-transitory computer-readable storage medium of any of Aspects 15 to 17, wherein the image mask is generated using a machine learning model.

Aspect 19. The non-transitory computer-readable storage medium of any of Aspects 15 to 18, wherein image mask is dilated before it is applied to the collected second sensor data.

Aspect 20. The non-transitory computer-readable storage medium of any of Aspects 15 to 19, wherein the sensor data comprises time of flight (TOF) data.

The various embodiments described above are provided by way of illustration only and should not be construed to limit the scope of the disclosure. For example, the principles herein apply equally to optimization as well as general improvements. Various modifications and changes may be made to the principles described herein without following the example embodiments and applications illustrated and described herein, and without departing from the spirit and scope of the disclosure.