AUTONOMOUS VEHICLE PERCEPTION MULTIMODAL SENSOR DATA MANAGEMENT

The automated driving perception systems described herein provide technical solutions for technical problems facing navigation sensors for autonomous vehicle navigation. These systems may be used to combine inputs from multiple navigation sensors to provide a multimodal perception system. These multimodal perception systems may augment raw data within a development framework to improve performance of object detection, classification, tracking, and sensor fusion under varying external conditions, such as adverse weather and light, as well as possible sensor errors or malfunctions like miss-calibration, noise, and dirty or faulty sensors. This augmentation may include injection of noise, occlusions, and misalignments from raw sensor data, and may include ground-truth labeling to match the augmented data. This augmentation provides improved robustness of the trained perception algorithms against calibration, noise, occlusion, and faults that may exist in real-world scenarios.

TECHNICAL FIELD

Embodiments described herein generally relate to autonomous vehicle sensors and sensor data processing.

BACKGROUND

Autonomous vehicles may be used to provide transportation without requiring full driver (e.g., operator) control. Fully autonomous vehicles may be used to navigate to a destination without any driver input while avoiding pedestrians, other vehicles, and other obstacles. Partially autonomous vehicles may receive a control input from a driver and may modify the vehicle control (e.g., steering, braking) to augment the navigation to a destination. These autonomous vehicles may identify and avoid obstacles using one or more input navigation sensors, such as an image capture device (e.g., camera), Light Detection and Ranging System (LiDAR), and RADAR. However, these navigation sensors are often subject to degraded performance under varying external conditions (e.g., adverse weather, varying light conditions) or under sensor errors or malfunctions (e.g., miss-calibration, noise, dirty sensors, faulty sensors).

DETAILED DESCRIPTION

The automated driving perception systems described herein provide technical solutions for technical problems facing navigation sensors for autonomous vehicle navigation. These systems may be used to improve the performance of individual navigation sensors, and may be used to combine inputs from multiple navigation sensors to provide a multimodal perception system. In an example, image capture devices (e.g., cameras) may be used to generate an image dataset, and ranging devices (e.g., LiDAR, RADAR) may be used to generate a ranging dataset (e.g., point cloud dataset). These multimodal perception systems may make use of overlapping fields of view when placing sensors instrumentation to provide independent inputs in every surrounding region, which may be used to provide omnidirectional detection of vehicles and other objects around the vehicle.

These multimodal perception systems may work to improve or guarantee complete environmental sensing and robust perception performance under varying external conditions, such as adverse weather and light, as well as possible sensor errors or malfunctions like miss-calibration, noise, and dirty or faulty sensors. This difference between performance under ideal conditions and performance under adverse weather or sensor conditions may be referred to as a domain gap. To improve perception system tasks of localization, object detection, object classification, and sensor fusion (e.g., combining various navigation sensors), the multimodal perception systems described herein are designed to identify and minimize or eliminate perception degradation due to these external conditions or sensor errors.

In an example, these multimodal perception systems may improve performance of object detection, classification, tracking, and sensor fusion by augmenting raw data (e.g., datasets) within a development framework. This augmentation may include injection of noise, occlusions, and misalignments from raw sensor data, and may include ground-truth labeling to match the augmented data. This augmentation provides improved robustness of the trained perception algorithms against calibration, noise, occlusion, and faults that may exist in real-world scenarios. When using these multimodal perception systems within a development framework, designers of automated driving perception systems may programmatically inject sensor errors during a machine learning training phase. Training perception models based on the datasets with injected errors may be used to improve the ability of data-driven machine learning solutions to operate (e.g., correctly identify objects) in the presence of adverse conditions that are likely to occur.

FIG. 1is a pictorial drawing illustrating a multimodal perception system output100, according to an embodiment. Images105through120illustrate the effect of artificially induced rain on object detection. Typically rain or other weather effects are introduced during inclement weather, though weather conditions may also be introduced through a navigation system attack (e.g., navigation cyberattack). A clear input object detection image105may include one or more vehicles, which may be identified by bounding boxes as shown in clear output object detection image115. The clear input object detection image105may be modified to include synthetically generated physics-based rain drops, as shown in rainy input object detection image110. The artificially induced rain may reduce the performance of object detection, as can be seen in the reduced number of vehicles identified in rainy output object detection image120.

Images125through140illustrate the effect of artificially induced rain on semantic segmentation. A clear input semantic segmentation image125may include one or more vehicles, road areas, foliage areas, and other object areas, which may be identified by patterns, colors, or other region indications as shown in clear output semantic segmentation image135. The clear input semantic segmentation image125may be modified to include synthetically generated physics-based rain drops, as shown in rainy input semantic segmentation image130. The artificially induced rain may reduce the performance of semantic segmentation, as can be seen in the differences between the regions in the clear output semantic segmentation image135and the rainy output semantic segmentation image140. The present multimodal perception systems may be used to train and validate perception models to improve the performance of the perception models under adverse weather conditions, such as shown inFIG. 2.

FIG. 2is a block diagram illustrating a multistage perception system training200, according to an embodiment. Training200may include application of two or more adverse weather conditions in a multistage self-supervised machine learning environment. Training200may include a first stage205, which pre-trains a machine learning model on an automated driving perception system dataset that includes clear weather. In an example, for each source image220in the dataset, the first stage205may generate a segmented and labeled output225. Training200may include a second stage210, which may include training on a dataset with artificially induced adverse weather. The second stage210may include two-step self-supervised learning230, such as self-supervised source-free domain adaptation supervised learning. This two-step self-supervised learning230may include receiving target images235with artificially induced adverse weather, then using target images235to generate pseudo-label images240generated after initialization within the first stage205. Training200may include a third stage215, which may include fine-tuning the labeled images with K labeled images to receive fine-tuning input image245to generate fine-tuning labeled image250. Training200may apply this two-step self-supervised learning on a pretrained good-weather model to improve automatic label generation.

FIG. 3is a block diagram illustrating a multistage perception system environment300, according to an embodiment. Environment300provides an overview of the data manipulation and obfuscation modules that may be used within the present multimodal perception system. Environment300may include a multistage perception system framework310. The framework310may receive one or more perception datasets302via a data loader312, which may be accessible via a dataset application programming interface (API)340. Each data frame314may include one or more input images, such as images from multiple cameras in various locations on a vehicle. Framework310may include data augmentation324, which may include one or more of a signal augmentation326, a transform augmentation328, a noise augmentation330, a cross-dataset augmentation332, and an annotation augmentation334. The data augmentation324is described in greater detail below with respect toFIGS. 4A-11.

Environment300may include a perception module under test356. This perception module under test356may be used by framework310to provide training338, and may be accessible via a training API346. This perception module under test356may include a multimodal 3D object detection machine learning network358. In an example, the multimodal 3D object detection machine learning network358may be implemented as an artificial neural network (ANN), and more specifically may be implemented as a deep neural network (DNN) with multiple layers between the input and output image layers. Network358may receive one or more point cloud inputs360, such as point clouds (e.g., ranging dataset) generated by a LiDAR or RADAR sensor, and may receive one or more corresponding input images362that correspond to each of the point cloud inputs360. Independent feature extraction364may be used to apply a point cloud feature extraction366to each of the point cloud inputs360to generate an extracted point cloud feature output372, and similarly apply an image feature extraction368to each of the image inputs362to generate an extracted image feature output374. Sensor fusion376may receive and combine both the extracted point cloud feature output372and the extracted image feature output374to improve or maximize fault tolerance. A detection network380may receive combined data from the sensor fusion376and detect various objects or other features, which may be used to generate one or more of a bounding box output382, an image output384, a class classifier output386, or an aleatoric uncertainty (e.g., statistical uncertainty) estimator output388.

Environment300may include a test result output350. This test result output350may be used by framework310to provide evaluation336, and may be accessible via an evaluation API344. Test result350may generate various raw data or data plots to analyze the training or model performance, such as accuracy as a function of point removal352or accuracy as a function of image erasing354.

In an example, a user348may access various features within environment300using various APIs, such as dataset API340, framework API342, evaluation API344, or training API346. The user348may use these APIs to implement or test various features within this multistage perception system, such as application of data augmentation or data obfuscation via error injection and transformation used during model training tasks, which may be used to train perception models. These APIs also provide access to metrics for evaluation of the robustness under the programmatic fault injection methods, which may be used to validate the performance of perception models.

FIGS. 4A-4Dare pictorial drawings illustrating sensor signal augmentation, according to an embodiment. To interface one or more datasets with a training framework of a selected perception model, a user may specify how the dataset input is to be converted to match an input type and shape expected by the neural network architecture of the selected perception model. Various datasets may contain sensor data in various formats, and input signal modifications may be used to modify the input dataset to match the input type and shape of the target perception model. In addition, modifications to the input dataset may be accompanied by corresponding updates to the ground-truth annotations of the dataset to ensure the annotations are equally transformed into the expected targets of the perception model.

Augmentation400may include an augmentor transformation. The augmentor transformation may produce an output object (e.g., output image) with augmented characteristics, such as resizing an image, reducing a field of view of a LiDAR point cloud, adding transformations to a transform tree, or adding noise to an image or LiDAR point cloud. As shown inFIGS. 4A-4D, augmentation may include one or more annotated image operations, such a resizing, cropping, or other operations.FIG. 4Ashows an annotated input intersection image400, which may include pedestrians437,488,417,441,411,423,444,409and one or more cars406,439,449. In an example, annotated input intersection image400represents a dataset generated by multiple navigation sensors that have been combined and converted into a 2D top view image of an intersection. In an example, this annotated input intersection image400may include an annotated image of a first size (e.g., 1600×1600 pixels), and it may need to be converted to a target size of a second, smaller size (e.g., 800×600 pixels) for use in a target detection neural network architecture.FIG. 4Bshows an example cropped image405, which shows an example cropping (not necessarily to scale) from the 1600×1600 pixels of the annotated input intersection image400down to the target size of 800×600 pixels. As can be seen inFIG. 4B, information outside of the specified target dimension may be lost when only using this cropping function.

FIG. 4Cshows an example resized image410, which shows an example resizing (not necessarily to scale) of the 1600×1600 pixels of the annotated input intersection image400down to the target size of 800×600 pixels. As can be seen in comparingFIG. 4AandFIG. 4C, this resize operation may result in some distortions of objects in the original image.FIG. 4Dshows an example resized image415, which shows an example cropping and resizing (not necessarily to scale) of the 1600×1600 pixels of the annotated input intersection image400down to the target size of 800×600 pixels. This cropping and resizing may include expanding the cropping frame in the target size ratio (e.g., 4:3 ratio cropping frame) to the widest extent, cropping pixels outside of this region, and then resizing the resulting image down to the target size of 800×600 pixels. The use of the combined cropping and resizing operation may improve or maximize the amount of captured information when converting a dataset into a target size dataset.

Additional dataset augmentation may be used to support a conversion of an input dataset to conform to the input requirements of a target neural network architecture. A dataset transformer may be used to define a mapping from an input dataframe to input tensors of the target neural network. In an example, the dataset transformer may return channel data for each input dataframe into a specified format for the target neural network, such as a NumPy array, a PyTorch tensor, or other format. A dataset target generator may be used to define a mapping from an input dataframe to learning targets of the target neural network. In an example, for 3D object detection models, the dataset target generator may return a list with data values (e.g., x, y, z, length, width, height, yaw) of the annotations in the input dataframe. Similarly, for 2D object detection models, the dataset target generator may return a list with data values of corresponding bounding boxes visible on an input image (e.g., x, y, width, height). The dataset target generator may also include a target generator threshold to update labels to the format of the target neural network, which provides the ability to train the dataset using the updated labels that map to the desired input format of the target neural network.

FIG. 5is a block diagram illustrating data preparation500, according to an embodiment. Data preparation500may be used to select and prepare a dataset for a target neural network. In an example, a user512may select one or more datasets502, select a sampler504, generate dataset splits506, assign augmentors, transformers, and target generators508, and trigger data loading or a training loop510. The augmentors may include one or more of the augmentor transformation, dataset transformer, and dataset target generator described above. The user may select one or more of these assets from an asset library514, which may include a dataset library516, a sampler library518, an augmentor library520, a transformer library522, and a target generator library524.

To prepare a dataset for training and validation, one or more datasets may be retrieved from data storage526to generate a multimodal dataset or data superset528. A dataset sampler530may be used to sample the data. The data may be split into training data532and validation data542. The training data532may include a multimodal subset training dataset534, which may be modified using one or more of a training augmentor536, a training transformer538, or a training target generator540. Similarly, the validation data542may include a multimodal subset validation dataset544, which may be modified using one or more of a validation augmentor546, a validation transformer548, or a validation target generator550.

The generation of the multimodal dataset or data superset528may include application of one or more cross-dataset operations to create joint datasets or resampled data subsets. The dataset sampler530may include a multimodal subset operation to define a subset of the multimodal dataset or data superset528. In an example, the multimodal subset operation includes sampling the dataset to create the multimodal subset training dataset534and the multimodal subset validation dataset544, and augmentors536and546, transformers538and548, and target generators540and550are applied to each data subset.

The generation of the multimodal data superset528may include application of a multimodal superset operation to combine multiple multimodal datasets into a single multimodal superset. This multimodal superset may be used to improve cross-dataset analysis and evaluation. This multimodal superset may also be sampled to generate one or more subsets, such as to create a dataset specific to a region (e.g., USA, Europe). The generation of the multimodal data superset528may include application of a dataset sampler operation to iterate over multiple dataframes within a dataset to generate a descriptor for one or more dataframes. This descriptor may be used to improve balance in a dataset, such as by generating one or more balanced subsets. In an example, the descriptor characterize a distribution of a detected object count (e.g., number of cars, number of pedestrians) within a source dataset, and the descriptor may be used to generate a training data subset and a validation data subset that each reflect the same distribution of detected object counts as in the source dataset.

Training data preparation552may be used to prepare each dataframe554. In an example, an augmentor is applied to each dataframe554to generate an augmented dataframe556, a transformer is applied to generate model inputs558, and a target generator is applied to generate learning targets560.

FIG. 6is a pictorial drawing illustrating a perception sensor obfuscation600, according to an embodiment. In addition to input dataset cropping and resizing transformations matched with ground truth labels, perception sensor obfuscation600may be used to obfuscate the input signal by adding noise to an image or LiDAR point cloud (e.g., ranging dataset).FIG. 6depicts a scene captured by a LiDAR point cloud and an image capture device, where the calibrated captured image and point cloud are extracted from a multimodal dataset and calibrated, overlapped, and output to provide an image of a common scene area. A LiDAR region occlusion is shown in point cloud removal output605, which shows a removal of points from the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image removal output615, which shows a masking of pixels from the source image in one or more regions. A LiDAR noise occlusion is shown in point cloud noise output610, which shows an injection of noise (e.g., Gaussian noise) into the source LiDAR point cloud in a LiDAR range scanning region. An image region occlusion is shown in image noise output620, which shows an injection of noise into the source image in one or more regions.

FIG. 7is a flowchart illustrating a noise injection training environment700, according to an embodiment. The training environment700shows the noise injection process as part of the training loop740of the perception algorithm. One or more datasets705may be loaded by data loading module710. These input datasets may be split, augmented, or transformed in training data preparation715to generate a training dataset. Noise injection720may apply one or more of a region occlusion or noise occlusion to the training dataset. This noise-injected dataset may be used within the perception model735to detect one or more objects, regions of interest, or other detected features within the dataset. The perception model735generates an inference output that is used to determine perception performance metrics725. These metrics are used in loss function back-propagation730to generate weight updates that are used within the perception model735on subsequent epochs. The performance metrics may also be used by noise injection720to modify the type, magnitude, area, or other characteristics of the injected noise.

Noise injection720may include application of data occlusion, which may receive input region ranges (e.g., region minimum, region maximum) that define lower and upper bounds of an area to be occluded on the dataset sensor field. Placement of occlusions may be randomized or guided by a user-defined function. The use of user-defined occlusions may be used to place the obfuscation on critical areas, such as a driving path of the vehicle or near an area of a particular object to be detected. The center location of the bounding boxes of ground truth data labels may be used to bias automatic occlusion generation, such as for generating region occlusions or injected noise.

Noise injection720may include application of data noise, which may be used to define a type of noise to be used and noise type configuration parameters. In an example, the noise type may include Gaussian noise and may take parameters including mean and standard deviation. In another example, the noise type may include Perlin noise and may take parameters including octaves and seed. In yet another example, the noise type may include open simplex noise and may take parameters including seed and dimensions.

Noise injection720may include application of a dataset sampler to provide temporal noise injection. The effect of noise on an input perception dataset depends on the quality and quantity of noise (e.g., Gaussian noise in a specified region), but also on the temporal duration of the noise. Noise injection720may be used to define when to inject noise within a scene and a duration for noise injection, which may be used to improve the performance of perception models in the presence of sporadic or constant noise artifacts. A dataset sampler may provide temporal balancing of noise injection, which may include one or more of a sequential dataset sampler, a sequential subset sampler, and a temporal noise generator sampler. This dataset sampler may provide improved control over sequential order of temporal perception data, and may be used to improve balance and portioning of training data for the model and determination of the temporal aspects of the noise injection.

The dataset sampler used by noise injection720may include application of a sequential dataset sampler, which may be used to organize selected input data in a sequential manner. This sequential dataset sampler may be used to provide ordered scenes across one or multiple datasets. The sequential dataset sampler may generate an ordered dictionary of scenes, where each scene includes a sequence of consecutive frames in a common location. A dataset dictionary may be used to provide an automatic mapping between datasets to minimize or eliminate repeated timestamp sequences.

The dataset sampler used by noise injection720may include application of a sequential subset sampler, which may be used to determine a subset of data based on provided attributes. This sequential subset sampler may be used to sample a set of scenes and generate a data subset using only scenes with an associated minimum length (e.g., minimum number of seconds, minimum number of frames) or using only scenes with certain environmental conditions (e.g., rain, fog, snow) indicated within dataframe metadata. In an example, scenes may be selected based on whether they include sequences with specified road actors (e.g., pedestrians, bicyclists), whether they include vegetation (e.g., trees, bushes, grass), or whether they include a particular type of weather (e.g., rainy, cloudy, cloudless). This sequential subset sampler may receive an ordered dictionary (e.g., an output of sequential dataset sampler) and the subset features identified by keyword and value (e.g., as a tuple data type), and may generate a subset of sequences matching some or all of the identified subset features.

The dataset sampler used by noise injection720may include application of a temporal noise generator sampler, which may be used to control the temporal characteristics of the noise injection on a selected training dataset scene. The temporal noise generator sampler may be used to define an approach for application of a particular type of noise as random, balanced, constant, or guided. The random noise approach randomly determines a start and duration of the noise injection in the scene. The balanced noise approach takes into consideration the varying lengths of each scene, and injects a substantially consistent magnitude of noise across the different scenes at different time segments (e.g., beginning, middle, end). The constant noise approach is a simpler approach that may follow user rules to inject noise for a determined duration and position across all training scenes. The guided noise approach allows for a user-specified loss function, which may be used within loss function back propagation720within the training loop740. User-specified performance metrics may be used by data occlusion used within noise injection720. Depending on a selected machine learning method or architecture, the noise injection720may also be integrated into the design of the loss function loss function back propagation720, such as may be used in Reinforcement Learning or Adversarial Machine Learning.

FIG. 8is a pictorial drawing illustrating multimodal perception sensor synthetic adverse weather800, according to an embodiment. Sensor noise may be used to generate synthetic adverse weather or atmospheric conditions, which may be used to improve the performance of a perception model under real-world adverse weather or atmospheric conditions. The synthetic adverse weather800may include a ground truth805associated with data gathered on a cloudless day. A synthetic fog may be applied to ground truth805, such as to generate fog with 30-meter visibility810, fog with 40-meter visibility815, or fog with 750-meter visibility820. Similarly, a synthetic rain may be applied to ground truth805, such as to generate 20-millimeter per hour (mm/hr) rain825, 100 mm/hr rain830, or 17 mm/hr rain835.

The synthetic weather or atmospheric conditions may include a multimodal cross-dataset analysis to simulate diverse types of synthetic adverse weather or atmospheric conditions such as rain, fog, snow, and night conditions. The multimodal cross-dataset analysis may be used to improve a selection of training data for use during training. In an example, one or more datasets that include real-world rain events may be selected, and one or more additional datasets without real-world rain events may be selected and augmented to generate corresponding rain datasets, such as rain events shown inFIG. 8. This multimodal cross-dataset analysis may include creating a joint dataset and then use a dataset sampler to generate balanced data subsets according to the weather conditions in each data frame.

FIG. 9is a pictorial drawing illustrating cross-dataset 3D object insertion900, according to an embodiment. In addition to image augmentation, cross-dataset 3D object insertion900may be used to extract object samples from cross-datasets and injecting them into frames of other datasets to improve accuracy. In an example, a captured image910may include a bicycle915. As shown in extracted feature point cloud920, the bicycle may be identified by a bounding box925. The captured images and point-clouds associated with bicycles in various datasets may be stored in a database and a lookup table, which may be used for augmentation in other datasets. As shown in point cloud930, a vehicle may be identified by a bounding rectangular area935. The bicycle from bounding box925may be used to augment the point cloud930by placing the bicycle on top of the vehicle bounding rectangular area935, and may be used to generate augmented point cloud940with augmented bicycle945. Orientation information may be saved for each object sample to make a more realistic insertion, such as by rotating bicycle915to be parallel with the longest dimension of vehicle bounding rectangular area935. Data from various bicycles may be used to improve training and model performance, such as using captured images or point-clouds associated with a first bicycle in a first set of training dataframes, then using captured images or point-clouds associated with other bicycles in subsequent training dataframes.

FIG. 10is a schematic drawing illustrating a method1000, according to an embodiment. Method1000includes receiving1010a multimodal perception dataset. The multimodal perception dataset may include an image dataset captured by an image capture device and a ranging dataset captured by a ranging sensor device. Method1000includes generating1020a perception model dataset based on the multimodal perception dataset and based on a target perception model. The perception model dataset may match a target perception model input type and a target perception model shape. Method1000includes generating1030a multimodal obfuscated machine learning dataset based on the perception model dataset. The multimodal obfuscated machine learning dataset including a sensor noise injection for both the image dataset and the ranging dataset. The sensor noise injection may include a sensor signal noise injection, a temporal noise injection, and a multimodal adverse weather sensor injection. The multimodal adverse weather sensor injection may include at least one of a night noise injection, a rain noise injection, a fog noise injection, and a snow noise injection. Method1000includes training a perception model based on the generated multimodal obfuscated machine learning dataset.

The generation1020of the perception model dataset may be further based on a sensor signal augmentation to augment the perception model dataset to match the target perception model input type and the target perception model shape. The sensor signal augmentation may include at least one of a crop augmentation, a resize augmentation, and a crop-resize augmentation. The sensor signal augmentation may include a data set transformer to map the multimodal perception dataset to a plurality of target model input tensors. The generation1020of the perception model dataset may be further based on a cross-dataset manipulation. The cross-dataset manipulation may include at least one of a multimodal subset, a multimodal superset, and a dataset sampler.

Method1000may include generating1050a perception ground-truth annotation dataset based on the target perception model input type and the target perception model shape. The perception ground-truth annotation dataset may be used to transform a plurality of ground-truth annotations to maintain compatibility with the perception model dataset. Method1000may include generating1060a noise-injected ground-truth annotation dataset based on the perception ground-truth annotation dataset and the sensor noise injection. The noise-injected ground-truth annotation dataset may identify a sensor noise annotation associated with the sensor noise injection. Method1000may include generating1070a balanced dataset based on the multimodal perception dataset. The balanced dataset may include a substantially balanced number of samples in each of the image dataset and the ranging dataset.

Example computer system1100includes at least one processor1102(e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both, processor cores, compute nodes, etc.), a main memory1104and a static memory1106, which communicate with each other via a link1108(e.g., bus). The computer system1100may further include a video display unit1110, an alphanumeric input device1112(e.g., a keyboard), and a user interface (UI) navigation device1114(e.g., a mouse). In one embodiment, the video display unit1110, input device1112and UI navigation device1114are incorporated into a touch screen display. The computer system1100may additionally include a storage device1116(e.g., a drive unit), a signal generation device1118(e.g., a speaker), a network interface device1120, and one or more sensors (not shown), such as a global positioning system (GPS) sensor, compass, accelerometer, gyroscope sensor, inertial sensor, magnetometer, or other sensor.

The storage device1116includes a machine-readable medium1122on which is stored one or more sets of data structures and instructions1124(e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The instructions1124may also reside, completely or at least partially, within the main memory1104, static memory1106, and/or within the processor1102during execution thereof by the computer system1100, with the main memory1104, static memory1106, and the processor1102also constituting machine-readable media.

The instructions1124may further be transmitted or received over a communications network1126using a transmission medium via the network interface device1120using well-known transfer protocols (e.g., HTTP). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, mobile telephone networks, plain old telephone (POTS) networks, and wireless data networks (e.g., Bluetooth, Wi-Fi, 3G, and 4G LTE/LTE-A, 5G, DSRC, or satellite communication networks). The term “transmission medium” shall be taken to include any intangible medium that is capable of storing, encoding, or carrying instructions for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

A processor subsystem may be used to execute the instruction on the machine-readable medium. The processor subsystem may include one or more processors, each with one or more cores. Additionally, the processor subsystem may be disposed on one or more physical devices. The processor subsystem may include one or more specialized processors, such as a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or a fixed function processor.

Example 1 is a system for autonomous vehicle perception development and training, the system comprising: processing circuitry; and a memory that includes, instructions, the instructions, when executed by the processing circuitry, cause the processor circuitry to: receive a multimodal perception dataset, the multimodal perception dataset including an image dataset captured by an image capture device and a ranging dataset captured by a ranging sensor device; generate a perception model dataset based on the multimodal perception dataset, the perception model dataset matching a plurality of target perception model parameters of a target perception model; generate a multimodal obfuscated machine learning dataset based on the perception model dataset, the multimodal obfuscated machine learning dataset including a sensor noise injection for both the image dataset and the ranging dataset; and train a perception model based on the multimodal obfuscated machine learning dataset.

In Example 2, the subject matter of Example 1 includes, wherein the sensor noise injection includes at least one of a sensor signal noise injection, a temporal noise injection, an adverse light condition injection, or a multimodal adverse weather sensor injection.

In Example 3, the subject matter of Example 2 includes, wherein the adverse light condition injection includes at least one of a night lighting injection, a glare injection, or a reflection injection.

In Example 4, the subject matter of Examples 2-3 includes, wherein the multimodal adverse weather sensor injection includes at least one of a night noise injection, a rain noise injection, a fog noise injection, or a snow noise injection.

In Example 5, the subject matter of Examples 1˜4 includes, wherein the generation of the perception model dataset is further based on a sensor signal augmentation to augment the perception model dataset to match the plurality of target perception model parameters.

In Example 6, the subject matter of Example 5 includes, wherein the sensor signal augmentation includes at least one of a crop augmentation, a resize augmentation, or a crop-resize augmentation.

In Example 7, the subject matter of Examples 5-6 includes, wherein the sensor signal augmentation includes a dataset transformation to map the multimodal perception dataset to a plurality of target model input tensors.

In Example 8, the subject matter of Examples 1-7 includes, wherein the generation of the perception model dataset is further based on a cross-dataset operation.

In Example 9, the subject matter of Example 8 includes, wherein the cross-dataset manipulation includes at least one of a multimodal subset, a multimodal superset, or a dataset sampler.

In Example 10, the subject matter of Examples 1-9 includes, the instructions further causing the processing circuitry to generate a perception ground-truth annotation dataset based on the plurality of target perception model parameters, the perception ground-truth annotation dataset to transform a plurality of ground-truth annotations to maintain compatibility with the perception model dataset.

In Example 11, the subject matter of Example 10 includes, the instructions further causing the processing circuitry to generate a noise-injected ground-truth annotation dataset based on the perception ground-truth annotation dataset and the sensor noise injection, the noise-injected ground-truth annotation dataset identifying a sensor noise annotation associated with the sensor noise injection.

In Example 12, the subject matter of Examples 1-11 includes, the instructions further causing the processing circuitry to generate a balanced dataset based on the multimodal perception dataset, the balanced dataset including a substantially balanced number of samples in each of a plurality of multimodal perception data subsets generated based on the multimodal perception dataset.

Example 13 is at least one non-transitory machine-readable storage medium, comprising a plurality of instructions that, responsive to being executed with processor circuitry of a computer-controlled device, cause the processor circuitry to: receive a multimodal perception dataset, the multimodal perception dataset including an image dataset captured by an image capture device and a ranging dataset captured by a ranging sensor device; generate a perception model dataset based on the multimodal perception dataset, the perception model dataset matching a plurality of target perception model parameters of a target perception model; generate a multimodal obfuscated machine learning dataset based on the perception model dataset, the multimodal obfuscated machine learning dataset including a sensor noise injection for both the image dataset and the ranging dataset; and train a perception model based on the multimodal obfuscated machine learning dataset.

In Example 14, the subject matter of Example 13 includes, wherein the sensor noise injection includes at least one of a sensor signal noise injection, a temporal noise injection, an adverse light condition injection, or a multimodal adverse weather sensor injection.

In Example 15, the subject matter of Example 14 includes, wherein the adverse light condition injection includes at least one of a night lighting injection, a glare injection, or a reflection injection.

In Example 16, the subject matter of Examples 14-15 includes, wherein the multimodal adverse weather sensor injection includes at least one of a night noise injection, a rain noise injection, a fog noise injection, or a snow noise injection.

In Example 17, the subject matter of Examples 13-16 includes, wherein the generation of the perception model dataset is further based on a sensor signal augmentation to augment the perception model dataset to match the plurality of target perception model parameters.

In Example 18, the subject matter of Example 17 includes, wherein the sensor signal augmentation includes at least one of a crop augmentation, a resize augmentation, or a crop-resize augmentation.

In Example 19, the subject matter of Examples 17-18 includes, wherein the sensor signal augmentation includes a dataset transformation to map the multimodal perception dataset to a plurality of target model input tensors.

In Example 20, the subject matter of Examples 13-19 includes, wherein the generation of the perception model dataset is further based on a cross-dataset operation.

In Example 21, the subject matter of Example 20 includes, wherein the cross-dataset manipulation includes at least one of a multimodal subset, a multimodal superset, or a dataset sampler.

In Example 22, the subject matter of Examples 13-21 includes, the instructions further causing the processing circuitry to generate a perception ground-truth annotation dataset based on the plurality of target perception model parameters, the perception ground-truth annotation dataset to transform a plurality of ground-truth annotations to maintain compatibility with the perception model dataset.

In Example 23, the subject matter of Example 22 includes, the instructions further causing the processing circuitry to generate a noise-injected ground-truth annotation dataset based on the perception ground-truth annotation dataset and the sensor noise injection, the noise-injected ground-truth annotation dataset identifying a sensor noise annotation associated with the sensor noise injection.

In Example 24, the subject matter of Examples 13-23 includes, the instructions further causing the processing circuitry to generate a balanced dataset based on the multimodal perception dataset, the balanced dataset including a substantially balanced number of samples in each of a plurality of multimodal perception data subsets generated based on the multimodal perception dataset.

Example 25 is a method for autonomous vehicle perception development and training, the method comprising: receiving a multimodal perception dataset, the multimodal perception dataset including an image dataset captured by an image capture device and a ranging dataset captured by a ranging sensor device; generating a perception model dataset based on the multimodal perception dataset, the perception model dataset matching a plurality of target perception model parameters of a target perception model; generating a multimodal obfuscated machine learning dataset based on the perception model dataset, the multimodal obfuscated machine learning dataset including a sensor noise injection for both the image dataset and the ranging dataset; training a perception model based on the multimodal obfuscated machine learning dataset.

In Example 26, the subject matter of Example 25 includes, wherein the sensor noise injection includes at least one of a sensor signal noise injection, a temporal noise injection, an adverse light condition injection, or a multimodal adverse weather sensor injection.

In Example 27, the subject matter of Example 26 includes, wherein the adverse light condition injection includes at least one of a night lighting injection, a glare injection, or a reflection injection.

In Example 28, the subject matter of Examples 26-27 includes, wherein the multimodal adverse weather sensor injection includes at least one of a night noise injection, a rain noise injection, a fog noise injection, or a snow noise injection.

In Example 29, the subject matter of Examples 25-28 includes, wherein the generation of the perception model dataset is further based on a sensor signal augmentation to augment the perception model dataset to match the plurality of target perception model parameters.

In Example 30, the subject matter of Example 29 includes, wherein the sensor signal augmentation includes at least one of a crop augmentation, a resize augmentation, or a crop-resize augmentation.

In Example 31, the subject matter of Examples 29-30 includes, wherein the sensor signal augmentation includes a dataset transformation to map the multimodal perception dataset to a plurality of target model input tensors.

In Example 32, the subject matter of Examples 25-31 includes, wherein the generation of the perception model dataset is further based on a cross-dataset operation.

In Example 33, the subject matter of Example 32 includes, wherein the cross-dataset manipulation includes at least one of a multimodal subset, a multimodal superset, or a dataset sampler.

In Example 34, the subject matter of Examples 25-33 includes, generating a perception ground-truth annotation dataset based on the plurality of target perception model parameters, the perception ground-truth annotation dataset to transform a plurality of ground-truth annotations to maintain compatibility with the perception model dataset.

In Example 35, the subject matter of Example 34 includes, generating a noise-injected ground-truth annotation dataset based on the perception ground-truth annotation dataset and the sensor noise injection, the noise-injected ground-truth annotation dataset identifying a sensor noise annotation associated with the sensor noise injection.

In Example 36, the subject matter of Examples 25-35 includes, generating a balanced dataset based on the multimodal perception dataset, the balanced dataset including a substantially balanced number of samples in each of a plurality of multimodal perception data subsets generated based on the multimodal perception dataset.

Example 37 is an apparatus for autonomous vehicle perception development and training, the apparatus comprising: means for receiving a multimodal perception dataset, the multimodal perception dataset including an image dataset captured by an image capture device and a ranging dataset captured by a ranging sensor device; means for generating a perception model dataset based on the multimodal perception dataset, the perception model dataset matching a plurality of target perception model parameters of a target perception model; means for generating a multimodal obfuscated machine learning dataset based on the perception model dataset, the multimodal obfuscated machine learning dataset including a sensor noise injection for both the image dataset and the ranging dataset; means for training a perception model based on the multimodal obfuscated machine learning dataset.

In Example 38, the subject matter of Example 37 includes, wherein the sensor noise injection includes at least one of a sensor signal noise injection, a temporal noise injection, an adverse light condition injection, or a multimodal adverse weather sensor injection.

In Example 39, the subject matter of Example 38 includes, wherein the adverse light condition injection includes at least one of a night lighting injection, a glare injection, or a reflection injection.

In Example 40, the subject matter of Examples 38-39 includes, wherein the multimodal adverse weather sensor injection includes at least one of a night noise injection, a rain noise injection, a fog noise injection, or a snow noise injection.

In Example 41, the subject matter of Examples 37-40 includes, wherein the generation of the perception model dataset is further based on a sensor signal augmentation to augment the perception model dataset to match the plurality of target perception model parameters.

In Example 42, the subject matter of Example 41 includes, wherein the sensor signal augmentation includes at least one of a crop augmentation, a resize augmentation, or a crop-resize augmentation.

In Example 43, the subject matter of Examples 41-42 includes, wherein the sensor signal augmentation includes a dataset transformation to map the multimodal perception dataset to a plurality of target model input tensors.

In Example 44, the subject matter of Examples 37-43 includes, wherein the generation of the perception model dataset is further based on a cross-dataset operation.

In Example 45, the subject matter of Example 44 includes, wherein the cross-dataset manipulation includes at least one of a multimodal subset, a multimodal superset, or a dataset sampler.

In Example 46, the subject matter of Examples 37-45 includes, means for generating a perception ground-truth annotation dataset based on the plurality of target perception model parameters, the perception ground-truth annotation dataset to transform a plurality of ground-truth annotations to maintain compatibility with the perception model dataset.

In Example 47, the subject matter of Example 46 includes, means for generating a noise-injected ground-truth annotation dataset based on the perception ground-truth annotation dataset and the sensor noise injection, the noise-injected ground-truth annotation dataset identifying a sensor noise annotation associated with the sensor noise injection.

In Example 48, the subject matter of Examples 37-47 includes, means for generating a balanced dataset based on the multimodal perception dataset, the balanced dataset including a substantially balanced number of samples in each of a plurality of multimodal perception data subsets generated based on the multimodal perception dataset.

Example 50 is an apparatus comprising means to implement of any of Examples 1-48.

Example 51 is a system to implement of any of Examples 1-48.

Example 52 is a method to implement of any of Examples 1-48.

Circuitry or circuits, as used in this document, may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry such as computer processors comprising one or more individual instruction processing cores, state machine circuitry, and/or firmware that stores instructions executed by programmable circuitry. The circuits, circuitry, or modules may, collectively or individually, be embodied as circuitry that forms part of a larger system, for example, an integrated circuit (IC), system on-chip (SoC), desktop computers, laptop computers, tablet computers, servers, smart phones, etc.

As used in any embodiment herein, the term “logic” may refer to firmware and/or circuitry configured to perform any of the aforementioned operations. Firmware may be embodied as code, instructions or instruction sets and/or data that are hard-coded (e.g., nonvolatile) in memory devices and/or circuitry.

“Circuitry,” as used in any embodiment herein, may comprise, for example, singly or in any combination, hardwired circuitry, programmable circuitry, state machine circuitry, logic and/or firmware that stores instructions executed by programmable circuitry. The circuitry may be embodied as an integrated circuit, such as an integrated circuit chip. In some embodiments, the circuitry may be formed, at least in part, by the processor circuitry executing code and/or instructions sets (e.g., software, firmware, etc.) corresponding to the functionality described herein, thus transforming a general-purpose processor into a specific-purpose processing environment to perform one or more of the operations described herein. In some embodiments, the processor circuitry may be embodied as a stand-alone integrated circuit or may be incorporated as one of several components on an integrated circuit. In some embodiments, the various components and circuitry of the node or other systems may be combined in a system-on-a-chip (SoC) architecture