Methods and apparatus for unsupervised multimodal anomaly detection for autonomous vehicles

An example includes obtaining first sensor data from a first sensor and second sensor data from a second sensor, the first sensor of a first sensor type different than a second sensor type of the second sensor; generating first encoded sensor data based on the first sensor data and second encoded sensor data based on the second sensor data; generating a contextual fused sensor data representation of the first and second sensor data based on the first and second encoded sensor data; generating first and second reconstructed sensor data based on the contextual fused sensor data representation; determining a deviation estimation based on the first and second reconstructed sensor data, the deviation estimation representative of a deviation between: (a) the first reconstructed sensor data, and (b) the first sensor data; and detecting an anomaly in the deviation estimation, the anomaly indicative of an error associated with the first sensor.

FIELD OF THE DISCLOSURE

This disclosure relates generally to computer processing in vehicles, and more specifically to methods and apparatus for unsupervised multimodal anomaly detection for autonomous vehicles.

BACKGROUND

Autonomous vehicles, or self-driving vehicles, use multiple sensors to acquire information about the surrounding areas in which those vehicles navigate. Computers analyze the sensor data from such multiple sensors to make decisions regarding speed, braking, maneuvers, traction handling, external lighting requirements, etc. Autonomous vehicles are often provided with different types of sensors to detect objects under different types of environmental conditions.

Wherever possible, the same reference numbers are used throughout the drawing(s) and accompanying written description to refer to the same or like parts. Connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and/or physical or logical couplings between the various elements.

DETAILED DESCRIPTION

Autonomous robotic systems such as autonomous vehicles use multiple cameras as well as range sensors to perceive characteristics of their environments. The different sensor types (e.g., infrared (IR) sensors, red-green-blue (RGB) color cameras, Light Detection and Ranging (LIDAR) sensors, Radio Detection and Ranging (RADAR) sensors, SOund Navigation And Ranging (SONAR) sensors, etc.) can be used together in heterogeneous sensor configurations useful for performing various tasks of autonomous vehicles. For example,FIG. 1shows an example autonomous vehicle100and example usages of a heterogeneous sensor configuration of the autonomous vehicle100. The performances of these sensors are affected differently under diverse environmental conditions. Such inconsistent performance across different environment conditions makes it difficult (in some cases impossible) to detect anomalous outputs of the sensors using prior techniques. In some instances, anomalies could be the result of either hardware or software failures or adversarial attacks targeted at the autonomous systems.

In autonomous robotic applications such as autonomous vehicle operation, redundant sensors can be employed to safely drive under a variety of environmental conditions. An anomaly (e.g., due to a hardware failure, a software failure, or an adversarial attack) can jeopardize the safe operation of the autonomous system. Examples disclosed herein enable unsupervised multimodal anomaly detection for autonomous vehicles and/or any other type of autonomous system. For example, examples disclosed herein enable an autonomous system to qualitatively and/or quantitatively detect deviations of sensor data from an expected norm and/or detect other complex and/or uncommon behaviors of multiple types of sensors through probabilistic modeling and feature fusion in latent space using unsupervised deep neural auto-encoding systems. Unsupervised multimodal anomaly detection examples disclosed herein integrate sensor data (e.g., in a seamless manner) collected from sensor signals of a heterogeneous sensor configuration to evaluate the consistency, stability, and/or quality of the information provided by each sensor. Examples disclosed herein leverage sensor redundancy of autonomous vehicles to detect anomalous behavior represented in sensor data and determine a level of confidence for each sensor at different points in time. In some examples, such anomaly detection and confidence level assessments for multiple sensors can be performed simultaneously. In this manner, a processing system for an autonomous vehicle can make quick decisions (e.g., split-second decisions) during operation of the autonomous vehicle by considering whether sensor data is providing anomalous information or non-anomalous information.

As used herein, an anomaly occurs when a first environmental characteristic represented in first sensor data from a first sensor of an autonomous vehicle deviates beyond a threshold from a reconstructed version of the first sensor data after performing encoding, aggregating, fusing, and decoding processes on the first sensor data in accordance with teachings of this disclosure. When an anomaly occurs, the contributing sensor data is regarded as providing anomalous information because the sensor data is abnormal when considered relative to a whole-scene sensor-based representation of an environment. The anomalous information leading to the detection of an anomaly is abnormal information in that it is not within a normal range or is not commensurate with sensor data of one or more other sensors of the autonomous vehicle. That is, examples disclosed herein use the one or more other sensors as redundant to the first sensor for purposes of anomaly detection. When one or more redundant sensors more consistently and/or accurately characterize the same environmental aspect that is mis-characterized by the first sensor, the first sensor is regarded as an anomalous sensor. For example, imaging sensor data from a rear-view camera of an autonomous vehicle may report an obstruction, thus preventing the autonomous vehicle from backing up. However, unsupervised multimodal anomaly detection examples disclosed herein can determine that the rear-view camera is anomalous when the rear-view camera exhibits abnormal characteristics relative to a whole-scene representation generated based on other sensors such as a rear-facing RADAR sensor. This may be the case when the obstruction detected by the rear-view camera is operating or sensing abnormally due to, for example, a leaf obstructing the camera field of view, dirt on the camera lens, a crack in the camera lens, a malfunctioning imaging sensor of the camera, etc.

In the heterogeneous sensor configuration ofFIG. 1, the autonomous vehicle100is provided with camera sensors, RADAR sensors, and LIDAR sensors. Cameras are high resolution image sensors that can detect both shape and texture (e.g., road markings, traffic sign text, traffic light color, road conditions, etc.). Visible light cameras work well in illuminated conditions but sometimes do not work as well under less-illuminated conditions (e.g., at night, in dark parking garages, etc.) or under other challenging weather conditions (e.g., low-visibility fog, rain, snowfall, dust storms, etc.). On the other hand, LIDAR sensors work well in almost all light conditions but their performance decreases in the presence of dust, mist, snow, fog and rain. However, although LIDAR sensor technology adapts better to all lighting conditions than visible light camera technology, LIDAR sensors have lower resolution than cameras. RADAR sensors are able to sense through fog and can measure both distance and velocity of moving objects (using Doppler frequency shift). An advantage of RADAR over LIDAR is that electromagnetic waves produced by RADAR have lower absorption by objects than light produced by LIDAR sensors. As such, RADAR sensors can work over longer distances than LIDAR sensors. Unfortunately, RADAR technology offers significantly lower resolution than other sensors due to its power and bandwidth limitations. SONAR sensors use ultrasound for near-field sensing which complements RADAR and LIDAR technologies. That is, while RADAR and LIDAR technologies can be used for long-distance object detection, SONAR technology can be used for short-distance object detection. In addition, SONAR sound waves can travel through transparent objects such as glass.

FIG. 2illustrates an example heterogeneous sensor placement configuration200in the autonomous vehicle100ofFIG. 1. In the illustrated example ofFIG. 2, there are 12 cameras202in a 360-degree configuration on the autonomous vehicle100. Eight of those cameras202support near-field sensing. In some examples, the cameras202include multiple types of cameras such as visible light cameras, thermal cameras, and/or infrared cameras. Visible light cameras perceive fields of view similar to human vision. Thermal cameras perceive fields of view based on temperature such that a human being, an animal, a warm operating vehicle, etc. produce heat signatures relative to other surrounding objects. Infrared cameras can perceive fields of view in dark or low-light environments in which visible light cameras are less effective.

The autonomous vehicle100also includes six total sector LIDAR sensors204in which a sector is a field of detection. Three of the LIDAR sensors204are located at the front of the autonomous vehicle100, and three are located at the rear. The autonomous vehicle100is also provided with six RADAR sensors206that include a mix of short-range and long-range detection capabilities, providing 360-degree coverage around the autonomous vehicle100. In some examples, the autonomous vehicle100is also provided with SONAR sensors. In the illustrated example ofFIG. 2, the LIDAR sensors204, the RADAR sensors206, and SONAR sensors are used as redundant sensors for the camera sensors202. An advantage of the cameras202is that they have high image resolution. Thus, the cameras202offer a view of surrounding environments in a similar way as humans perceive such surroundings. The LIDAR sensors204, the RADAR sensors206, and SONAR sensors provide low-resolution point clouds (e.g., mosaics or aggregations of data point values the represent a depth-based rendering of the perceived environment) that are fused together to create a 3D representation of the environment. As such, the heterogeneous sensor placement configuration200ofFIG. 2provides the autonomous vehicle100with two sets of redundant sensors that include primary sensors in the form of the cameras202and the secondary sensors in the form of the LIDAR sensors204, the RADAR sensors206, and the SONAR sensors.

Examples disclosed herein may employ redundant heterogeneous sensor configurations such as the heterogeneous sensor placement configuration200ofFIG. 2to implement high-dimensional feature fusion that works independent of the types of sensors and/or independent of specific data formats by using unsupervised machine learning. In the context of dimensionality of data that represents features in input data, dimension, dimensional, and dimensionality refer to a size in elements for representing features in the input data. That is, the number of dimensions for a feature refers to the size of the feature container such as the size of a vector or a matrix that includes values representing the feature. For example, techniques disclosed herein process input data (e.g., sensor data) from multiple sensors the same way and re-arrange the input data in the same manner on the basis that each element of a feature is considered as a separate dimension of a plurality of dimensions of that feature. Examples disclosed herein use such multi-dimensional characteristics of feature data to perform feature aggregation across multiple sensors on a dimension-per-dimension basis. For example, if a feature is described using values in an array of size [1×128], that feature is described with a feature vector of 128 dimensions, and examples disclosed herein aggregate each dimension separately. This enables fusing sensor data from multiple sensors and/or multiple types of sensors at the dimension level so that anomaly analyses and detection can be performed across an environment monitored using multiple different types of sensors. As such, unsupervised multimodal anomaly detection techniques for autonomous vehicles disclosed herein are not limited to use with particular types of sensors or particular types of data structure for representing environmental characteristics. This is advantageous over prior techniques designed for analyzing sensor data from homogeneous sensor type configurations (configurations of multiple sensors of the same type), specific quantities of sensors and actuators, specific signal structures, and specific routines that are specifically developed for a particular task and environment. Examples disclosed herein may be flexibly implemented across heterogeneous sensor configurations in which the quantity of sensors can vary over time. For example, in the autonomous vehicle100ofFIGS. 1 and 2, one or more sensors may drop out or malfunction over time. In such situations, examples disclosed herein are sufficiently unaffected by such changes in sensor operation by virtue of recognizing such changes as anomalous and taking such anomalous activity into account to not produce unexpected or abnormal operation of the autonomous vehicle100that is inconsistent with the actual environmental conditions in which the autonomous vehicle100is operating.

By recognizing anomalous situations, examples disclosed herein are not hindered by disadvantages of prior solutions which include restriction to only predefined sensors in amount and type, lack of broad deployment potential due to the explicit or implicit supervised approach while training models or while fitting model parameters, lack of context-invariant confidence value estimates per sensor and per scenario, dependency to sensing specific environment using fixed positions and fixed orientations of sensors, lack of generalized and optimized leveraging of mutual sensors' fields of view that overlap to establish robust signal-context fusion representations for determining confidence metrics based on multiple device outputs, and lack of computational parallelization, and compactness limitations. The property of compactness refers to the reduced size of encoding or representation of a scene performed by a neuro-encoder (e.g., an auto-encoder). With improved compactness, an entire scene and large redundant masses of information are spatially efficiently encoded. Examples disclosed herein exhibit better compactness than prior techniques of analyzing sensor data because they enable more meaningful computation, communication, and storage of high-density features.

Examples disclosed herein create and use a fused content representation of multiple heterogeneous signals from multiple redundant heterogeneous sensors. Examples disclosed herein leverage the implicit unified spatial context and visual context in the fused content representation to enable autonomous vehicles to determine when, how, and which sensor(s) is/are behaving abnormally.

An example advantage of examples disclosed herein over prior techniques is that examples disclosed herein leverage unsupervised learning by using sparse convolutional auto-encoders that preserve various structural properties of input sensor data. These sparse convolutional auto-encoders have compact, injective, and quasi-subjective properties. They can detect inconsistencies in the sensor state to leverage structural metric properties of input sensor data independent of the scene or physical space that the input sensor data represents. Examples disclosed herein enable determining the sensor, the timestamp, and a normalized numerical score of deviation when an anomaly is detected (e.g., a malfunction of a sensor, an attack on the system, etc.). The confidence score enriches the data to be analyzed by probabilistic inference algorithms in multiple tasks such as motion planning which is a part of an autonomous driving stack (e.g., the autonomous driving apparatus300ofFIG. 3) and other decision-making algorithms, thus, enabling the blending of probabilistically optimized information and analysis thereof. In addition, unsupervised learning makes examples disclosed herein substantially invariant to specific structures or visual conditions of the environment. As such examples disclosed herein are suitable for deployment across a large variety of road systems in multiple countries or even across a variety of natural places (e.g., off-road environments).

FIG. 3is an example autonomous driving apparatus300of the autonomous vehicle100ofFIGS. 1 and 2. The example autonomous driving apparatus300represents operational components that execute as part of operating the autonomous vehicle100to collect sensor data, process the sensor data, and analyze the sensor data as disclosed herein to detect anomalies in the sensor data and control operations of the autonomous vehicle100. In some examples, the autonomous driving apparatus300may be implemented as an autonomous driving stack.

In the illustrated example, the autonomous driving apparatus300collects sensor data corresponding to characteristics of an environment302via the camera sensors202, the LIDAR sensors204, and/or the RADAR sensors206ofFIG. 2. In the illustrated example, the autonomous driving apparatus300may additionally or alternatively collect sensor data from one or more other sensors304(e.g., a global positioning system (GPS) sensor, an odometer, etc.) of the autonomous vehicle100.

The example autonomous driving apparatus300ofFIG. 3is provided with an anomaly detection apparatus306to create and analyze a fused content representation of the environment302based on multiple heterogeneous signals from multiple redundant heterogeneous sensors. Based on the fused content representation, the example anomaly detection apparatus306performs unsupervised multimodal anomaly detection to facilitate operation of the autonomous vehicle100. For example, analyses performed by the anomaly detection apparatus306can be used to determine when, how, and which sensor(s) is/are behaving abnormally so that sensor data from such sensor(s) can be ignored and/or supplemented by data from one or more other sensor(s) to control operation of the autonomous vehicle100. The example anomaly detection apparatus306is described in greater detail below in connection withFIG. 4.

The example autonomous driving apparatus300is provided with an example scene contextualizer308to implement scene understanding. Through scene understanding, the scene contextualizer308generates location-based context awareness for the autonomous vehicle100based on sensor data from one or more of the sensors202,204,206,304and map information from an example maps data interface310. For example, the scene contextualizer308determines geospatial information about the autonomous vehicle100such as geographic location, positioning, orientation, etc. For example, the scene contextualizer308receives map information from the example maps data interface310and GPS location coordinates from a GPS sensor interface304to identify a street/road position of the autonomous vehicle100. The maps data interface310may obtain map information from a server via a network and/or from a local data store so that the scene contextualizer308can associate GPS location coordinates to street-level map locations in the map information. The scene contextualizer308can also obtain sensor data from others of the sensors202,204,206,304to assess characteristics of the environment302to better understand exact positioning (e.g., street corner location, left/right-side street positioning, street-shoulder positioning, etc.) at a geographic location. The scene contextualizer308can also determine orientation such as facing direction of the autonomous vehicle100based on sensor data from others of the sensors202,204,206,304.

The example autonomous driving apparatus300is provided with an example scene analyzer312that performs sensor fusion and object tracking. For example, the scene analyzer312performs scene analyses on combinations of sensor data from one or more of the sensors202,204,206,304relative to scene context information from the scene contextualizer308. For example, the scene analyzer312can identify surface conditions (e.g., wet, gravel, dirt, etc.), weather conditions (e.g., rain, snow, hail, sleet, etc.), lighting conditions (e.g., night, day, lighted/dark tunnel, lighted/dark parking garage, etc.) of the environment302relative to geographic location, positioning, and orientation from the scene contextualizer308. The example scene analyzer312can track objects in the environment302such as other vehicles, pedestrians, animals, and/or other objects that affect travel of the autonomous vehicle100.

In the illustrated example, the scene contextualizer308and the scene analyzer312are in communication with the anomaly detection apparatus306to receive anomaly information from the anomaly detection apparatus306. In this manner, the scene contextualizer308and the scene analyzer312can perform their operations by taking into account anomalous information present in the sensor data collected from the sensors202,204,206,304. For example, the scene contextualizer308may use the anomaly information from the anomaly detection apparatus306and the scene analyzer312to ignore anomalous information and/or replace the anomalous information with reliable information from sensor data of one or more non-anomalous sensors.

The example autonomous driving apparatus300is provided with an example drivability map generator314to identify acceptable or safe paths of travel based on information from the scene contextualizer308and an example object tracker module in the scene analyzer312. The example object tracker module estimates kinematics and trajectories for dynamic objects in the environment302. Based on locations of static objects and the trajectories of the dynamic objects, the example drivability map generator314can estimate available free space. The identified free space surrounding the autonomous vehicle100is the path that can be travelled by the autonomous vehicle. If the scene contextualizer308and/or the scene analyzer312identify a pedestrian or another vehicle, for example, in any direction, the drivability map generator314updates navigation path information to indicate that travel is not permitted in the direction of the pedestrian or other vehicle.

The example autonomous driving apparatus300is provided with an example control scheduler316to generate motion planning control information and/or vehicle dynamics control information. For example, based on information from the drivability map generator314, the control scheduler316can generate maneuver control information for the autonomous vehicle100and schedule such maneuvering. Example motion planning performed by the control scheduler316includes driving direction such as forward driving, left turning, right turning, backing up, northward driving, southward driving, eastward driving, westward driving, etc. Example vehicle dynamics information can include breaking, traction control (e.g., for different surface conditions, weather conditions, etc.), suspension settings (e.g., based on surface conditions, traffic conditions, city driving vs. highway driving, etc.), and/or any other types of vehicle dynamics.

The example autonomous driving apparatus300is provided with an example controller318to implement the motion planning control information and/or vehicle dynamics control information from the control scheduler316for operation of the autonomous vehicle100. The example controller318is in communication with actuators, motors, switches, lights, and/or any other electrical, mechanical, and/or electromechanical components of the autonomous vehicle100to implement motion and vehicle dynamics.

FIG. 4is a block diagram of the example anomaly detection apparatus306ofFIG. 3that may be used to implement unsupervised multimodal anomaly detection for the autonomous vehicle100. The example anomaly detection apparatus306includes an example sensor data interface402, an example multimodal encoder404, an example aggregator405, an example dimensionally interleaving encoder406, an example extractive decoder408, an example extractive deviation distribution analyzer410, and an example anomaly detector412.

The anomaly detection apparatus306is provided with the example sensor data interface402to collect raw sensor data (Ii(x,y,t)) from one or more of the sensors202,204,206,304ofFIGS. 2 and 3. For example, the configuration of the sensors202,204,206,304is a heterogeneous configuration in which the sensors202,204,206,304are of different types. In such heterogeneous configuration, the sensor data interface402can, for example, obtain first collected sensor data from a first one of the sensors202,204,206,304and second collected sensor data from a second one of the sensors202,204,206,304, in which the first one of the sensors202,204,206,304is of a first sensor type different than a second sensor type of the second one of the sensors202,204,206,304. In some examples, the first collected sensor data and the second collected sensor data may be from different ones of camera data, LIDAR data, RADAR data, and/or SONAR data that represent 2D (x,y) images and/or 3D (x,y,z) images.

The anomaly detection apparatus306is provided with the example multimodal encoder404to convert collected raw sensor data to encoded sensor data (e.g., the encoded-compact sparse sensor data representation (Fi(t))606ofFIG. 6) represented as, for example, an array or vector for a particular time (t) at which the raw sensor data was collected. For example, the multimodal encoder404can generate first encoded sensor data based on first collected sensor data collected by the sensor data interface402in a multi-dimensional format and second encoded sensor data based on second collected sensor data collected by the sensor data interface402in a multi-dimensional format. Examples of the multimodal encoder404converting multi-dimensional sensor data to encoded sensor data are described below in connection withFIG. 7. In the illustrated example, the multimodal encoder404is referred to as multimodal because it encodes sensor data of different modalities. In this context, modalities refers to different types of sensors (e.g., camera sensors, LIDAR sensors, RADAR sensors, SONAR sensors, etc.).

The anomaly detection apparatus306is provided with the example aggregator405to generate a contextual aggregated sensor data representation (e.g., the contextual aggregated sensor data representation710) of collected raw sensor data based on the encoded sensor data generated by the multimodal encoder404. For example, the aggregator405stacks the encoded sensor data generated by the multimodal encoder404to generate the contextual aggregated sensor data representation. The anomaly detection apparatus306is provided with the example dimensionally interleaving encoder406to generate a contextual fused sensor data representation of collected raw sensor data. For example, the dimensionally interleaving encoder406can generate a contextual fused sensor data representation of the first and second collected sensor data collected by the sensor data interface402based on the contextual aggregated sensor data representation generated by the aggregator405. Examples of the dimensionally interleaving encoder406generating a contextual fused sensor data representation of sensor data are described below in connection withFIGS. 7 and 9.

The anomaly detection apparatus306is provided with the example extractive decoder408to generate reconstructed sensor data based on the contextual fused sensor data representation generated by the dimensionally interleaving encoder406. For example, the extractive decoder408can generate first reconstructed sensor data corresponding to the first collected sensor data from a first one of the sensors202,204,206,304, and can generate second reconstructed sensor data corresponding to the second collected sensor data from a second one of the sensors202,204,206,304, in which the first one of the sensors202,204,206,304. Generating of the reconstructed sensor data is described further below in connection withFIG. 7.

The anomaly detection apparatus306is provided with the example extractive deviation distribution analyzer410to determine probabilistic deviation estimations representative of likelihoods of deviations between expected sensor data and actual collected sensor data. For example, the extractive deviation distribution analyzer410can determine probabilistic deviation estimations based on the first and second reconstructed sensor data decoded by the extractive decoder408and corresponding ones of the first and second collected sensor data. In such examples, the probabilistic deviation estimation is representative of likelihoods of deviations between: (a) the first reconstructed sensor data, and (b) the first collected sensor data collected by the sensor data interface402from the first one of the sensors202,204,206,304. Examples of the extractive deviation distribution analyzer410determining a probabilistic deviation estimation are described below in connection withFIG. 7.

The anomaly detection apparatus306is provided with the example anomaly detector412to detect anomalies represented in collected sensor data. For example, the anomaly detector412can detect an anomaly in the probabilistic deviation estimation generated by the extractive deviation distribution analyzer410. In such examples, the anomaly is indicative of an error in the operation of the first one of the sensors202,204,206,304from which the sensor data interface402collected sensor data.

FIG. 5is an example signal structure representation500for sensor data collection from multiple heterogeneous sensors of the autonomous vehicle100. The left side of theFIG. 5shows multiple locations of range sensors, sometimes referred to as Spherical Mapping Sensor (SMS). Example range sensors include solid state LIDAR sensors such as the LIDAR sensors204(FIGS. 2 and 3) and solid state RADAR sensors such as the RADAR sensors206(FIGS. 2 and 3). In the example ofFIG. 5, the range sensors are routed by lines having a dotted line pattern, one of which is indicated by reference number502. In the illustrated example ofFIG. 5, the range sensors (i) produce the range distance function (Di(α,β,t)) stored as a 2D array structure indexed by Azimuth (α) and Elevation angles (β) in association with a data collection time indicated by a timestamp (t).

The right side ofFIG. 5shows multiple locations of image sensors (e.g., visible light cameras and thermal cameras). In the illustrated example, the image sensors (i) produce images (Ii(x,y,t)) organized as a rectangular raster grid indexed by horizontal and vertical dimensions (x, y) and collected at a data collection time indicated by a timestamp (t). In the example ofFIG. 5, the image sensors are routed by lines having a line pattern of dots and dashes, one of which is indicated by reference number504. In some examples, some image sensors of the autonomous vehicle100produce a single type of output while others provide multiple types of outputs. For example, one type of camera of the autonomous vehicle100may produce a single channel (near infrared) of sensor data, and another type of camera of the autonomous vehicle100may produce multiple channels of sensor data such as red, green, and blue (RGB) information for visible light cameras.

FIG. 6is an example auto-encoder (AE)600for use in performing anomaly detection. The auto-encoder600is a neural network with a symmetric structure of networked nodes that generate output data corresponding to their input data in accordance with their training. For example, a middle layer output of the neural network represents an encoding of input data. In the illustrated example, the auto-encoder600is a single-sensor encoder that processes sensor data collected by the sensor data interface402(FIG. 4) from one of the multiple sensors202,204,206,304. In the illustrated example ofFIG. 6, the input sensor data is shown as an input image603. The example auto-encoder600is trained to reconstruct its input onto an output layer, while verifying certain restrictions that prevent the auto-encoder600from merely copying the input data along the neural network and outputting the same input data. To process input sensor data from multiple ones of the sensors202,204,206,304, multiple ones of the auto-encoder600can be employed as described below in connection withFIG. 7.

A convolutional auto-encoder is a neural network with a convolution at any layer but is typically applied at the input. An auto-encoder is undercomplete if the encoding layer has a lower dimensionality than the input. In the illustrated example ofFIG. 6, reducing the number of nodes or neurons in the subsequent layers of the auto-encoder600imposes size and redundancy rejection and reshaping restrictions. As such, the neural network of the auto-encoder600compresses the information from the input (in higher dimensional space) into a lower dimensional space. Such compressing maps features of the input data from an over-redundant manifold to a less-redundant manifold with content-preserving properties. This results in the compressed output information retaining features of interest that are represented in the input data while simultaneously reducing the size of the data to be analyzed which, in turn, decreases the processing resources and processing time to analyze the collected sensor data for anomalies.

In the illustrated example ofFIG. 6, the first half of the auto-encoder600is an encoder sub-net602shown as the left portion of the auto-encoder600, and the second half of the auto-encoder600is a decoder sub-net604shown as the right portion of the auto-encoder600. The output of the encoder sub-net602portion of the auto-encoder600is a feature vector shown as an encoded-compact sparse sensor data representation (Fi(t))606(e.g., encoded sensor data) which depicts the same input image in less space and, thus, with higher information density. In the illustrated example ofFIG. 6, the encoded-compact sparse sensor data representation (Fi(t))606is decoded by the decoder sub-net604to produce or generate an example encoded-decoded image608(e.g., reconstructed sensor data). Moreover, the feature vector represented by the encoded-compact sparse sensor data representation (Fi(t))606preserves similarity metrics and is highly sparse (low-firing values on neuron outputs). These properties are explicitly obtained during training of the auto-encoder600using an engineered cost function. An example of such an engineered cost function includes, in addition to input-output deviations, cost factors to enforce convergence. Convergence minimizes the number of triggered neurons and derivative properties to accommodate similarity metrics criteria.

Once a full (unsupervised) training of the auto-encoder600has been conducted, the auto-encoder600can be used to process input sensor data collected from different sensor signals (e.g., images, scans, etc. depending on the sensor type) and generate compressed output data to determine a pixel-to-pixel (for camera sensors202) and/or a point-to-point (for LIDAR sensors204, RADAR sensors206, SONAR sensors) absolute discretized and quantized deviation D(Ii(x,y,t), Ii(x,y,t)) between an original input image (Ii(x,y,t))603and a reconstructed image (Ii(x,y,t)′)608output by the decoder sub-net604. The absolute discretized and quantized deviation D(Ii(x,y,t), Ii(x,y,t)) is a measure of deviation between corresponding pixels or points of the original input image (Ii(x,y,t))603and the reconstructed image (Ii(x,y,t)′)608expressed as a statistical deviation model N(μi, σi2) per sensor (i). In the statistical deviation model N(μi, σi2), N refers to a normal distribution, the variable μ represents the mean error, and the variable a represents the standard deviation of the mean error. By keeping track of the pixel-to-pixel deviations and/or point-to-point deviations, the absolute discretized and quantized deviation can be represented as an example validation deviation histogram610as shown inFIG. 6. The example validation deviation histogram610is a probability distribution function (PDF) of frequencies of pixel-to-pixel deviation amounts and/or point-to-point deviation amounts as an explicit deviation pattern. Based on this discretized format (the histogram610) of the PDF, the PDF can be fit to any suitable parametric model (e.g., a Gaussian PDF, a Log-Gaussian PDF, a Laplace distribution for images, and/or any other efficiently parametric tractable PDF) to generally, compactly, consistently, and efficiently assert the associated density of a deviation in continuous space. This phase of the anomaly analysis process produces a per-pixel or per-point continuous PDF model representative of the probabilistic deviation pattern of the whole process executed by the auto-encoder600. This is shown by an example fitted parametric distribution612ofFIG. 6.

In the example ofFIG. 6, when the auto-encoder600is trained to meet a desired performance, the statistical deviation model N(μi, σi2) is computed by fitting a histogram error (e.g., fitting the parametric distribution612to the validation deviation histogram610as shown inFIG. 6). The encoder sub-net602of the example auto-encoder600ofFIG. 6is an encoding part for a single sensor encoder (e.g., a single channel encoder in the). The example encoder sub-net602and the per-point PDF that it generates are the systematic structural components to apply a core modeling process to each of the sensors202,204,206,304regardless of the sensor type (e.g., a camera type, a LIDAR type, a RADAR type, a SONAR type, etc.).

FIG. 7is an example feature fusion and deviation estimation data flow700for use in anomaly detection based on sensor data collected from the autonomous vehicle100ofFIGS. 1-5. The example feature fusion and deviation estimation data flow700ofFIG. 7is implemented using multiple hierarchical auto-encoders that form the example multimodal encoder404, the example dimensionally interleaving encoder406, and the example extractive decoder408ofFIG. 4. In the illustrated example ofFIG. 7, the multimodal encoder404is formed of multiple example encoder sub-nets702a-d, and the dimensionally interleaving encoder406is formed of an example global encoder sub-net703. The example encoder sub-nets702a-dand the example global encoder sub-net703are substantially similar or identical to the encoder sub-net602of the auto-encoder600ofFIG. 6. Also in the illustrated example ofFIG. 7, the extractive decoder508is implemented by multiple example decoder sub-nets705a-dwhich are substantially similar or identical to the decoder sub-net604of the auto-encoder600. In the example ofFIG. 7, encoder sub-nets702a-d, the global encoder sub-net703, and the decoder sub-nets705a-dare based on convolutional, under-complete, sparse and structure-preserving auto-encoders. The example feature fusion and deviation estimation data flow700ofFIG. 7can be employed after training the hierarchical auto-encoders702a-d,703,705a-dbased on input sensor data and desired output results.

In the example ofFIG. 7, sensor data (Ii(x,y,t)) from ones of the sensors202,204,206,304is provided to the encoder sub-nets702a-dof the multimodal encoder404. For example, the sensor data (Ii(x,y,t)) may be obtained by the sensor data interface402ofFIG. 4. In the example ofFIG. 7, three input sensor data (e.g., shown as i=1 to i=n−1) are from visible light cameras, and fourth input sensor data (e.g., shown as i=n) is from a LIDAR sensor. However, this is merely illustrative, and the example ofFIG. 7is more broadly applicable to any number of sensors and/or any suitable types of sensors arranged in a heterogeneous sensor configuration as described above in connection withFIGS. 1 and 2. In any case, the input sensor data (Ii(x,y,t)) is collected at a same time (t).

At a first phase (1)704ofFIG. 7, the example encoder sub-nets702a-dof the multimodal encoder404process the input collected sensor data to generate corresponding latent lower dimensional representations. Because of the diverse resolutions and types of sensors202,204,206,304, at the first phase (1)704, the example encoder sub-nets702a-dgenerate encoded sensor data representations of different sizes (e.g., due to the multiple dimensionality of the input data). The encoded sensor data outputs are shown at a second phase (2)706as diverse-length sensor data representations that are normalized to a length of m compressed data points on a dimension-by-dimension basis. The encoded sensor data outputs at the second phase (2)706are encoded-compact sparse sensor data representations (Fi(t)) such as the encoded-compact sparse sensor data representation (Fi(t))606ofFIG. 6. Generating such encoded-compact sparse sensor data representations (Fi(t)) for each set of synchronized signals from the sensors202,204,206,304is based on an inference execution of the n-encoder sub-nets702a-d. This has low computational cost and can be parallelized using any suitable form of neural accelerator including graphics processing units (GPUs), field programmable gate arrays (FPGAs), and hybrid processors such XPU (e.g., combined CPU/GPU).

At a third phase (3)708ofFIG. 7, the example aggregator405(FIG. 4) receives a quantity of n encoded-compact sparse sensor data representations (Fi(t)) (e.g., encoded sensor data) from the encoder sub-nets702a-dand generates a contextual aggregated sensor data representation710of the collected sensor data (Ii(x,y,t)) based on the encoded-compact sparse sensor data representations (Fi(t)). In the illustrated example, the contextual aggregated sensor data representation710is of the same length m compressed data points as the encoded-compact sparse sensor data representations (Fi(t)). The contextual aggregated sensor data representation710at the third phase (3)708is an image containing neuro-compressed sparse partial representations of dimensionally rearranged components of each sensor data representation. For example, the aggregator405aggregates or stacks the encoded-compact sparse sensor data representations (Fi(t)) together on a per-dimension basis to generate the contextual aggregated sensor data representation710. The dimension-by-dimension structural arrangement of the contextual aggregated sensor data representation710is well-suited for scalability because additional channels of input sensor data can be added through the same dimension-by-dimension aggregation used to form the contextual aggregated sensor data representation710. An example technique for generating the contextual aggregated sensor data representation710is described below in connection withFIG. 9.

In the illustrated example, the contextual aggregated sensor data representation710describes the entire 360-degree scene at which the autonomous vehicle100is located. Despite its compressed encoding, there are still redundancies in the contextual aggregated sensor data representation710. These redundancies are produced by content that repeats two or more times from overlapping fields of view of the multiple sensors202,204,206,304. The sensors202,204,206,304overlap and their associated overlapping sensor data acts as an information melting agent that is used by the example global encoder sub-net703of the dimensionally interleaving encoder406(FIG. 4) to fuse the partial encoded representations (e.g., the encoded-compact sparse sensor data representation (Fi(t)) of the second phase (2)706) at overlapping data of the whole-environment representation. For example, the example global encoder sub-net703implements the dimensionally interleaving encoder406to generate a contextual fused sensor data representation716as depicted in fourth phase (4)718of the pipe-line inFIG. 7. The training procedure and the topology of the example global encoder sub-net703is substantially similar or identical to the training principle and architecture of the auto-encoder600described above in connection withFIG. 6.

By fusing the encoded-compact sparse sensor data representation (Fi(t)) of the second phase (2)706into the contextual fused sensor data representation716, the global encoder sub-net703creates a second level of compression that wholly represents the environment302in which the autonomous vehicle100is located. In the illustrated example ofFIG. 7, the contextual fused sensor data representation716is a whole-scene context fused representation (CFR) of the environment302based on sensor data (Ii(x,y,t)) from multiple ones of the sensors202,204,206,304.

The contextual fused sensor data representation716of the fourth phase (4)718removes redundancies (from multiple ones of the sensors202,204,206,304) that were still present in the contextual aggregated sensor data representation710. At the same time, the contextual fused sensor data representation716combines common or matching aspects from the different encoded-compact sparse sensor data representations (Fi(t)) in the contextual aggregated sensor data representation710. Such combining of common or matching aspects from the different encoded-compact sparse sensor data representations (Fi(t)) reduces the amount of data used to represent the input sensor data (Ii(x,y,t)) relative to the amount of data of the contextual aggregated sensor data representation710. In addition, such combining of common or matching aspects emphasizes anomalies in the input sensor data (Ii(x,y,t)) because decoding a reconstructed version (Ii(x,y,t)′) of the input sensor data (Ii(x,y,t)) based on the contextual fused sensor data representation716at the fifth phase (5)720and the sixth phase (6)722will not result in a sufficiently similar version of the input sensor data (Ii(x,y,t)) as described below. A sufficiently large deviation between reconstructed input sensor data (Ii(x,y,t)′) and original input sensor data (Ii(x,y,t)) is indicative of an anomaly that could be due to a failing sensor, an obstructed sensor, a malicious attack on the autonomous vehicle100, etc.

The example contextual fused sensor data representation716is a larger information source than each individual reconstructed input sensor data (Ii(x,y,t)′). Thus, the decoding processes of the sixth phase (6)722also generate an uncertainty model corresponding to the deviation of the reconstructed input sensor data (Ii(x,y,t)′) relative to the original input sensor data (Ii(x,y,t)), not relative to the larger-size CFR represented in the example contextual fused sensor data representation716. After completion of the sixth phase (6)722of the pipe-line, the extraction of the reconstructed input sensor data (Ii(x,y,t)′) has been completed.

At a seventh phase (7)724ofFIG. 7, the example extractive deviation distribution analyzer410(FIG. 4) performs a deviation analysis on the reconstructed input sensor data (Ii(x,y,t)′) and the original input sensor data (Ii(x,y,t)) to determine a probabilistic deviation estimation in the form of a statistical deviation model N(μi, σi2). The deviation analysis assesses the deviation of the reconstructed input sensor data (Ii(x,y,t)′) relative to corresponding ones of the original input sensor data (Ii(x,y,t)) from ones of the sensors202,204,206,304. For example, at the seventh phase (7)724, point-to-point or pixel-to-pixel deviations are evaluated based on PDFs (illustrated by normal distributions) of the reconstructed input signals (Ii(x,y,t)′) to produce a density value D(Ii(x,y,t), Ii(x,y,t)). Density values D(Ii(x,y,t), Ii(x,y,t)) are closer to one when there is more similarity between the values of the reconstructed sensor signals (Ii(x,y,t)′) and corresponding ones of the original signals (Ii(x,y,t)). Inversely, density values D(Ii(x,y,t), Ii(x,y,t)) decay as a function of the PDF. This deviation analysis process produces a deviation index (Si) for the original input sensor data (Ii(x,y,t)) of each sensor202,204,206,304. The deviation index (Si) is represented as shown in Equation 1 below.
Si:=1/hwΣ{(x,y)}log(N(li(x,y)−li′(x,y))  (Eq. 1)

In Equation 1 above, the variable Si is an anomaly or abnormal behavior index for a sensor channel (i) (e.g., a sensor channel corresponding to one of the sensors202,204,206,304), the variable h is the image or scan height, the variable w is the image or scan width, the variable Ii is the original input image (e.g., original input sensor data), the variable Ii′ is the reconstructed image (e.g., reconstructed sensor data) also referred to as an encoded-decoded image, the variable x is a horizontal pixel or beam index, the variable y is a vertical pixel or beam index, N is a fitted probabilistic distribution model (e.g., a statistical deviation model) for a sensor channel (i) (e.g., a Gaussian distribution or normal distribution). The deviation indices (Si) of Equation 1 above are depicted at an eighth phase (8)726ofFIG. 7.

FIG. 8shows a graph800of example line plots of deviation indices (Si) over time (t). In the illustrated example ofFIG. 8, for each of the deviation indices (Si), there is a learned threshold (Θi) obtained for each sensor channel (i) as the maximal deviation index (Si) computed during the training. As such, an abnormal behavior trigger function κ is a function of the deviation indices (Si) (e.g., an anomaly or abnormal behavior index for a sensor channel (i)) and the threshold (Θi) as shown in Equation 2 below.
κ(Si,Θi):={1 ifSi<Θi, else 0}  (Eq. 2)

Equation 2 above is used by the example anomaly detector412ofFIG. 4to determine when a deviation is considered as abnormal based on the per-sensor-channel threshold (Θi). The per-sensor-channel threshold value (Θi) is selected as a level that, when exceeded (e.g., satisfied), indicates an anomaly in sensor data. The threshold value (Θi) may be selected per sensor channel (i) autonomously by a processor based on analyzing historical anomalies. For example, the threshold value (Θi) may be selected based on the statistical deviation model N(μi, σi2) such that Θi=μ+4σ, where the variable μ represents the mean error, and the variable σ represents the standard deviation of the mean error. The per-sensor-channel threshold value (Θi), the mean error μ, and the standard deviation a are single scalar, one-dimensional data. Alternatively, instead of a processor autonomously selecting the per-sensor-channel (i) threshold value (Θi), the per-sensor-channel (i) threshold value (Θi) may be selected by a user. In this manner, the anomaly detector412detects an anomaly in a probabilistic deviation estimation represented by a deviation index (Si) based on the deviation index (Si) and the threshold (Θi) evaluated using the abnormal behavior trigger function κ is of Equation 2 above. In such examples, the detected anomaly is indicative of an error in operation of a sensor202,204,206,304corresponding to the deviation index (Si) for which the anomaly was detected. In the illustrated example ofFIG. 8, anomalies are shown as indicated by reference number802for line plots of three deviation indices (Si) that exceed (e.g., satisfy) the threshold (Θi). In Equation 2, a value of one (1) for the trigger function κ is indicative of no anomaly, while a value of zero (0) for the trigger function κ is indicative of a detected anomaly.

In addition, the anomaly detector412can also generate per-sensor-channel confidence scores (Ci) for ones of the sensors202,204,206,304. A confidence score (Ci) is expressed as a density in accordance with Equation 3 below.
Ci=e(−Si/Θi)(Eq. 3)

In Equation 3 above, the per-sensor-channel confidence score (Ci) is equal to the exponential function (e) for a quotient of the per-sensor-channel deviation index (Si) divided by the per-sensor-channel threshold value (Θi). The confidence score (Ci) is a metric that represents a level of confidence in a sensor's input sensor data (Ii(x,y,t)) relative to an overall acquired context from multiple sensors of the autonomous vehicle100at a particular timestamp (t). The confidence score (Ci) can be used as a gain or density-per-sensor which is useful for use in multiple probabilistic inference and state assertion techniques. The confidence score (Ci) is single scalar, one-dimensional data.

FIG. 9is an example technique for generating the example contextual aggregated sensor data representation710(FIG. 7) of input sensor data (Ii(x,y,t)). The illustrated example ofFIG. 9represents an example process conducted to arrange the dimensions from the compressed representations of sensor data from the sensors202,204,206,304(FIGS. 2 and 3). This example process generates a dimension-by-dimension interleave arrangement (e.g., the horizontal dimension (x), the vertical dimension (y), the time dimension (t)) of the encoded-compact sparse sensor data representations (Fi(t)) corresponding to the multiple sensors202,204,206,304to form the contextual aggregated sensor data representation710. In other words, the contextual aggregated sensor data representation710is composed by individually stacking corresponding dimensions or like dimensions of the multiple encoded-compact sparse sensor data representations (Fi(t)). That is, data at x,y,t dimension values from one encoded-compact sparse sensor data representation (Fi(t)) are stacked along data of matching x,y,t dimension values of another encoded-compact sparse sensor data representation (Fi(t)).

The example process900ofFIG. 9to generate the contextual aggregated sensor data representation710(CAR(t)) is shown as three operations including an example sensor encoding operation902, an example expanded sensor encoding operation904, and an example contextual aggregated representation operation906. At the example sensor encoding operation902, the multimodal encoder404(FIGS. 4 and 7) sorts a quantity (n) of encoded-compact sparse sensor data representations (Fi(t)) stored in a list U by length (number of dimensions (dim(Fi(t))) in such a way that the first in the list U is the shortest encoding.

At the example expanded sensor encoding operation904, the aggregator405(FIGS. 4 and 7) creates an empty contextual fused representation CFR as a square matrix of dimensions (n), such as if l=Σindim(Fn(t)) then n=ceil(l1/2).

At the example contextual aggregated representation operation906, the aggregator405obtains the first element (e.g., F0(t)) in the list (U) and copies each of its dimension values to the CRF to the closest empty dimension, assuming a regular distribution. For example, if the dimension is n=2000, then the CFR has a dimension n2=4000000. In addition, if in this example dim(F0(t))=1672, a filling factor E0=round(n2/dim(F0(t)))=2392, which means that an insertion of an encoded-compact sparse sensor data representations (Fi(t)) will happen approximately each 2392 dimensions. This is an approximation due to accounting for possible index collations. In such examples, the index is shifted into the next dimension until an empty slot (in this sense of dimension value) is found. While the aggregator405computes this index to interleave the data of the encoded-compact sparse sensor data representation (Fi(t)) into the CFR, the aggregator405employs a module n2, resulting in a circular process that converges based on the calculations at the example contextual aggregated representation operation906.

When any additional encoded-compact sparse sensor data representation (Fi(t)) remains in the list (U), control returns to the example expanded sensor encoding operation904to process the next encoded-compact sparse sensor data representation (Fi(t)). When there are no more encoded-compact sparse sensor data representations (Fi(t)) in the list (U), the example aggregator405finalizes the example process900. As a result of the interleaving of the contextual aggregated representation operation906for the encoded-compact sparse sensor data representations (Fi(t)) in the list (U), the aggregator405generates the contextual aggregated sensor data representation710ofFIG. 7and shown inFIG. 9as CAR(t).

FIGS. 10A and 10Bdepict an example end-to-end system training data flow1000of the anomaly detection apparatus306ofFIGS. 3 and 4to perform unsupervised multimodal anomaly detection for autonomous vehicles using the example feature fusion and deviation data flow700ofFIG. 7. The example end-to-end system training data flow1000is shown as including six phases. At an example first phase (1)1002(FIG. 10A), the sensor data interface402(FIG. 4) obtains multimodal raw sensor data samples (Ii(x,y,t)) from a database to train auto-encoders for corresponding ones of the sensors202,204,206,304(FIGS. 2 and 3). Also at the first phase (1)1002, the multimodal encoder404(FIGS. 4 and 7) employs trained auto-encoders to process the raw sensor data to compute encoded-compact sparse sensor data representations (Fi(t))606(FIG. 6) (e.g., encoded sensor data). The auto-encoders of the first phase (1)1002correspond to the encoder sub-nets702a-dofFIG. 7and are only a part of the composed encoder-decoder schema being trained in the example end-to-end system training data flow1000. The auto-encoder being trained at the third phase (3)1006corresponds to the global encoder sub-net703ofFIG. 7.

At the example second phase (2)1004, the example dimensionally interleaving encoder406(FIGS. 4 and 7) aggregates the encoded-compact sparse sensor data representations (Fi(t))606to generate a dataset of contextual fused representations for each of the multimodal raw sensor data samples (Ii(x,y,t)). The dataset of contextual fused representations of the second phase (2)1004form the contextual aggregated sensor data representation710ofFIGS. 7 and 9. At the example third phase (3)1006, the contextual aggregated representation samples of the contextual aggregated sensor data representation710are provided to train the encoder sub-net703ofFIG. 7.

At the example fourth phase (4)1008(FIG. 10B), the dimensionally interleaving encoder406(FIGS. 4 and 7) employs the encoder sub-net703to process the contextual aggregated representation samples of the contextual aggregated sensor data representation710and generate the contextual fused sensor data representation716(FIG. 7). At the example fifth phase (5)1010, The contextual fused sensor data representation716is used to train the decoder sub-nets705a-dper sensor202,204,206,304. The input of each decoder sub-net705a-dis the same contextual fused sensor data representation716, and the output is reconstructed input sensor data (Ii(x,y,t)′) of a corresponding original raw input sensor data (Ii(x,y,t)).

At the example sixth phase (6)1012, raw sensor data (Ii(x,y,t)) is obtained from corresponding ones of the sensors202,204,206,304, corresponding reconstructed sensor data (Ii(x,y,t)′) is obtained from corresponding trained decoder sub-nets705a-d, and the extractive deviation distribution analyzer410generates a parametric distribution based on this information. The extractive deviation distribution analyzer410fits the difference between the raw sensor data (Ii(x,y,t)) and the corresponding reconstructed sensor data (Ii(x,y,t)′). A threshold value (Θi) per sensor202,204,206,304is estimated for each of the fitted distributions. Subsequently, the trained encoder sub-nets702a-dof the first phase (1)1004, the trained encoder sub-net703of the third phase (3)1006, the trained decoder sub-nets705a-dof the fifth phase (5)1010, and the estimated threshold value (Θi) parameters from the sixth phase (6)1012are used to detect anomalies in one or more of the sensors202,204,206,304.

While an example manner of implementing the autonomous driving apparatus300and the anomaly detection apparatus306ofFIG. 3is illustrated inFIGS. 3 and 4, one or more of the elements, processes and/or devices illustrated inFIGS. 3 and 4may be combined, divided, re-arranged, omitted, eliminated and/or implemented in any other way. Further, the example scene contextualizer308, the example scene analyzer312, the example drivability map generator314, the example control scheduler316, the example controller318, the example sensor data interface402, the example multimodal encoder404, the example aggregator405, the example dimensionally interleaving encoder406, the example extractive decoder408, the example extractive deviation distribution analyzer410, the example anomaly detector412, and/or more generally, the example autonomous driving apparatus300and/or the example anomaly detection apparatus306ofFIGS. 3 and 4may be implemented by hardware, software, firmware and/or any combination of hardware, software and/or firmware. Thus, for example, any of the example scene contextualizer308, the example scene analyzer312, the example drivability map generator314, the example control scheduler316, the example controller318, the example sensor data interface402, the example multimodal encoder404, the example aggregator405, the example dimensionally interleaving encoder406, the example extractive decoder408, the example extractive deviation distribution analyzer410, the example anomaly detector412, and/or more generally, the example autonomous driving apparatus300and/or the example anomaly detection apparatus306could be implemented by one or more analog or digital circuit(s), logic circuits, programmable processor(s), programmable controller(s), graphics processing unit(s) (GPU(s)), digital signal processor(s) (DSP(s)), application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)) and/or field programmable logic device(s) (FPLD(s)). When reading any of the apparatus or system claims of this patent to cover a purely software and/or firmware implementation, at least one of the example scene contextualizer308, the example scene analyzer312, the example drivability map generator314, the example control scheduler316, the example controller318, the example sensor data interface402, the example multimodal encoder404, the example aggregator405, the example dimensionally interleaving encoder406, the example extractive decoder408, the example extractive deviation distribution analyzer410, and/or the example anomaly detector412is/are hereby expressly defined to include a non-transitory computer-readable storage device or storage disk such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc. including the software and/or firmware. Further still, the example autonomous driving apparatus300and/or the example anomaly detection apparatus306ofFIGS. 3 and 4may include one or more elements, processes and/or devices in addition to, or instead of, those illustrated inFIGS. 3 and 4, and/or may include more than one of any or all of the illustrated elements, processes and devices. As used herein, the phrase “in communication,” including variations thereof, encompasses direct communication and/or indirect communication through one or more intermediary components, and does not require direct physical (e.g., wired) communication and/or constant communication, but rather additionally includes selective communication at periodic intervals, scheduled intervals, aperiodic intervals, and/or one-time events.

A flowchart representative of example hardware logic, machine readable instructions, hardware implemented state machines, and/or any combination thereof for implementing the example autonomous driving apparatus300and/or the example anomaly detection apparatus306ofFIG. 3andFIG. 4is illustrated inFIG. 11. The machine readable instructions may be an executable program or portion of an executable program for execution by a computer processor such as the processor1212shown in the example processor platform1200discussed below in connection withFIG. 12. The program may be embodied in software stored on a non-transitory computer-readable storage medium such as a CD-ROM, a floppy disk, a hard drive, a DVD, a Blu-ray disk, or a memory associated with the processor1212, but the entire program and/or parts thereof could alternatively be executed by a device other than the processor1212and/or embodied in firmware or dedicated hardware. Further, although the example program is described with reference to the flowchart illustrated inFIG. 11, many other methods of implementing the example autonomous driving apparatus300and/or the example anomaly detection apparatus306may alternatively be used. For example, the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and/or integrated analog and/or digital circuitry, an FPGA, an ASIC, a comparator, an operational-amplifier (op-amp), a logic circuit, etc.) structured to perform the corresponding operation without executing software or firmware.

As mentioned above, the example process ofFIG. 11may be implemented using executable instructions (e.g., computer and/or machine readable instructions) stored on a non-transitory computer-readable and/or machine-readable medium such as a hard disk drive, a flash memory, a read-only memory, a compact disk, a digital versatile disk, a cache, a random-access memory and/or any other storage device or storage disk in which information is stored for any duration (e.g., for extended time periods, permanently, for brief instances, for temporarily buffering, and/or for caching of the information). As used herein, the term non-transitory computer-readable medium is expressly defined to include any type of computer readable storage device and/or storage disk and to exclude propagating signals and to exclude transmission media.

Turning now in detail toFIG. 11, the depicted example process may be used to perform unsupervised multimodal anomaly detection for the autonomous vehicle100. The example program ofFIG. 11begins at block1102at which the example sensor data interface402(FIG. 4) obtains first collected sensor data (Ii(x,y,t)) from a first one of the sensors202,204,206,304(FIGS. 2 and 3) and second collected sensor data (Ii(x,y,t)) from a second one of the sensors202,204,206,304. For example, the first one of the sensors is of a first sensor type different than a second sensor type of the second one of the sensors. In addition, the first collected sensor data (Ii(x,y,t)) and the second collected sensor data (Ii(x,y,t)) are of a multi-dimensional format.

The example multimodal encoder404(FIG. 4) generates encoded sensor data (block1104) in the form of the encoded-compact sparse sensor data representation (Fi(t))606(FIG. 6). For example, the multimodal encoder404generates first encoded sensor data based on the first collected sensor data (Ii(x,y,t)) and second encoded sensor data based on the second collected sensor data (Ii(x,y,t)).

The example aggregator405(FIG. 4) generates a contextual aggregated sensor data representation (block1106) of collected sensor data based on the encoded sensor data. For example, the aggregator405generates the contextual aggregated sensor data representation710of the first and second collected sensor data (Ii(x,y,t)) based on the first and second encoded sensor data determined at block1104(e.g., in the form of the encoded-compact sparse sensor data representation (Fi(t))606).

The example dimensionally interleaving encoder406(FIGS. 4 and 7) generates a contextual fused sensor data representation (block1108). For example, the dimensionally interleaving encoder406generates the contextual fused sensor data representation716(FIG. 7) based on the contextual aggregated sensor data representation710generated at block1106. The example extractive decoder408(FIGS. 4 and 7) generates reconstructed sensor data (block1110). For example, the extractive decoder408decodes first reconstructed sensor data (Ii(x,y,t)′) and second reconstructed sensor data (Ii(x,y,t)′) from the contextual fused sensor data representation716.

The example extractive deviation distribution analyzer410determines a deviation estimation (block1112). For example, the extractive deviation distribution analyzer410determines the probabilistic deviation estimation in the form of a statistical deviation model N(μi, σi2) based on the first reconstructed sensor data (Ii(x,y,t)′) and the second reconstructed sensor data (Ii(x,y,t)′) decoded from the contextual fused sensor data representation716. The first reconstructed sensor data (Ii(x,y,t)′) corresponds to the first collected sensor data (Ii(x,y,t)), and the second reconstructed sensor data (Ii(x,y,t)′) corresponds to the second collected sensor data (Ii(x,y,t)). The probabilistic deviation estimation represents likelihoods of point-to-point deviations or pixel-to-pixel deviations between: (a) the first reconstructed sensor data (Ii(x,y,t)′), and (b) the first collected sensor data (Ii(x,y,t)) from the first one of the sensors202,204,206,304. In some examples, the extractive deviation distribution analyzer410can also determine a probabilistic deviation estimation representing likelihoods of point-to-point deviations or pixel-to-pixel deviations between: (a) the second reconstructed sensor data (Ii(x,y,t)′), and (b) the second collected sensor data (Ii(x,y,t)) from the second one of the sensors202,204,206,304.

The example anomaly detector412(FIG. 4) determines confidence scores (Ci) for the sensors202,204,206,304(block1114). For example, the anomaly detector412determines the confidence scores (Ci) based on Equation 3 above as a metric to represent a confidence level of the reliability of a sensor's input sensor data (Ii(x,y,t)). In some examples, such confidence scores (Ci) can be used to assess the likelihood that an anomaly detected by the anomaly detector412is merely an anomaly or an actual feature detected in the environment302(FIG. 3).

The example anomaly detector412determines whether it detects an anomaly in the probabilistic deviation estimation (block1116). A detected anomaly is indicative of an error in operation of the first one of the sensors. For example, the anomaly detector412detects an anomaly when the probabilistic deviation estimation (determined at block1112) exceeds (e.g., satisfies) a threshold value (Θi) for the corresponding sensor as described above in connection withFIGS. 7 and 8.

If an anomaly is detected at block1116, the anomaly detector412generates an anomaly notification (block1118). For example, the anomaly notification may be in the form of an electronic message communicated by the anomaly detector412via a bus and/or stored by the anomaly detector412in memory for subsequent retrieval by another component of the example autonomous driving apparatus300and/or the example anomaly detection apparatus306. In the illustrated example, the anomaly detector412provides the anomaly notification to the scene contextualizer308and/or the scene analyzer312ofFIG. 3. For example, the anomaly notification identifies the collected sensor data (Ii(x,y,t)) (e.g., a record entry of the sensor data) and its corresponding one of the sensors as exhibiting abnormal characteristics about the environment302. In some examples, the anomaly detector412stores an identifier of the collected sensor data (Ii(x,y,t)) (e.g., a key value or memory location of a record entry at which the sensor data is stored in memory) in the anomaly notification. In some examples, the anomaly detector412stores a sensor identifier of the corresponding sensor in the anomaly notification. In some examples, the anomaly detector412also stores the confidence scores (Ci) corresponding to the sensors202,204,206,304in the anomaly notification.

The scene contextualizer308and/or the scene analyzer312ofFIG. 3adjust system operation based on the anomaly notification (block1120). For example, the scene contextualizer308and/or the scene analyzer312can adjust system operation of the autonomous driving apparatus300(FIG. 3) by accounting for the anomaly and performing mitigating procedures to discard the anomalous collected sensor data (Ii(x,y,t)) and/or replace the anomalous collected sensor data (Ii(x,y,t)) with more reliable collected sensor data (Ii(x,y,t)) from another sensor. In some examples, the scene contextualizer308and/or the scene analyzer312select another sensor that is deemed more reliable based on the confidence scores (Ci) determined by the anomaly detector412.

After adjusting system operation at block1120, or if an anomaly is not detected at block1116, the example sensor data interface402determines whether there is additional collected sensor data (Ii(x,y,t)) to process (block1122). For example, there may be additional collected sensor data (Ii(x,y,t)) corresponding to a subsequent timestamp (t) that is to be processed. In this manner, the process ofFIG. 11can repeat for multiple timestamps (t) to process collected sensor data (Ii(x,y,t)) in real time while the autonomous vehicle100is operating. If there is additional collected sensor data (Ii(x,y,t)) to process, control returns to block1102. Otherwise, the example process ofFIG. 11ends. For example, the autonomous vehicle100is no longer operating (e.g., it is turned off).

FIG. 12is a block diagram of an example processing platform1200structured to execute the instructions ofFIG. 11to implement the example autonomous driving apparatus300ofFIG. 3and/or the example anomaly detection apparatus306ofFIGS. 3 and 4to perform unsupervised multimodal anomaly detection for the example autonomous vehicle100(FIGS. 1-5). The processor platform1200can be, for example, a server, a computer, a self-learning machine (e.g., a neural network), or any other type of computing device.

The processor platform1200of the illustrated example includes a processor1212. The processor1212of the illustrated example is hardware. For example, the processor1212can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor1212may be a semiconductor based (e.g., silicon based) device. In this example, the processor implements the example scene contextualizer308, the example scene analyzer312, the example drivability map generator314, the example control scheduler316, the example controller318, the example sensor data interface402, the example multimodal encoder404, the example aggregator405, the example dimensionally interleaving encoder406, the example extractive decoder408, the example extractive deviation distribution analyzer410, and the example anomaly detector412.

The processor1212of the illustrated example includes a local memory1213(e.g., a cache). The processor1212of the illustrated example is in communication with a main memory including a volatile memory1214and a non-volatile memory1216via a bus1218. The volatile memory1214may be implemented by Synchronous Dynamic Random Access Memory (SDRAM), Dynamic Random Access Memory (DRAM), RAMBUS® Dynamic Random Access Memory (RDRAM®) and/or any other type of random access memory device. The non-volatile memory1216may be implemented by flash memory and/or any other desired type of memory device. Access to the main memory1214,1216is controlled by a memory controller.

The processor platform1200of the illustrated example also includes an interface circuit1220. The interface circuit1220may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), a Bluetooth® interface, a near field communication (NFC) interface, a PCI express interface, an I2C bus, and/or a control area network (CAN) bus.

In the illustrated example, one or more input devices1222are connected to the interface circuit1220. Some input device(s)1222permit(s) a user to enter data and/or commands into the processor1212. The input device(s) can be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, a button, a mouse, a touchscreen, a track-pad, a trackball, isopoint and/or a voice recognition system. In example disclosed herein, the sensors202,204,206,304ofFIGS. 2 and 3are input devices that provide collected sensor data (Ii(x,y,t)) to the example autonomous driving apparatus300and/or the example anomaly detection apparatus306via the interface circuit1220.

The processor platform1200of the illustrated example also includes one or more mass storage devices1228for storing software and/or data. Examples of such mass storage devices1228include floppy disk drives, hard drive disks, compact disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

Machine executable instructions1232representative of the machine executable instruction ofFIG. 11may be stored in the mass storage device1228, in the volatile memory1214, in the non-volatile memory1216, and/or on a removable non-transitory computer-readable storage medium such as a CD or DVD.

From the foregoing, it will be appreciated that example methods, apparatus and articles of manufacture have been disclosed that perform unsupervised multimodal anomaly detection for autonomous vehicles. Examples disclosed herein may employ redundant heterogeneous sensor configurations to implement high-dimensional feature fusion that works independent of the types of sensors and/or independent of specific data formats by using unsupervised machine learning. That is, unsupervised multimodal anomaly detection techniques for autonomous vehicles disclosed herein are not limited to use with particular types of sensors or particular types of data structure for representing environmental characteristics. This increases the flexibility of a computer in an autonomous vehicle by enabling the autonomous vehicle computer to implement autonomous operation of the vehicle based on different types of sensors. This is advantageous over prior techniques designed for analyzing sensor data from homogeneous sensor type configurations (configurations of multiple sensors of the same type), specific quantities of sensors and actuators, specific signal structures, and specific routines that are specifically developed for a particular task and environment. Examples disclosed herein may be flexibly implemented across heterogeneous sensor configurations in which the quantity of sensors can vary over time. For example, one or more sensors may drop out or malfunction over time in an autonomous vehicle. In such situations, examples disclosed herein increase the flexibility and robustness of an autonomous vehicle computer by making the autonomous vehicle computer sufficiently unaffected by such changes in sensor operation. For example, by virtue of recognizing such changes as anomalous and taking such anomalous activity into account, the autonomous vehicle computer can refrain from producing unexpected or abnormal operation of the autonomous vehicle that is inconsistent with the actual environmental conditions in which the autonomous vehicle is operating. The disclosed methods, apparatus and articles of manufacture are accordingly directed to one or more improvement(s) in the functioning of a computer.

While example unsupervised multimodal anomaly detection examples disclosed herein are mostly described with respect to autonomous vehicles, examples disclosed herein can be used in many other non-vehicle applications as well. For example, examples disclosed herein may be used with multiple types of sensors in many applications such as to monitor a manufacturing line, to detect and identify people at an airport, and/or any other situation where different types of sensors are used together.

The following pertain to further examples disclosed herein.

Example 1 is an apparatus to detect an anomaly based on heterogeneous sensor data of an autonomous vehicle. The apparatus of Example 1 includes a sensor data interface to obtain first collected sensor data from a first sensor and second collected sensor data from a second sensor, the first sensor of a first sensor type different than a second sensor type of the second sensor; a multimodal encoder to generate first encoded sensor data based on the first collected sensor data and second encoded sensor data based on the second collected sensor data; a dimensionally interleaving encoder to generate a contextual fused sensor data representation of the first and second collected sensor data based on the first and second encoded sensor data; an extractive decoder to generate first reconstructed sensor data and second reconstructed sensor data based on the contextual fused sensor data representation; an extractive deviation distribution analyzer to determine a deviation estimation based on the first reconstructed sensor data and the second reconstructed sensor data, the deviation estimation representative of a deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor; and an anomaly detector to detect an anomaly in the deviation estimation, the anomaly indicative of an error associated with the first sensor.

In Example 2, the subject matter of Example 1 can optionally include that the first sensor is a visible light camera and the second sensor is a Light Detection and Ranging (LIDAR) sensor.

In Example 3, the subject matter of any one of Examples 1-2 can optionally include that the anomaly detector is further to determine confidence scores representative of confidence levels of reliabilities of the first and second collected sensor data.

In Example 4, the subject matter of any one of Examples 1-3 can optionally include that the anomaly detector is to generate an anomaly notification based on the detected anomaly, the anomaly notification to include at least one of: (a) an identifier of the first collected sensor data, (b) a sensor identifier of the first sensor, or (c) a confidence score representative of a confidence level of reliability corresponding to the first sensor.

In Example 5, the subject matter of any one of Examples 1-4 can optionally include a scene analyzer to adjust operation of the autonomous vehicle based on the detected anomaly.

In Example 6, the subject matter of any one of Examples 1-5 can optionally include that the scene analyzer is to adjust operation of the autonomous vehicle by replacing the first collected sensor data with the second collected sensor data.

In Example 7, the subject matter of any one of Examples 1-6 can optionally include that the scene analyzer is to select the second collected sensor data to replace the first collected sensor data based on a confidence score representative of a confidence level of reliability corresponding to the second sensor.

In Example 8, the subject matter of any one of Examples 1-7 can optionally include that the anomaly detector is to detect the anomaly in the deviation estimation based on a threshold being satisfied by the deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor.

In Example 9, the subject matter of any one of Examples 1-8 can optionally include an aggregator to generate a contextually aggregated sensor data representation based on the first and second encoded sensor data, the dimensionally interleaving encoder to generate the contextual fused sensor data representation based on the first and second encoded sensor data as represented in the contextually aggregated sensor data representation.

In Example 10, the subject matter of any one of Examples 1-9 can optionally include that the first collected sensor data and the second collected sensor data are of a multi-dimensional format.

Example 11 is a non-transitory computer-readable storage medium comprising instructions that, when executed, cause at least one processor to at least: obtain first collected sensor data from a first sensor and second collected sensor data from a second sensor, the first sensor of a first sensor type different than a second sensor type of the second sensor; generate first encoded sensor data based on the first collected sensor data and second encoded sensor data based on the second collected sensor data; generate a contextual fused sensor data representation of the first and second collected sensor data based on the first and second encoded sensor data; generate first reconstructed sensor data and second reconstructed sensor data based on the contextual fused sensor data representation; determine a deviation estimation based on the first reconstructed sensor data and the second reconstructed sensor data, the deviation estimation representative of a deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor; and detect an anomaly in the deviation estimation, the anomaly indicative of an error associated with the first sensor.

In Example 12, the subject matter of Example 11 can optionally include that the first sensor is a visible light camera and the second sensor is a Light Detection and Ranging (LIDAR) sensor.

In Example 13, the subject matter of any one of Examples 11-12 can optionally include that the instructions are further to cause the at least one processor to determine confidence scores representative of confidence levels of reliabilities of the first and second collected sensor data.

In Example 14, the subject matter of any one of Examples 11-13 can optionally include that the instructions are further to cause the at least one processor to generate an anomaly notification based on the detected anomaly, the anomaly notification to include at least one of: (a) an identifier of the first collected sensor data, (b) a sensor identifier of the first sensor, or (c) a confidence score representative of a confidence level of reliability corresponding to the first sensor.

In Example 15, the subject matter of any one of Examples 11-14 can optionally include that the instructions are further to cause the at least one processor to adjust operation of an autonomous vehicle based on the detected anomaly.

In Example 16, the subject matter of any one of Examples 11-15 can optionally include that the instructions are to cause the at least one processor to adjust operation of the autonomous vehicle by replacing the first collected sensor data with the second collected sensor data.

In Example 17, the subject matter of any one of Examples 11-16 can optionally include that the instructions are to cause the at least one processor to select the second collected sensor data to replace the first collected sensor data based on a confidence score representative of a confidence level of reliability corresponding to the second sensor.

In Example 18, the subject matter of any one of Examples 11-17 can optionally include that the instructions are to cause the at least one processor to detect the anomaly in the deviation estimation based on a threshold being satisfied by the deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor.

In Example 19, the subject matter of any one of Examples 11-18 can optionally include that the instructions are further to cause the at least one processor to generate a contextually aggregated sensor data representation based on the first and second encoded sensor data, the contextual fused sensor data representation generated based on the first and second encoded sensor data as represented in the contextually aggregated sensor data representation.

In Example 20, the subject matter of any one of Examples 11-19 can optionally include that the first collected sensor data and the second collected sensor data are of a multi-dimensional format.

Example 21 is a method to detect an anomaly based on heterogeneous sensor data of an autonomous vehicle. The method of Example 21 includes obtaining first collected sensor data from a first sensor and second collected sensor data from a second sensor, the first sensor of a first sensor type different than a second sensor type of the second sensor; generating, by executing an instruction with a processor, first encoded sensor data based on the first collected sensor data and second encoded sensor data based on the second collected sensor data; generating, by executing an instruction with the processor, a contextual fused sensor data representation of the first and second collected sensor data based on the first and second encoded sensor data; generating, by executing an instruction with the processor, first reconstructed sensor data and second reconstructed sensor data based on the contextual fused sensor data representation; determining, by executing an instruction with the processor, a deviation estimation based on the first reconstructed sensor data and the second reconstructed sensor data, the deviation estimation representative of a deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor; and detecting, by executing an instruction with the processor, an anomaly in the deviation estimation, the anomaly indicative of an error associated with the first sensor.

In Example 22, the subject matter of Example 21 can optionally include that the first sensor is a visible light camera and the second sensor is a Light Detection and Ranging (LIDAR) sensor.

In Example 23, the subject matter of any one of Examples 21-22 can optionally include determining confidence scores representative of confidence levels of reliabilities of the first and second collected sensor data.

In Example 24, the subject matter of any one of Examples 21-23 can optionally include generating an anomaly notification based on the detected anomaly, the anomaly notification to include at least one of: (a) an identifier of the first collected sensor data, (b) a sensor identifier of the first sensor, or (c) a confidence score representative of a confidence level of reliability corresponding to the first sensor.

In Example 25, the subject matter of any one of Examples 21-24 can optionally include adjusting operation of the autonomous vehicle based on the detected anomaly.

In Example 26, the subject matter of any one of Examples 21-25 can optionally include that the adjusting of the operation of the autonomous vehicle includes replacing the first collected sensor data with the second collected sensor data.

In Example 27, the subject matter of any one of Examples 21-26 can optionally include selecting the second collected sensor data to replace the first collected sensor data based on a confidence score representative of a confidence level of reliability corresponding to the second sensor.

In Example 28, the subject matter of any one of Examples 21-27 can optionally include that the detecting of the anomaly in the deviation estimation is based on a threshold being satisfied by the deviation between: (a) the first reconstructed sensor data, and (b) the first collected sensor data from the first sensor.

In Example 29, the subject matter of any one of Examples 21-28 can optionally include generating a contextually aggregated sensor data representation based on the first and second encoded sensor data, the contextual fused sensor data representation generated based on the first and second encoded sensor data as represented in the contextually aggregated sensor data representation.

In Example 30, the subject matter of any one of Examples 21-29 can optionally include that the first collected sensor data and the second collected sensor data are of a multi-dimensional format.