REAL-TIME DEEP LEARNING FOR DANGER PREDICTION USING HETEROGENEOUS TIME-SERIES SENSOR DATA

A computer-implemented method and a system are provided for, in turn, providing driver assistance for a vehicle. The method includes forming, by a processor, a deep High-Order Long Short-Term Memory (HOLSTM)-based model by applying, to a HOLSTM, high-order interactions captured between global pattern distribution probabilities and local feature representations of an input sensor signal vector at each of a plurality of time steps. The input sensor signal vector is formed from multiple time series. Each of the multiple time series corresponds to a different one of a plurality of driving related sensors. The method further includes generating, by the processor, one or more predictions of impending dangerous conditions related to driving the vehicle based on the deep HOLSTM-based model. The method also includes informing, by an operator-perceptable warning device, an operator of the vehicle of the one or more predictions of impending dangerous conditions.

BACKGROUND

Technical Field

The present invention relates to data processing and more particularly to real-time deep learning for danger prediction using heterogeneous time-series sensor data.

Description of the Related Art

With the advancement of sensing and computing technology, smart vehicles have been made and are becoming more popular as commercial products. Advanced commercial vehicles with on-board cameras and sensors can even drive autonomously in some constrained traffic environments. However, making such autonomous smart vehicles is subject to many government regulations and is also highly expensive. To make affordable smart vehicles widely sold as standard automobiles, many auto manufactures are trying to design on-board sensing systems capable of understanding a surrounding driving environment and generating immediate danger alerts in real-time.

Thus, there is a need for a real-time system for danger prediction for vehicles.

SUMMARY

According to an aspect of the present invention, a computer-implemented method is provided for, in turn, providing driver assistance for a vehicle. The method includes forming, by a processor, a deep High-Order Long Short-Term Memory (HOLSTM)-based model by applying, to a HOLSTM, high-order interactions captured between global pattern distribution probabilities and local feature representations of an input sensor signal vector at each of a plurality of time steps. The input sensor signal vector is formed from multiple time series. Each of the multiple time series corresponds to a different one of a plurality of driving related sensors. The method further includes generating, by the processor, one or more predictions of impending dangerous conditions related to driving the vehicle based on the deep HOLSTM-based model. The method also includes informing, by an operator-perceptable warning device, an operator of the vehicle of the one or more predictions of impending dangerous conditions.

According to another aspect of the present invention, a computer program product is provided for, in turn, providing driver assistance for a vehicle. The computer program product includes a non-transitory computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a computer to cause the computer to perform a method. The method includes forming, by a processor, a deep High-Order Long Short-Term Memory (HOLSTM)-based model by applying, to a HOLSTM, high-order interactions captured between global pattern distribution probabilities and local feature representations of an input sensor signal vector at each of a plurality of time steps. The input sensor signal vector is formed from multiple time series. Each of the multiple time series corresponds to a different one of a plurality of driving related sensors. The method further includes generating, by the processor, one or more predictions of impending dangerous conditions related to driving the vehicle based on the deep HOLSTM-based model. The method also includes informing, by an operator-perceptable warning device, an operator of the vehicle of the one or more predictions of impending dangerous conditions.

According to yet another aspect of the present invention, a system is provided for, in turn, providing driver assistance for a vehicle. The system includes a processor. The processor is configured to form a deep High-Order Long Short-Term Memory (HOLSTM)-based model by applying, to a HOLSTM, high-order interactions captured between global pattern distribution probabilities and local feature representations of an input sensor signal vector at each of a plurality of time steps. The input sensor signal vector is formed from multiple time series. Each of the multiple time series corresponds to a different one of a plurality of driving related sensors. The processor is further configured to generate one or more predictions of impending dangerous conditions related to driving the vehicle based on the deep HOLSTM-based model. The system also includes an operator-perceptable warning device configured to inform an operator of the vehicle of the one or more predictions of impending dangerous conditions.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention is directed to real-time deep learning for danger prediction using heterogeneous time-series sensor data.

In an embodiment, a real-time system is provided that uses guided deep high-order recurrent neural networks based on heterogeneous time-series sensor data.

In contrast to using a simple shallow model based on a limited number of features for danger prediction, in an embodiment, the present invention provides a driving assistance system for generating immediate alerts by integrating many sources of real-time sensor data. In an embodiment, the present invention uses a deep learning approach to analyze real-time heterogeneous time-series data generated by on-board sensors such as Global Positioning System (GPS) sensors with maps, Laser Imaging Detection and Ranging (LIDAR), driving mechanics sensors, cameras, and so forth. It is to be appreciated that the preceding types of sensors are illustrative and, thus, other types of sensors can also be used in accordance with the present invention, while maintaining the spirit of the present invention.

Unlike recent deep learning approaches to autonomous driving based on standard deep convolutional neural networks applied to a stream of static input images, the present invention provides a guided deep high-order long short-term memory for modeling the original heterogeneous time series of rich sensory input signals and also the time series of learned pattern distribution probabilities of the raw (sensory input) signals.

In an embodiment, consider a set of training time series data X. For the sake of illustration, it is presumed that all the time series have the same length. However, it is to be appreciated that the present invention can readily apply to a set of training time series data having different lengths. X is n-by-m-by-T tensor, where n is the number of training time series, m is the dimensionality of the input sensory signal vector at each time step, and T is the length of each time series. At first, clustering is performed on the training data by treating X as n times T data points with dimensionality m, through which the pattern distribution probabilities of an input signal vector at each time step is obtained for each training time series. Then, a Deep High-Order Convolutional Neural Network (DHOCNN) is used to get feature presentations of an input sensory signal vector of each time step, and we concatenate the pattern distribution vector and the feature representation vector from the DHOCNN as a new input feature vector. Time series of this new combined feature vector of input sensory signals is fed into a novel Deep High-Order Long Short-Term Memory (DHOLSTM) for danger prediction or alert category prediction. A resultant model formed by the DHOLSTM captures the high-order interactions between global pattern distribution probabilities and local feature representations generated by DHOCNN, which combines both global and local information for making better decisions. The DHOLSTM is trained by standard back-propagation. Furthermore, to prevent over-fitting and increase model robustness, we use many auxiliary tasks, for which supervision labels are easy to obtain, to pre-train the DHOCNN and the DHOLSTM and guide the parameter learning based on the curriculum learning concept. Therefore, the model formed by the present invention is interchangeably referred to as a “guided deep high-order long short-term memory”.

FIG. 1shows a block diagram of an exemplary processing system100to which the invention principles may be applied, in accordance with an embodiment of the present invention. The processing system100includes at least one processor (CPU)104operatively coupled to other components via a system bus102. A cache106, a Read Only Memory (ROM)108, a Random Access Memory (RAM)110, an input/output (I/O) adapter120, a sound adapter130, a network adapter140, a user interface adapter150, and a display adapter160, are operatively coupled to the system bus102.

A speaker132is operatively coupled to system bus102by the sound adapter130. The speaker132can be used to provide an audible alarm or some other indication relating to resilient battery charging in accordance with the present invention. A transceiver142is operatively coupled to system bus102by network adapter140. A display device162is operatively coupled to system bus102by display adapter160.

Moreover, it is to be appreciated that system200described below with respect toFIG. 2is an environment for implementing respective embodiments of the present invention. Part or all of processing system100may be implemented in one or more of the elements of system200.

Further, it is to be appreciated that processing system100may perform at least part of the method described herein including, for example, at least part of method300ofFIG. 3. Similarly, part or all of system200may be used to perform at least part of method300ofFIG. 3.

FIG. 2shows a block diagram of an exemplary driving assistance system200, in accordance with an embodiment of the present invention. The driving assistance system200uses real-time deep learning for danger prediction that, in turn, uses heterogeneous time series sensor data. The driving assistance system200is included in a vehicle299.

The driving assistance system200includes an on-board computer210, a LIDAR system220, a GPS system230, a set of sensors240, and a set of on-board cameras250.

The on-board computer210includes a CPU210A for running deep learning for danger prediction. In an embodiment, the on-board computer210further includes a GPU210B for running deep learning for danger prediction.

The GPS system230includes maps and generates positional and map information.

The set of sensors240measure vehicle related parameters such as, for example, speed, acceleration, and other real-time driving-related signals.

The set of cameras250capture images/video of a real-time driving environment.

FIG. 3shows a flow diagram of an exemplary method300for driving assistance, in accordance with an embodiment of the present invention.

At step310, integrate heterogeneous time-series data from different components such as GPS, maps, cameras, and other sensors into one time series of multi-variates.

At step320, perform clustering such as a Mixture of Gaussians on training time series. Record the final clustering model. Calculate the pattern distribution probabilities of the input sensory signal vector at each time step for the training data. Combine the pattern distribution vector with a raw sensory input vector.

At step330, create auxiliary tasks for which labels are easily obtained and helpful for danger prediction.

At step340, pre-train a Deep High-Order Convolutional Neural Network (DHOCNN) for feature extraction in an auxiliary classification framework and a Deep High-Order Long Short-Term Memory (DHOLSTM) for prediction. That is, using additional labeled data from auxiliary tasks, we first pre-train the DHOCNN for better feature extraction, and then we pre-train the DHOLSTM. DHOCNN can be pre-trained by treating each time step of a time series as a data point without considering any temporal structure. DHOLSTM can be pre-trained on time series by considering temporal structures.

At step350, fine-tune the DHOCNN and the DHOLSTM.

At step360, calculate the pattern distribution probabilities of the input sensory signal vector at each time step for real-time test data using the recorded final clustering model, and combine them with the real-time sensory input signals from all sensors.

At step370, perform a test on the DHOLSTM for danger prediction and generate possible immediate alerts.

At step380, provide an alert to an operator of the vehicle of an impending danger relating to driving the vehicle.

FIG. 4shows a block diagram of an exemplary Deep High-Order Long Short-Term Memory (DHOLSTM)400, in accordance with an embodiment of the present invention.

The DHOLSTM400includes, for each time step from time step t1to time step tT, a raw sensory input (at that time step)410, pattern distribution probabilities of the sensory input vector (at that time step)420, a DHOCNN (for receiving the raw sensory input at that time step)430, high-order interaction operations440, and multiple High-Order Long Short-Term Memories (HOLSTMs)450that generate a respective prediction y (y1through yT).

FIG. 5shows a block/flow diagram of an exemplary DHOCNN/DHOCNN method500, in accordance with an embodiment of the present invention.

At step510, receive all sensory input signals511and an input image512.

At step520, perform high-order convolutions on the sensory input signals511and the input image512to obtain high-order feature maps521.

At step530, perform sub-sampling on the high-order feature maps521to obtain a set of hf.maps531.

At step540, perform high-order convolutions on the set of hf.maps531to obtain another set of hf.maps541.

At step550, perform sub-sampling on the other set of hf.maps541to obtain yet another set of hf.maps551that form a fully connected layer552. The fully connected layer552includes a feature vector.

FIG. 6shows a block diagram of an exemplary basic building block Long Short-Term Memory (LSTM)600to which the present invention can be applied, in accordance with an embodiment of the present invention.

The basic building block LSTM600includes an input gate it601, a forget gate ft602, and an output gate ot603. The basic building block LSTM600further includes multipliers621, and a sigmoid function unit622.

The equations for the 3 gates are as follows:

Correspondingly, the update equations are as follows:

FIG. 7shows a block diagram of an exemplary basic building block Gate Recurrent Unit (GRU)700to which the present invention can be applied, in accordance with an embodiment of the present invention. InFIG. 7, z denotes an update gate vector, r denotes a reset gate vector, h denotes an output vector, {hacek over (h)} denotes candidate activation, IN denotes the input to the GRU700, and OUT denotes the output from the GRU700.

The GRU700can performs comparable or better than a LSTM.

The update equations are as follows:

In LSTM and GRU, the gate functions at time t are all sigmoid functions over a linear combination of current input xtand the memory represented via ht-1. While gating functions are crucial for the network's performance, we further introduce a high order gating function as follows:

where all vectors have dimension n. Here we only consider second order information. Assuming we are using m high order kernels, then we have the following:

where P is a mapping from m kernel output to a vector of dimension n as required.

If we use low rank approximation, i.e., wxh(i)=Σj=1r(vj(i))(uj(i))T, we can rewrite each element in the high order term to be as follows:

As we are learning distributed feature representation, it's reasonable to use vj(i)same uj(i)in order to reduce the number of parameters, i.e., high order kernel weight matrices wxh(i)are all symmetric. Thus we have the following:

For each gating function, the number of parameters we introduced is n*m+r*n*m, in addition to linear part 2*n*n+n.

Alternatively, the high order term can be as follows:

where ⊙ represents for element-wise multiplication, and U,Vεrm×n, Wεn×m. The corresponding total number of parameters for each gating function is n*m+2*n*m in addition to linear 2*n*n+n. The difference between Equation 3 and Equation 1, besides using different U and V, is that Equation 3 only uses one high kernel term whereas Equation 1 uses m high order terms. However, Equation 1 is not a general case for Equation 3.

Also, we can have a multiple layer perceptron for modeling the transition between hidden states.

As shown in Equation 2, the high order term can be represented as a concatenation of a fully connected layer and a dot-product layer. Thus learning could also be done via standard back-propagation.

A description will now be given regarding specific competitive/commercial advantages of the solution achieved by the present invention.

One advantage is that the proposed driving assistance system is universal and can be widely used to build many types of smart vehicles or even autonomous vehicles.

Another advantage is that the proposed driving assistance system has a much lower cost than an autonomous driving system.

Yet another advantage is that the proposed system is much more accurate and robust than previous driving assistance systems.

Still another advantage is that the proposed system can be easily adapted and deployed for traffic surveillance and manufacturing monitoring.