Extreme Event Forecasting with LSTM Autoencoders | by Marco Cerliani | Towards Data Science

Dealing with extreme event prediction is a frequent nightmare for every Data Scientist. Looking around I found very interesting resources that deal with this problem. Personally, I literally fall in love with the approach released by Uber Researchers. In their papers (two versions are available here and here) they developed an ML solution for daily future prediction of traveler demand. Their methodology stole my attention for its geniality, good explanation, and easy implementation. So my purpose is to reproduce their discovery in pythonic language. I’m very satisfied with this challenge and in the end, I improved my knowledge of regression forecasting.
The most important takeaways from this post can be summarized as:
Develop a stable approach to evaluate and compare Keras models (avoiding at the same time the problem of weights seed generator);
Implement a simple and clever LSTM Autoencoder for new features creation;
Improve forecast prediction performance for time series with easy tricks (see step above);
Deal with the nested dataset, i.e. problems where we have observations that belong to different entities (for example time series of different stores/engines/people and so on)... in this sense, we develop only a high-performance model for all!
But Keep Kalm and let’s proceed step by step.
At Uber accurate prediction for completed trips (particularly during special events) provides a series of important benefits: more efficient driver allocation resulting in a decreased wait time for the riders, budget planning and other related tasks.
In order to reach high accurate predictions of driver demand for ride-sharing, Uber Researchers developed a high-performance model for time series forecasting. They are able to fit (one-shot) a single model with a lot of heterogeneous time series, coming from different locations/cities. This process permits us to extract relevant time patterns. In the end, they were able to forecast demand, generalizing for different locations/cities, outperforming the classical forecasting methods.
For this task Uber made use of an internal dataset of daily trips among different cities, including additional features; i.e. weather information and city-level information. They aimed to forecast the next day’s demand from a fixed window of past observations.
Unfortunately, we don’t have at our disposal this kind of data, so we, as Kaggle Fans, chose the nice Avocado Prices Dataset. This data shows historical avocado prices, of two different species, and sales volume in multiple US markets.
Our choice was due to the need for a nested dataset with temporal dependency: we have time series for each US market, 54 in total, a number that grows to 108 if we consider one time series for each type (conventional and organic). This data structure is highlighted as important by Uber Researchers because it permits our model to discover important invisible relations. Also, the correlation among series brings advantages for our LSTM Autoencoder during the process of feature extraction.
To build our model we utilized the time series of prices at our disposal up to the end of 2017. The first 2 months of 2018 are stored and used as a test set. For our analysis, we will take into consideration also all the provided regressors. The observations are shown with a weakly frequency so our purpose is: given a fixed past window (4 weeks) of features, predict the upcoming weakly price.
Due to the absence of exponential growth and trending behavior, we don’t need to scale our price series.
In order to solve our prediction task, we replicate the novel model architecture, proposed by Uber, which provides a single model for heterogeneous forecasting. As the below figure shows, the model first primes the network by auto feature extraction, training an LSTM Autoencoder, which is critical to capture complex time-series dynamics at scale. Features vectors are then concatenated with the new input and fed to LSTM Forecaster for prediction.
Our forecasting workflow is easy to imagine: we have our initial windows of weekly prices (plus other features) for different markets. We start to train our LSTM Autoencoder on them; next, we remove the encoder and utilize it as a features creator. The second and final step required to train a prediction LSTM model for forecasting. Based on real/existing regressors and the previous artificial generated features, we are able to provide next week’s avocado price prediction.
We easily recreate this logic with Keras.
Our LSTM Autoencoder is composed of a simple LSTM encoder layer, followed by another simple LSTM decoder. You will understand the utility of dropout during the evaluation, at this point they are harmless, trust me!
inputs_ae = Input(shape=(sequence_length, n_features))encoded_ae = LSTM(128, return_sequences=True, dropout=0.5)(inputs_ae, training=True)decoded_ae = LSTM(32, return_sequences=True, dropout=0.5)(encoded_ae, training=True)out_ae = TimeDistributed(Dense(1))(decoded_ae)sequence_autoencoder = Model(inputs_ae, out_ae)
We compute features extraction and concatenate the result with other variables. At this point I made a little deviation from the Uber solution: they suggest manipulating the feature vectors extracted by our encoder aggregating them via an ensemble technique (e.g., averaging). I decided to let them original and free. I make this choice because it permits me to achieve better results in my experiments.
In the end, the prediction model is another simple LSTM based neural network:
inputs = Input(shape=(sequence_length, n_features))lstm = LSTM(128, return_sequences=True, dropout=0.5)(inputs, training=True)lstm = LSTM(32, return_sequences=False, dropout=0.5)(lstm, training=True)dense = Dense(50)(lstm)out = Dense(1)(dense)
Finally, we are almost ready to see some results and make predictions. The last steps involve the creation of a rival model and the consequence definition of a robust forecasting methodology for results comparison.
Personally, the best way to evaluate two different procedures is to replicate them as much as possible, in order to mark attention only at the points of real interest. In this implementation, I want to show evidence of LSTM Autoencoder power as a tool for relevant feature creation for time series forecasting. In this sense, to evaluate the goodness of our methodology, I decided to develop a new model for price forecasting with the same structure as our previous forecasting NN.
The only difference between them is the features they received as input. The first one receives the encoder output plus the external regressors; the second one receives past raw prices plus the external regressors.
Time series forecasting is critical in nature for the extreme variability of the domain of interest. In addition, if you try to build a model based on Neural Network your results are also subject to internal weight initialization. To overcome this drawback a number of approaches exist for uncertainty estimation: from Bayesian to those based on the bootstrap theory.
In their work Uber Researchers combine Bootstrap and Bayesian approaches to produce a simple, robust and tight uncertainty bound with good coverage and provable convergence properties.
This technique is extremely simple and practical... indirectly we have already implemented it! As you can see in the figure below, during the feedforward process, dropout is applied to all layers in both the encoder and the prediction network. As a result, the random dropout in the encoder perturbs the input intelligently in the embedding space, which accounts for potential model misspecification and gets further propagated through the prediction network.
Pythonic speaking we have simply to add trainable dropout layers in our Neural Network and reactivate them during prediction (Keras used to cut dropout during prediction).
For the final evaluation, we must iterate the calling of the prediction function and store the results. I also compute the scoring of the prediction at each interaction (I chose Mean Absolute Error). We must set the number of times we compute evaluation (100 times in our case). With the scores stored, we are able to compute the mean, the standard deviation, and the relative uncertainty of MAE.
We replicate the same procedure for our ‘rival model’ made by only the LSTM prediction network. After averaging scores and computing uncertainty, the final results are better for the LSTM Autoencoder + LSTM Forecaster instead of the single LSTM Forecaster.
In this post, I replicate an end-to-end neural network architecture developed at Uber for special event forecasting. I want to emphasize: the power of LSTM Autoencoder in the role of feature extractor; the scalability of this solution to generalize well avoiding training multiple models for every time series; the ability to provide a stable and profitable method for neural networks evaluation.
I also remark that this kind of solution suits well when you have at your disposal an adequate number of time series that share common behaviors... It’s not important that these are immediately visible, the Autoencoder makes this for us.
CHECK MY GITHUB REPO
Keep in touch: Linkedin
REFERENCES
[1] Deep and Confident Prediction for Time Series at Uber: Lingxue Zhu, Nikolay Laptev
[2] Time-series Extreme Event Forecasting with Neural Networks at Uber: Nikolay Laptev, Jason Yosinski, Li Erran Li, Slawek Smyl