Time Series in Python — Part 3: Forecasting taxi trips with LSTMs
By: Benjamin Etienne
Tags: Time Series Analysis, Deep Learning, Forecasting, Lstm, Machine Learning

Stationarity

We can notice a couple of things which you should be familiar with if you’re used to analyzing time-series:

We observe a clear upwards trend Mean and variance increase through time We also observe spikes which may be caused by external events (holidays and weather ?)

The first two bullet points are a sign that our series is clearly not stationary. The latter shows that we might need to incorporate external data to our series. Stationarity could be checked with an Augmented Dickey-Fuller test or a KPSS test. So should we carry out these tests just like we did for ARIMA models ?

— The answer is : not necessarily.

What we want, when we stationarize a series, is to have no change in mean and variance throughout time. This guarantees that if you take an N-points sample to make a forecast , and repeat this with another N-point different samples, then the relationship between your N-point samples and the N+1 point you are trying to predict are the same (ie they are drawn from the same distribution). If the mean and variance are not equal, then you are taking samples from different distributions to make forecasts, and your model will surely fail to generalize.

Unlike ARIMA, RNNs are able to model nonlinear relationships in the data.

RNNs, and particularly LSTM and GRU, are able to capture long-term dependencies (provided you have sufficient amounts of data !). Issues like de-trending and de-seasonalizing are therefore less important, but you should always ask yourself :

Does the data I use for testing follow the same behavior (ie is from the same distribution) as the data I use for training ?

Holidays & Weather

Adding holidays indications is quite straightforward. We use the holidays library in Python to get the holidays dates from 2015 to 2018.

After adding a holiday boolean to our series, we still observe unexplained spikes…

A quick look on the internet shows that a couple of them were actually days when New York was hit by extreme weather events such as blizzards and snow storms.

We plot below the data with holidays marked in red and extreme weather events in green:

In our dataframe, we therefore have a “counts” column, a “is_holiday” column, and a “is_bad_weather” column. However, given we want to make predictions, we need to “anticipate” these dates as we would do for future predictions, we will therefore create two additional columns indicating that the next day is a holiday or that extreme weather is expected the next day :

weather = [datetime.datetime.strptime(date, "%Y-%m-%d") for date in ['2018-01-04', '2018-03-21','2017-03-14','2017-02-09','2016-01-23']] holidays = [date for y in range(2015, 2019) for date, _ in sorted(holidays.US(years=y).items())] df['is_holiday'] = np.where(df.index.isin(holidays), 1, 0)

df['bad_weather'] = np.where(df.index.isin(weather), 1, 0)

df['next_is_holiday'] = df.is_holiday.shift(-1)

df['next_bad_weather'] = df.bad_weather.shift(-1)

Choice of window size & Backtesting

When doing time series forecasting you might hear about backtesting. Backtesting is a procedure used during training which consists in splitting your data into chunks, in an incremental manner. At each iteration, a chunk is used as your training set. You then try to predict 1 or more values ahead of your chunk. Two approaches can be used, expanding and sliding windows:

In our case study, the authors use samples consisting of 28-days sliding windows with step size equal to 1, used to predict the next value (1-step ahead forecast).

Logging and scaling

In the paper, the authors start by taking the log of the data to “ alleviate exponential effects”. They then within each window substract the first value of the window to remove the trend and train the network on fluctuations with regard to the first value of the window(eq(1)). Other approaches can also be thought of, such as substracting the first value and dividing by the first value (eq(2)):

Building the Dataset

Our dataset will be a generator yielding batches of sliding windows (each batch is the previous batch shifted of 1 value in the future). To follow the paper’s instructions, we will also substract, within each batch, the first value to all other values. Each batch is then split between 28-days samples and their 1-day targets.

Defining the model

We will use PyTorch to define our model. A simple reason for that is that we will use dropout during inference and that it is simple to implement in PyTorch. We will start by using a simple LSTM network as defined in the paper: 1 LSTM layer with 128 units, 1 LSTM layer with 32 units, and a fully connected layer with 1 output. Dropout is added after each layer.

— An aparté on Dropout

Dropout can be seen as a way of doing Bayesian inference (though there is still a debate around this). Technically, dropout is the process used in neural networks consisting in randomly dropping units (along with their connections).

The fact of randomly turning neurons on and off is roughly equivalent to performing a sampling of a Bernoulli distribution, and therefore “simulates” the mechanics of a Bayesian Neural Network (where weights are distributions, and not single values). Applying dropout is a bit like if we were “sampling” from the network. And if we repeat this process several times duting inference, we will get different predictions with which we can estimate a distribution and eventually, uncertainty ! To sum up:

Let’s now define the model by adding dropout layers between each LSTM layer (notice how train is set to True so that Dropout is used during training and testing)

Training

We train for 5 epochs, with an Adam optimizer and learning rate set to 0.001, and batch_size of 1.

Results :

Fitted values on the train set

Testing — 1 day forecast horizon

One of the key interests of this paper is the estimation of uncertainty.

Specifically, stochastic dropouts are applied after each hidden layer, and the model output can be approximately viewed as a random sample generated from the posterior predictive distribution. As a result, the model uncertainty can be estimated by the sample variance of the model predictions in a few repetitions.

The idea behind this paper is therefore to run several times the model with random dropout, which will yield different output values. We can then compute the empirical mean and variance of our outputs to get confidence intervals for each time step !

For each step, we predict 100 values. All values are different given we keep the dropout set. This allows us to simulate sampling from our network (in fact, Dropout is closely linked to Bayesian Neural Networks, given that by randomly disconnecting weights, it simulates a probability distribution). We take the average of these 100 values as our predicted mean, and the standard deviation of our 100 values which will be used for our confidence intervals. Assuming that our predictions are drawn from a normal distribution N(μ, σ²) — with a mean μ equal to our empirical mean and a standard deviation σ equal to our empirical standard error —, we can then estimate confidence intervals. In our case, it is given by :

Predicted values and uncertainty intervals

The prediction and the test curves seem to be quite close ! However, we need to find a metric to see how our model performs.

If we look at the empirical coverage of 95% predictive intervals (ie the number of true test values included in the predicted 95% confidence intervals), we obtain a value of 28.51%, far from the values obtained on the test set in the paper…When we take the 99% CI, this value goes up to 44% coverage. This is not a great result, but keep in mind our confidence interval is quite small given we only predict model uncertainty…

Testing — 7 days forecast horizon

Results on a single day forecast are not so good for a prediction model, and still we are not asking a lot to our model as we are asking for a very short prediction horizon. What if we trained our model to predict larger horizon forecasts ?

Conclusion

LSTMs show interesting perspectives for modelling time series due to their ability to capture long time dependencies, but should be used only for large datasets. Never expect your RNN to give good results on a 100-sample dataset ! The balance between the number of parameters in a neural network and the size of the data available is important to avoid overfitting. Nikolay Laptev, one of the authors of the Uber paper, concludes by saying that :

Classical models are best for:

○ Short or unrelated time-series

○ Known state of world Neural Network is best for:

○ A lot of time-series

○ Long time-series

○ Hidden interactions

○ Explanation is not important

Learn more about time series in Part 1 and Part 2 !