Time series forecasting with AdaBoost, random forests and XGBoost
By: Julia Nikulski
Tags: Machine Learning, Bike Sharing, Xgboost, Time Series Analysis, Data Science

Photo by Jamie Street on Unsplash

Time series forecasting with AdaBoost, random forests and XGBoost

Edit: previous title of this post was “Go highly accurate or go home?”

Scrolling through notebooks on kaggle or reading those data science project stories where people get highly accurate results that perfectly solve the underlying business problem can be quite discouraging for beginners in the data science field who may not reach such results in their own projects. But these shiny notebooks and blog posts actually don’t tell the whole story.

Getting great results actually means a lot of work. Unless you have a perfectly comprehensive data set to begin with, you will need to run multiple cycles of researching, adding and exploring data, engineering and selecting features, transforming them and running and tuning your model. Even after multiple tries, your model may still be performing only moderately. So should you abandon the project and file it under “waste of time”?

Absolutely not!

In the below paragraphs, I want to share my own experience of working on a challenging project which taught me valuable lessons about data science.

TL;DR:

Don’t expect to get great prediction results right away

Time series forecasting is very difficult to do and often times naive models or traditional statistical methods will yield better results than advanced machine learning models

Hyperparameter tuning won’t turn bad or irrelevant data into great predictions; go back to the features you’re using and try to understand which ones to remove, re-engineer, transform or add

Understand why you’re using certain algorithms; contrast them against others and be able to explain their advantages and disadvantages

Use a naive baseline model to compare your results to

Move from hacking to understanding

The project

The aim of the project was to predict bike sharing demand in the metro DC area in the United States using data from Capital Bikeshare. There exists a whole body of research that just deals with the issue of how to know when people want to rent a bike at a certain location in a city to avoid oversupply and shortages of bikes at stations. I wanted to contribute to the problem of demand prediction by focusing on the comparison between tree-based ensemble algorithms (AdaBoost, random forests and XGBoost) and avoiding look-ahead bias that was prevalent in many of the implementations of bike sharing forecasting I found on kaggle and elsewhere. I, therefore, predicted the number of bike sharing customers for the next day across the entire city.

Image by Pablo Valerio from Pixabay

You can find the implementation of this project on my GitHub. I used bike sharing data from Capital Bikeshare, historical weather data from NOAA’s National Climatic Data Center and information on which days are holidays from the DC Department of Human Resources. The bike sharing data ranges from January 1, 2011, until December 31, 2018, and is a continuous time series providing daily bike demand. After merging the data sets, one date feature, nineteen categorical features (holiday and weather type features) and six numeric features (average, minimum, maximum, observed temperature, wind speed and precipitation) as well as the target variable (number of customers at t0) remained.

I will briefly outline parts of my EDA (which was informed by a comprehensive literature review of bike sharing studies), modeling and evaluation approach and then go over the lessons this project taught me and that I would like to pass on.

Exploratory data analysis

Boxplots showing the distribution of the target depending on the season

Plotting the target distribution based on season shows there is a clear connection between the number of people that rent bikes and time of year. During spring, when it’s rather chilly compared to summer and fall, bike sharing demand will be lower.

Heat map showing Pearson’s correlation coefficients between the numeric weather features and the target

The correlation coefficients showed that there were relatively strong correlations between minimum temperature and the target and maximum temperature and the target. I decided to use both of these features. Wind speed and precipitation are only weakly correlated with the target, however, I wanted to keep these features as they may still be useful for the model (especially given that Pearson correlations only show linear relationships). To avoid look-ahead bias, i.e. using data that is only available in the future to predict data in the future, I used lagged weather data. I engineered new features for the mean and the standard deviation of the weather trend of the past 8 days. Running my models with different time lags showed that 8 days yielded the best results. I also encoded the maximum temperature at t-1 to a categorical variable, signifying whether a day was very cold, cold, cool, warm or hot.

Boxplots showing the distribution of the target depending on the weather type

Instead of using the 18 different weather types included in the original dataset, I combined them based on similar distributions of the target variable into 4 categorical variables. I also engineered the mean and standard deviation of the 8-day-lag of these variables.

Boxplot showing the distribution of the target depending on the weekday

Weekdays actually don’t appear to have much of an influence on the number of customers. However, tree-based ensemble models are supposed to be able to handle noisy data well, so I kept those features and ran my models including the features and again after removing them to compare results.

Partial autocorrelation plot for the target

Because I’m using time series data, the target value can be engineered into features as well and actually should be! From the partial autocorrelation plot we can see that yesterday’s number of customers is very strongly correlated with today’s. I added number of customers at t-1 as a feature and added the mean and standard deviation of the past 8 days as well.

Time series of total customer target variable

When doing time series forecasting, statistics tells us that we need to ensure that our data is stationary to get accurate results, which means that any seasonal or non-seasonal trends need to be removed. I used the ADF and KPSS tests to find out that the target was not stationary. I compared the performance of the models predicting both the non-stationary and stationary target variables to see the effect of non-stationarity on prediction outcomes.

Final features used in the model and transformations applied to them

I used 25 features in total to predict the demand.

Modeling

Tree-based ensemble models have multiple advantages compared to other machine learning algorithms: they are easy to understanding; their results are better than those of the underlying weak learners; they don’t have a normality distribution assumption and can handle mixed data types; they are robust against overfitting, outliers, noise, multi-collinearity, and input scaling (for more details check this paper and this one); and they are computationally relatively inexpensive compared to ML algorithms such as Support Vector Machines or neural networks. Therefore, I used AdaBoost, random forests and XGBoost to predict bike sharing demand and determine which of these three models is best suited for this type of data set. As a benchmark for the performance I used the last value approach which predicts that tomorrow’s demand (t0) will be the same as today’s (t-1).

For more details on the implementation and the tuned hyperparameters, you can refer to the code in my Jupyter Notebook.

Evaluation

For my evaluation, I used three different metrics based on the approach Zeng et al. (2016) used: mean average error (MAE), root mean squared error (RMSE) and root mean squared log error (RMSLE). The MAE shows the average deviation of the prediction from the true value, while the RMSE is squaring the differences and punishing larger deviations more heavily than smaller ones. The RMSLE is punishing under-predictions more heavily than over-predictions. This makes it an interesting metrics for bike sharing demand as you would usually want to have more rather than less bikes at a sharing station for people to rent.

Summarized results of bike sharing demand forecasting

Looking at these results and comparing the advanced ML algorithms with the benchmark and the range of target values, all three models are performing only moderately well. The naive baseline model is even outperforming all tuned models on the MAE. Moreover, these models on average deviate by 1,500 people per day in their prediction. Considering that throughout the 8-year period that was analyzed there are only 7,726 average daily customers, those prediction performances aren’t great.

So how can we explain this performance and what can we learn from it?

Time series forecasting and look-ahead bias: Forecasting time series is difficult because there are many factors that can influence tomorrow’s value. Moreover, you need more statistical background knowledge to avoid pitfalls. And most importantly, to have a meaningful model you need to avoid look-ahead bias. In my model I did account for look-ahead bias by using lagged weather data. However, because weather fluctuates, trend data may not be that useful. It might be better to use weather report/forecast data to predict demand. But because this information is difficult to obtain, especially for a longer period of time in the past, I decided to use historical weather data.

Aside from this area of look-ahead bias, we also need to ensure that we don’t introduce randomness. The time series needs to be continuous and in order. We cannot be randomly splitting our data into train and test sets. Rather, we should use TimeSeriesSplit to avoid that we predict the target with information that we would not have been able to know at the time of the forecast. Furthermore, when using bagging or bootstrapping algorithms, such as random forests and XGBoost, ensure that you actually don’t bootstrap because this would again introduce randomness and look-ahead bias into our prediction (XGBoost by default does not bootstrap, while random forests does). Look-ahead bias is the reason why other implementations of similar projects that you see on kaggle actually yield better results: during training and testing they perform much better, while in a real-life application they would actually fail.

Statistics and how to know when they are important: Even if you don’t use traditional statistical methods to do time series forecasting, you still need to have a good understanding of the underlying statistical assumptions that come with a time series. Without knowing about autocorrelation, would you have looked at the partial autocorrelation of the target value to determine that this would be a good feature to engineer? And without being aware of stationarity, would you have transformed your numeric features and target variable to account for seasonal and non-seasonal trends? Because I hadn’t worked with time series forecasting before and wasn’t sure of the effect of stationarity, I simply ran all models predicting stationary and non-stationary targets and compared them. And I found that only AdaBoost was performing slightly better with stationary data. So it seems that ML algorithms are not as prone to inaccuracies when trend is still included in the target. But in the future, I will always try both options if I’m not sure what the outcome may actually be.

Simple baseline models: The last value approach did not perform much worse compared to the 3 ML algorithms. How is that possible? As the above partial autocorrelation plot showed, today’s and yesterday’s target values are highly correlated. So if I’m using a model that simply takes the value from yesterday to predict today’s target, then it will perform rather well. This shows that even if you have a fancy multivariate ML model that takes into account all kinds of features you may not actually be able to outperform very simple baseline models or more traditional statistical models, such as ARIMA. Especially in time series forecasting more complex is not always better.

Trial and error: Not only hyperparameter tuning is important to increase performance. If you have bad or irrelevant features, tuning won’t save your model. Feature selection and engineering are vital to improve an underfitting model. Looking at the performance of AdaBoost when comparing the inclusion of the weekday feature and the removal, it shows the importance of good feature selection and the apparent weakness against noisy data of this algorithm. AdaBoost is the only algorithm that improves when the weekday features are removed. Feature selection and engineering is a lengthy process of trial and error. But it is important to test your model with different features to determine which ones actually help and which ones hurt your predictions. The research part of identifying additional helpful datasets and features is at least as important as tuning your hyperparameters.

Understanding ML algorithms: Don’t just implement XGBoost because you saw a ton of people do it on kaggle. Think about the advantages and disadvantages of different algorithms, read through the actual scientific papers that introduce these algorithms and their pseudo code and compare them against each other. Through this project and by actually reading the papers by Freund & Schapire (1996), Breiman (2001) and Chen & Guestrin (2016) that introduced AdaBoost, random forests and XGBoost, respectively, I actually understood how to contrast these tree-based ensemble algorithms. Only then you can get a better understanding of why certain algorithms may be performing better than others. In my case, XGBoost is most likely performing better because of the additional regularization hyperparameter that avoids overfitting. While random forests decreases overfitting by using bagging/bootstrapping the training sample, this does not help with time series data where we want to avoid randomness.

Understanding performance metrics: Using MAE, RSME and RMSLE in combination can give you an idea where your models may have weaknesses. Because the MAE of the last value approach is lower but the RMSE and RMSLE are higher than for any of the tuned ML models, this indicates that the ML models are more often moderately over-predicting the target, while the last value approach is under-predicting more heavily. Especially random forests with the lowest RMSLE is over-predicting more often than under-predicting. This can arguably be better for a demand prediction model when more bikes at stations available for rent are better than less bikes.

Conclusion

XGBoost is the best performing model out of the three tree-based ensemble algorithms and is more robust against overfitting and noise. It also allows us to disregard stationarity in this particular data set. However, the results are still not great. To get better results for the forecasting of bike sharing demand, more time needs to be spent on feature selection and engineering. Because the model is underfitting most likely different and/or more features will need to be included. But more traditional statistical methods, such as ARIMA, should also not be dismissed as they may yield better results. To make it more useful for real-world applications, a longer forecast horizon and a station-specific instead of city-wide prediction would be valuable.

Some of these insights you were probably already familiar with, and during my Data Science Nanodegree at Udacity, I learnt about the importance of understanding performance metrics, ML algorithms and feature selection. But this particular project reminded me of how essential it is for your growth as a data scientist to move past just hacking and get to understanding. Moreover, I hadn’t worked with time series forecasting before and learnt a lot about potential pitfalls when dealing with this type of data.

Not every project you touch will immediately lead to high prediction performances. More often than not it takes a lot of time, research and trial and error. But even moderately performing models can tell us a lot about the underlying data and business problem we are trying to tackle and guide us towards how to improve our predictions.

Note: The content of this blog post is based on an unpublished research paper I wrote.