I wasted over 6 months on a machine learning project because of this stupid error…
By: Peijin Chen
Tags: Machine Learning, Forecasting, Artificial Intelligence, Data Science, Time Series Forecasting

I wasted over 6 months on a machine learning project because of this stupid error… Peijin Chen Follow Oct 14 · 6 min read

Well, it wasn’t a complete waste, you live and learn. But I spent a lot of time on the project and wanted to write it up as an academic paper — which means you have to consider what added value your paper is offering to the world, and I realized, after seeing my error, that I was not sharing a skillful time series forecasting model as I had hoped, but just, well, a barely better than an educated guess model.

In my case the data was drawn from the UCI Machine Learning Repository, and had to do with air quality and pollution in Beijing, China.

TL;DR — I should have used Mean Absolute Scaled Error (MASE), not R2 (coefficient of determination), or Symmetric Mean Absolute Percentage Error (SMAPE), or anything else for that matter, to judge the skill of the forecaster. In this hourly measured data set, I thought my model’s one-step ahead forecast R2 score of 0.95 was dope, but remember, R2 is just comparing your model’s performance against a naive regressor that just predicts the mean. That, in certain cases, is setting the bar a bit too low.

Formula for MASE, from Wikipedia

On the other hand, the MASE is just the ratio of two mean absolute error (MAE) performances — the one in the numerator are the errors from your model, and in the denominator are the errors from a “naive” regressor, meaning, a model that simply uses the previous step to forecast the next step — which is, in some cases, a lot smarter than just using the global mean.

Here’s some Python code that will tell you what the MAE for a naive forecaster would be:

import numpy as np

print(np.mean(np.abs(np.diff(my_time_series))))

Why this? Because if you predict the last step, then your error is just the difference between successive points — take the absolute value of that series, and then find the mean.

In that dataset, pick one of the sites and then use as your time series the measurements of PM2.5 (particulate matter < 2.5 microns in diameter, measured in micrograms per cubic meter). You will find that, roughly speaking, the MAE of a naive regressor might be something like 10.8 or so.

Sample of PM2.5 data from Beijing for a few days in May 2016.

This is hourly data. So that means on average, the net change in PM2.5 from hour to hour is about 10.8. That means that in order to be a more skillful forecaster than this, you have to improve on that number. And guess what? I couldn’t really do THAT much better. The accumulation and diffusion of PM2.5 isn’t THAT fast as to change drastically in an hour. Sure, I could cook up a model that managed a 10.3 MAE or so, but is that really so much better than 10.8? Not in terms of making any practical difference to anyone’s health, I don’t think.

The naive forecaster presumes you know the value of the observed value one step ahead. That means that if I know the PM2.5 value at 3PM, that I barely need a fancy machine learning model to give an educated guess as to what the error would be 4PM. With all the other features that the dataset features — such as the O3 (ozone) or CO (carbon monoxide) or PM10 measurements , as well as meteorological features such as dew point and temperature and time variables like day of the week, hour of the day and month of the year— it’s not easy to do much better than the naive regressor.

Sure, adding a few more lags might help. But I’ve tried that, and it seems to make things worse, with the MAE of the model coming in at 15 or so.

On a realistic level, what is more useful is a forecast for 6 hours or 24 hours or even a month ahead. Or maybe you just want some model that can generally tell you what the PM2.5 should be, without knowing in-depth pollutant measurements. That is, maybe you want a model that can say “okay, in Beijing, on a hot and humid but clear day in July in Beijing, what would you expect the PM2.5 to be at around 4PM?” How practical is that though? Well, it might be computationally cheaper — you could check out the weather forecast from any old app and use that as a “guess” as to what the weather will be, say, in a week. No fancy sensors needed. And maybe it’s easier to predict temperature and barometric pressure 2 days or a week into the future than it is to predict something like SO2 levels in a week. Again, it all comes down to the predictability of a series, which has a lot to do with its autocorrelation structure and general amount of (sample/permutation) entropy.

Specifically, the features that I used were wind direction (e.g. NW), month of the year (e.g. December), day of the week (e.g. Tuesday), temperature, hour of day (e.g. 16:00), the dew point, wind speed, and atmospheric pressure.

I decided to show here how H20.ai’s autoML creates and scores models. Here’s what I got. You can see that there are Gradient Boosting Models, XGBoost (a variant of GBM), as well as Random Forests and stacked ensemble forecasters as well.

H2o autoML results

Well, it looks like the top MAE of 26.8 isn’t so bad for just using weather variables, but then I took a look at how it did on the test data — which is just the last 25% of the data (about a year’s worth of hourly data) — and this is what I found:

MAE isn’t as good for the unseen data

The MAE on the unseen data is 41.4. I believe that with some tuning or some other learners you can get it down to under 40, but probably not much better than 36 or 37. How bad is that in real life? Well, if your model forecasted the PM2.5 levels to be 236 and it ended up being 200, well, that’s still bad air quality, right? And if you estimated it to be 0.00 but it ended up being 36, most people wouldn’t care either. It’s not like you would be deceiving them into thinking the air was really better or worse than it was.

But still, how does that compare against a “naive” forecaster that just uses monthly and hourly averages to predict the PM2.5 levels? The point is, maybe try to compare your model to 2–3 different levels of “naive” to see just how good it is, or how much performance gain there really is. Think about what someone who doesn’t know ML but knows some basic math might do to make an educated forecast. If your forecasts are just 5% better than theirs, well, is all that ML really worth it for creating a model that was nearly pointless because it wasn’t much better than the forecast of a naive regressor?

I lost sight of that fact. Don’t do what I did.