Keep it simple, keep it linear: A linear regression model for time series
By: Praveen Sundaresan Ramesh
Tags: Getting Started, Time Series Analysis, Statistics, Editors Pick, Data Science

Forecasting models

We split the data into a training set (2017–2020) and a testing set (2021). We use three years of data to predict the electricity demand for the first 18 days of 2021. We start with a baseline model where the predicted values are just the values from the previous year (2020).

In all the forecast plots that follows, I am going to be using a solid black line to represent the true demand, a dashed orange line to represent the forecast and a blue shaded region to indicate the 95% prediction intervals.

Model 1: Baseline model

Figure 6: Baseline model forecast for 2021

The predictions for baseline model are quite poor with a root mean square error (RMSE) of 520 MWh. We can see from figure 6 that the performance is pretty good for 1st of January, but gets poorer over time. This is because New year’s day is a public holiday and the predictions are going to be similar. Apart from that, weekdays and weekends do not fall on the same days in the beginning of 2020 and 2021. We know the demand during weekends to be significantly lower than weekdays. Therefore, a slightly better model (Baseline model II) can be achieved by shifting the days to match the weekends.

Model 2: Baseline model II

Figure 7: Baseline model II forecast for 2021

In this model, we can see the predictions to be significantly better than the baseline model, with an RMSE of 348 MWh. This model accounts for the differences in weekday and weekend demands. While this model is better than the baseline model, we can achieve a higher performance with simple linear regression models.

Model 3: Linear regression (8)

As discussed above, we are going to build eight linear models, one for each category. Specifically, Ridge regression is used and the coefficients for each model is presented below.

Figure 8: Linear regression (8) coefficients

The slope for Hot (temp_index) is always positive, while that for NotHot is always negative. This is due to positive and negative nature of the correlation respectively. The intercept is significantly higher during weekdays, while the slope is higher during weekends. This indicates higher electricity consumption during weekdays but a higher dependence on temperature during weekends. One potential explanation could be the higher usage of residential air conditioners during the weekend.

Figure 9: Linear regression (8) model forecast for 2021

The RMSE for this model is about 344 MWh and is only slightly better than baseline model II. Compared to baseline model II, this model better captures the overall trend, but fails to capture the daily peak. This makes sense, because we gave the model only two categories for the hour of the day; (7 am to 10 pm) and (11 pm to 6 am). From the perspective of the model, the peak could be anywhere between 7 am and 10 pm. Capturing the hour of the peak is particularly important from the supplier’s point of view. Therefore, to improve the model, we can split the hour of the day into more than two categories.

In the next model, we use five such categories, namely, Early morning (12 am to 6 am), Morning (6 am to 10 am), Noon (10 am to 3pm), Evening (3 pm to 8 pm) and Night (8 pm to 12 am).

Model 4: Linear regression (20)

Figure 10: Temperature vs Demand for the five categories of hour of the day

The temperature vs demand scatter plot is shown above. Once again, we can split this data based on Hot/NotHot and Weekday/Weekend along with the five categories for the hour of the day. Thus we now have, 2*5*2 = 20 sections and in each of these categories, we are going to be fitting a linear model (Ridge).

Pandas groupby is one of my favourite operations in Python. To run a ridge regression and obtain 20 sets of coefficients is extremely easy with the groupby operation. To do this, we first define a function that takes in the training data, shuffles the rows and returns the regression coefficients. We then use the groupby operation followed by the apply function as illustrated below.

def Ridge_model(data):

"""

Based on the provided data, this function returns the

coefficients of the ridge regression.



input: the training data

output: ridge regression coefficients

"""

data = data.sample(frac=1,random_state=42)

x = data['temperature'].values.reshape(-1,1)

y = data['demand'].values.reshape(-1,1)

model = Ridge().fit(x, y)

return (round(model.intercept_,3),round(model.coef_,3)) data_17_20.groupby(['temp_index','hour_index_1','week_index']).apply(Ridge_model) display(pd.DataFrame(ridge_coef_2,columns=['Regression Coefficients (Intercept,Slope)']))

Figure 11: Linear regression (20) coefficients

Figure 12: Linear regression (20) model forecast for 2021

The performance in terms of RMSE (222 MWh) is better than any of the previous models. It is now able to capture the daily peaks along with the overall trend.

Is there a better model? Of course there is. What if we use 24 categories, one for each hour, instead of working with five categories? Using 24 categories, we end up with 2*24*2 = 96 models.

Model 5: Linear regression (96)

We now model the electricity demands with 96 linear models. Following a similar approach to the previous model, we can obtain 96 sets of ridge regression coefficients. For the sake of brevity, only a sample of the coefficients are shown below. For the complete table feel free to use my code available in my GitHub repo.

Figure 13: Linear regression (96) coefficients

Figure 14: Linear regression (96) model forecast for 2021

The RMSE of this model is about 140 MWh. In the above figure, we can observe the predictions to perfectly capture the hourly patterns and also the daily trends.

Model 6: Polynomial regression of degree 2 (96)

In some of the categories, we saw slight deviations from linear behaviour. In these cases, a higher order polynomial would be a better fit. In this section, I am going to be using a second order polynomial to fit the temperature and demand data.

Figure 15: Distribution of the count in each of the 96 categories

We have a minimum of 100 data points in each of the 96 categories, with a mean of about 365 and we are fitting a simple polynomial regression of degree two in each of the categories. This should eliminate any concerns of overfitting.

Figure 16: Polynomial regression 2 (96) model forecast for 2021

The RMSE for this model is 136 MWh which is only slightly better than the linear (96) model.

Comparison of the models

Figure 17: Comparison of all the models presented in this article

The prediction RMSE was reduced from 519 to 136 MWh. In terms of percentage error, we reduced it from 9.25% to 2.42%. I have also provided the computation time to show that training 96 linear models in this time and era takes only a fraction of a second.

Is polynomial degree 2 (96) the best performance one can achieve? Absolutely not. This is merely a linear model (linear in 1,x,x²) that achieved about 2.4% error. Using more complex techniques like SARIMAX (Seasonal auto-regressive integrated moving average with exogenous inputs), Facebook Prophet, XGBoost, RNN (Recurrent neural networks) and LSTM (Long short term memory) to forecast the demand, could result in an increased performance.

Revisiting prediction intervals

As a practitioner, the actual value of the forecast does not contain much information. They understand the uncertainties involved in any time series forecast and hence it is more meaningful for them to look at the prediction interval rather than the actual predictions. A 95% prediction interval tells us that the predicted value falls in this range roughly about 95% of the times. The code snippet shown here was used to compute the prediction intervals the forecast.

def pred_interval(prediction,test_data,test_predictions,alpha=0.95):

"""

Obtain the prediction interval for each of the prediction

Input: single prediction, entire test data, test set predictions

Output: Prediction intervals and the actual prediction

"""

y_test = test_data['demand']

test_predictions = np.array(test_predictions) # Calculate the sum of squares of the residuals

err = np.sum(np.square((y_test - test_predictions))) # Estimate the standard error

std = np.sqrt((1 / (y_test.shape[0] - 2)) * err)



# Compute the z-score

z = stats.norm.ppf(1 - (1-alpha)/2) # Calculate the interval

interval = z*std

return [prediction-interval,prediction,prediction+interval

Figure 18: Prediction intervals for 2021 forecast for the best model

Based on these prediction intervals, a practitioner can determine the rate of electricity production. To them, this plot sparks joy, not the actual predictions.

Conclusion

In this article, we used linear models to forecast electricity demands in Toronto. There are several models that could deliver a better performance, but the advantage of using linear models are:

We divided the categories based on domain knowledge like temperature, hour of the day, weekday/weekend, etc. It is therefore, very easy to interpret these regression models in each of the categories.

Linear models are quicker to train. As seen earlier, CPU time to train 96 linear models is only about 0.2 seconds. With such a low training time, it is possible to model, deploy, and re-train in real-time.

Linear models do not require as much data to train with, as neural networks or SARIMAX. The latter models also consume a lot of time to tune the hyperparamters.

The linear models presented here account for both yearly and daily seasonality. Yearly seasonality is captured through temperature and the Hot/NotHot categories, while daily seasonality depends on the number of categories chosen in the hour of the day (2, 5 or 24).

It is impressive to see how simple linear regression can provide good results that are interpretable. Hope you guys learned a lot reading this article. All the analysis was done in Python and the relevant codes for can be found in my GitHub repo. Please feel free to share your thoughts here or connect with me on LinkedIn.

Thanks for reading!