Predicting Hourly Energy consumption of San Diego (short & long term forecasts)— II  | Towards Data Science

Part 1 of this post had covered the basics of energy (electricity) consumption, how to import, resample, and merge datasets gathered from different sources and EDA. Some inferences were also extracted from San Deigo’s energy consumption data merged with temperature and solar panels installation data.
towardsdatascience.com
In this post, I will explore some time series models like persistent forecast (as baseline), ARIMA and FB prophet; and then also extend my approach to include linear regression, random forests, XGBoost and an ensemble model, to see whether or not these linear and non-linear approaches can model our time series accurately.
Links to Jupyter notebooks on nbviewer:1. Data import and EDA (covered in Part 1)2. Building ML models and Predictions (covered in this post)
If you want a very detailed explanation of time series data processing, modeling and effectively visualizing time series data then please see the 2nd notebook listed above. Having said that, I will try my best to cover the important topics in this post.
Github link for the entire project: Hourly_Energy_Consumption_Prediction.
Below is the data with some added features from the last post:
#Import the 5 years of hourly energy consumption data previously #cleaned, explored, and stored as 'hourly1418_energy_temp_PV.csv' in # Part 1sdge = pd.read_csv('hourly1418_energy_temp_PV.csv', index_col = 'Dates', parse_dates=['Dates', 'Date'])
SDGE: Our target variable, San Diego’s hourly energy consumption in — MWh (also the hourly load or demand)
non_working: 0 if the day is either a weekend or a holiday (binary)
HourlyDryBulbTemperature: The dry bulb temperature measured at San Diego airport in deg F
cum_AC_kW: The cumulative installed capacity of solar panels at customer sites (until the date given in the index column) in kW
Some observations from the previous analysis done in Part 1:
43824 hours (rows) of data (5 years 2014–2018)
SDGE: Min: 1437.08, Mean: 2364.92 and Median: 2298.0, Max: 4867.0
The time series has multiple seasonal patterns — daily, weekly, and yearly.
It has a slightly decreasing trend which is inversely proportional to the solar installation capacity
Energy consumption is highly correlated with temperature.
I have used the time series from 1st Jan 2014 till 31st March 2018 as my training data. And have predicted the hourly energy consumption values for 1st April to 31 Dec 2018 (~15% of the entire data).
The final goal for this project is: “The developed forecasting model/s can be utilized by the electrical utilities to effectively plan their energy generation operations and balance the demand with appropriate supply. An efficient forecast can prove very useful for the utilities in planning their day to day operations, meeting their customers’ energy demand, and avoiding any excess generation of energy.”
Basically, we want to fit an ML model onto our hourly energy consumption time series and use that to predict the future energy consumption of San Diego.
Prediction WindowIn any time series problem, it is very important to define the window of prediction beforehand. Here I have tested the models on a future window of 1-hour, 1-week, and ~8 months.
Error Metrics UsedI have calculated R2 score, MAE, RMSE, and MAPE for each model for the training and test sets. Out of the 4 metrics, MAPE will be chosen to select the best model. MAPE helps us to understand the % error on the absolute value and is indifferent to the absolute magnitude of the time series.
AssumptionsAll ML models have some assumptions and here is one of them that was used for this project:
The future solar panel installations and temperature data are readily available to us and we don’t need to run separate forecasts for them (I wish!). In the real world, we need to use the forecasted values for these independent variables (or at least use the average expected values for the temperature and estimated future solar installations based on the past data). But for simplicity, I have used the actual observed values for these variables in this project.
Without much ado, let’s focus our energy on the time series prediction problem.
Time series is a sequence of observations taken sequentially in time. Any time-series data has the following components: ref link
Level: The baseline value for the series if it were a straight line.
Trend: The optional and often linearly increasing or decreasing behavior of the series over time.
Seasonality: The optional repeating patterns or cycles of behavior over time.
Noise: The optional variability in the observations that cannot be explained by the model.
Another important feature of most time series is that observations close together in time tend to be correlated (serially dependent). This feature is also called as auto-correlation and, as we will see later, forms an important aspect of the conventional time series modeling techniques like AutoRegressive Integrated Moving Average (ARIMA).
Stationarity: A time series is stationary when the mean, variance, and autocorrelation are constant over time, and there are no seasonal or cyclic patterns. It is very important to make a time series stationary before using any linear regression-based models like ARIMA (regression-based on lagged values of Y). A time series is usually made stationary by differencing the series with itself. Two good articles on stationarity — 1, 2.
Due to all the above features, modeling a time series involves a slightly different approach than regular ML problems. Here is a good link explaining the main differences.
Here is another good resource for diving deeper into time series forecasting modeling techniques.
Time series cross-validation: (Ref)Cross-validation for time series is a bit different since one cannot randomly mix values in a fold while still preserving the temporal structure. With randomization, all-time dependencies between observations will be lost. This is why we will have to use a more tricky approach in optimizing the model parameters such as “cross-validation on a rolling basis”. The idea is rather simple — we train our model on a small segment of the time series from the beginning until some t, make predictions for the next t+n steps, and calculate an error. Then, we expand our training sample to t+n values, make predictions from t+n until t+2∗n, and continue moving our test segment of the time series until we hit the last available observation. This can be established using the sklearn.model_selection's TimeSeriesSplit module. We will use this technique to calculate the validation set errors.
Short term hour ahead forecasts
""" Error metrics for hour ahead forecasts when simply repeating last hour's values """ # error_metrics(predicted_values, true_values) is a function that I # built to calculate the errors for a given model's predictions_ = error_metrics(sdge_lin.loc[X_test.index.shift(-1, freq='H'), 'SDGE'], y_test)>> RMSE or Root mean squared error: 122.27>> Variance score: 0.94>> Mean Absolute Error: 99.23>> Mean Absolute Percentage Error: 4.21 %
Long term ~8 months ahead forecasts
""" Error metrics on months ahead forecast when simply repeating last year's values """# the data is hourly and one year = 8760 hours_ = error_metrics(sdge_lin.loc[X_test.index.shift(-8760, freq='H'), 'SDGE'], y_test)>> RMSE or Root mean squared error: 330.74>> Variance score: 0.55>> Mean Absolute Error: 224.89>> Mean Absolute Percentage Error: 9.23 %
In the energy forecasting domain, it is even more important to get the daily max demand right, because that can determine if the plant operator needs to switch ON a peaker (mostly gas) plant or not. So, calculating the error on that too.
"""Resampling both the y_test and predictions at a 24 hours period and using the max as the aggregate function"""_ = error_metrics(sdge_lin.loc[X_test.index.shift(-8760, freq='H'), 'SDGE'].resample('24h').max(), y_test.resample('24h').max())>> RMSE or Root mean squared error: 389.37>> Variance score: 0.22>> Mean Absolute Error: 264.44>> Mean Absolute Percentage Error: 8.60 %
“We are ready to build the future-predicting models now.”
Note: Some of the classical time series models that are generally used for time series forecasting are Autoregressive Integrated Moving Average (ARIMA), Seasonal Autoregressive Integrated Moving-Average with Exogenous Regressors (SARIMAX), Vector Autoregression Moving-Average with Exogenous Regressors (VARMAX), Simple Exponential Smoothing (SES), Holt Winter’s Exponential Smoothing (HWES), and more. You can find more details about each model in this article. We will try our hands only on SARIMAX in this post and then try the traditional linear and non-linear ML models.
So, why should we even consider using traditional ML regression models for time series forecasting?
Because, in some cases, like sales forecasting or energy prediction, the y variable strongly depends on external factors, like the temperature in case of energy consumption. So, it becomes more of a regression problem than time series forecasting, and also supervised ML models can help us find better patterns in such data that rely heavily on exogenous variables.
Traditional time series models like SARIMAX cannot handle multiple seasonalities. They also require a lot of historical data to be able to forecast accurately and are more time consuming when it comes to hyperparameter tuning.
Please refer to [3] for use cases of traditional ML regression models for sales forecasting.
Also, this [4] paper proves that using simple regression-based ML models to predict electricity demand is possible and the errors are comparable to the more complex models.
The energy consumption values can also be expected to depend on it’s previous lagged values because the energy consumption of a region shouldn’t be expected to change much in the next few hours except for any unexpected or unfortunate events. So we will add the lagged values of energy consumption as the X parameters and check if we can predict better using the past values (in addition to the variables that we had already added).
""" Adding max 24 lags; lag1 is the value of the energy consumption in the previous hour, lag2 is the value of energy consumption2 hours before the current value and so on."""# Creating the lag variablesfor i in range(24):    sdge1_lin['lag'+str(i+1)] = sdge1_lin['SDGE'].shift(i+1)""" Since the first 24 values won't have any 24th lag, they will be NaN. So dropping the NaNs """lag_sdge = sdge1_lin.dropna()
plot_predvstrue_reg(elastic_net_lag.predict(X_test_lag), y_test_lag)
Error metrics for model Elastic net with all lags>> RMSE or Root mean squared error: 50.75>> Variance score: 0.99>> Mean Absolute Error: 37.36>> Mean Absolute Percentage Error: 1.58 %
We can see from the graphs and the errors, when compared to the baseline, the elastic net model performs much better on all error metrics.
The RMSE is only 50.75 MW compared to 122 MW with the baseline model.
The MAPE also dropped from 4.21% to 1.58%.
So, this model performs very well but it comes with a limitation — we can use it to predict only the next hour value. That is, the maximum time window it can predict accurately is 1 hour. So, if that is the application case then the elastic net model with previous lag values should be used.
As mentioned earlier, traditional time series models like SARIMAX cannot handle multiple seasonalities.
There are two interesting time series forecasting methods called BATS and TBATS that are capable of modeling time series with multiple seasonalities, check out link. But, they are computationally slow and SARIMAX models with Fourier series handling the multiple seasonalities can perform as good as the TBATS model.
The main diriving point behind this feature transformation is that we cannot just feed in the hours as 0,1,2,....22,23 to the model, because we need to make the model understand that 0th and 23rd hours are in reality as close as the 0th hour is to the 1st hour. Same for the weekdays and years.
Adding Fourier cyclical series for hour, year, and week periods
""" as said above the k terms for each yearly, weekly and daily seasonalities could be chosen by optimizing on the AIC values..but after some research on energy consumption time series k= 5 was chosen for each seasonality """add_fourier_terms(lag_sdge, year_k= 5, week_k=5 , day_k=5)# Visualizing the new variables on week seasonality_ = (1-lag_sdge.loc['01-01-2014':'01-09-2014', [col for col in lag_sdge if col.startswith('week')]]).sum(axis = 1).plot()
From the above plot, we can see how the discrete weekday values have been transformed into a more continuous variable pattern. The same can be visualized for hourly and yearly variables too.
sdgecyc.head(2)
The correlation of the time series observations calculated with values of the same series at previous times is called a serial correlation, or an autocorrelation (ACF). It is used to determine the moving average (MA or q) term of the ARIMA(p,d,q) models.
A partial autocorrelation (PACF) is a summary of the relationship between an observation in a time series with observations at prior time steps with the relationships of intervening observations removed. It is used to determine the autoregression (AR or p) term of the ARIMA(p,d,q) models.
And d in ARIMA(p,d,q) is the number of differencing required to make the time series stationary.
SARIMAX is an extension of ARIMA to handle seasonality terms (S) and exogenous variables (X). The basic architecture of a SARIMAX model is given by SARIMAX(p, d, q)x(P, D, Q, s) where p, d, q are as defined above and the (P, D, Q, s) is the seasonal component of the model for the AR parameters, differences, MA parameters, and periodicity respectively. s is an integer giving the periodicity (number of periods in season), often it is 4 for quarterly data or 12 for monthly data. The default is no seasonal effect.
The SARIMAX model takes many inputs which require some tuning to be able to pick the best model parameters. pmdarimapackage’s auto_arimamodule automates this task and gives us the best model by taking in some input ranges, similar to gridsearchcv.
Error metrics for model SARIMAX(2,1,1)x(1,0,1,24) with Fourier terms 1 week ahead forecast on test setRMSE or Root mean squared error: 150.46Variance score: 0.68Mean Absolute Error: 103.46Mean Absolute Percentage Error: 5.33 %
We see that the first-week forecast is pretty well but even at the end of the first week the forecasting performance decreases and the confidence interval values grow larger beyond the scale of the range of the energy consumption values. Thus, SARIMAX model was not able to capture long term trends but it did well on 1 week ahead forecast.
Errors for 1 hour ahead forecasts weren’t calculated above for SARIMAX model, by using (dynamic=True), because we get excellent results using elastic net regression for 1 hour ahead forecasts using the lag variables and it is much faster to fit than SARIMAX.
Let’s try using FBProphet for our problem. FBProphet provides a decomposition regression model that is extendable and configurable with interpretable parameters. Prophet frames the forecasting problem as a curve-fitting exercise rather than looking explicitly at the time-based dependence of each observation within a time series. Similar to SARIMAX, we can add extra regressor terms like temperature data to the model as well. (Reference link1, link2, link3)
At its core, Prophet is an additive model with the following components:
y(t)=g(t)+s(t)+h(t)+εty(t)=g(t)+s(t)+h(t)+εt
g(t) models trends(t) models seasonality with Fourier seriesh(t) models the effects of holidays or large eventsεt represents an irreducible error term
Using only the ‘SDGE’, ‘HourlyDryBulbTemperature’, ‘cum_AC_kW’, ‘non_working_working’ columns while using Prophet because Prophet, unlike SARIMAX, handles multiple seasonalities well. So, we don’t need to pass in the Fourier terms separately.
FB Prophet can be passed with a holiday feature, but since we have already captured the holidays and weekends in the ‘non_working_working’ column we won’t pass a separate holiday list to Prophet.
Error metrics for FB Prophet w/ auto seasonality 1 week aheadRMSE or Root mean squared error: 203.21Variance score: 0.76Mean Absolute Error: 164.76Mean Absolute Percentage Error: 7.66 %Error metrics for FB Prophet w/ auto seasonality 8 months aheadRMSE or Root mean squared error: 262.74Variance score: 0.72Mean Absolute Error: 201.06Mean Absolute Percentage Error: 8.55 %
So, though the errors of this model are higher than the other models, both on 1-week ahead and months ahead forecasts, it looks like FB Prophet has captured the multiple seasonalities and trend of our data very well.
I will just mention the errors for Random Forests here. Elastic net regression didn’t do very well for long term forecasts.
Error metrics for Tuned Random forest with fourier termsRMSE or Root mean squared error: 196.99Variance score: 0.84Mean Absolute Error: 137.41Mean Absolute Percentage Error: 5.73 %
Random forest performs well with the Fourier terms added. It captures the multiple seasonalities well and also gives a MAPE of only 5.76% compared to the baseline MAPE of 9.23%.
Elastic net regression doesn’t capture the higher energy consumption values as good as the random forest does.
Random forest’s better performance paved the way for trying the tree-based XGBoost model on the data.
XGBoost (Extreme Gradient Boosting) belongs to a family of boosting algorithms and uses the gradient boosting (GBM) framework at its core. It is an optimized distributed gradient boosting library.
XGBoost is well known to provide better solutions than other machine learning algorithms. It is not often used for time series, especially if the base used is trees because it is difficult to catch the trend with trees, but since our data doesn’t have a very significant trend and also since it has multiple seasonalities (modeled using Fourier series) and depends significantly on exogenous variables, we can try XGboost to see how it performs on the time series data of energy consumption.
Error metrics for Tuned XGBoost with Fourier termsRMSE or Root mean squared error: 172.21Variance score: 0.88Mean Absolute Error: 121.19Mean Absolute Percentage Error: 5.08 %
The entire model was tuned and fit in around 2 minutes.
And it seems to have learned the data patterns along with multiple seasonalities very well. The error on ~8 months-ahead forecast is just 5.08% compared to the baseline of 9.23 %. Also, the R2 fit is 88% compared to the baseline of 55%.
WHY should we consider this?
As discussed earlier, to model a time-series data it needs to be stationary. So, the ideal case would be to detrend the data and then feed it into the ML models and then add the trend to the forecasted results. Nonetheless, good results were obtained above without detrending because the energy consumption data from 2014 to 2018 has a very weak trend and the multiple seasonalities were handled well by the Fourier terms.
Alternatively, the overall data trend and also the effect of cum_AC_kW, which is the cumulative PV installation to date, can be modeled using FB Prophet and then merged with the XGBoost’s forecast. Any tree-based regression model like XGBoost cannot easily handle the X variables like cum_AC_kW because it is an ever-increasing variable and the test data will always have higher magnitude values not seen by the model in the training set.
I have extracted the trend and cum_AC_kW impact on energy from the FB Prophet model and subtracted these two components from our main dataframe with all the Fourier terms. Then this detrended energy consumption data was passed onto the XGBoost model and the XGBoost forecast results were added back to the total trend to get the final predictions.
Below is the trend and effect of cum_AC_kw from our previously trained FBProphet model.
We will employ the following architecture for creating our new model:
Error metrics for XGBoost with detrend Prophet, Fourier termsRMSE or Root mean squared error: 212.97Variance score: 0.81Mean Absolute Error: 170.77Mean Absolute Percentage Error: 7.30 %
The combined model is performing worse than the XGBoost alone because FB Prophet overestimates the trend and cum_AC_kW’s impact on energy consumption, especially at the end of the time scale, as can be seen from the plot above for forecast.trend + forecast.cum_AC_kW which shows a sudden dip at the end. This seems to result in an underprediction of energy consumption and impacts the overall results. But still, the model performs very well compared to the baseline and for reasons discussed above, is a better model for time series in terms of stability than XGboost alone.
Different models were tried to forecast the energy demand per hour, measured in MW, of the San Diego Gas and Electric (SDGE) utility region in CA.
The energy consumption is highly dependent on the outside temperature and has strong multiple seasonalities — daily, weekly, and yearly. The increasing PV (photovoltaic) installations in the region (cum_AC_kW) seem to have brought the decreasing trend in energy consumption since more renewable energy at the customer’s facility means less load on the utility. Note that there can be other factors causing this decreasing trend such as the energy storage installations at the customer facilities, increase in electric efficiencies of the household and commercial equipment, people becoming more conscious of their usage (morally or through utility incentives), etc.
The best way to capture the trend, which is a combination of all the above factors and maybe more, is to make the model learn the trend over a long period of time. The seasonality is an important part of predicting the energy consumption of a region, so getting that part right was also very crucial for improving the model’s performance (which was achieved by using Fourier series).
Error terms on hour ahead forecast on the test set
Error terms on months ahead forecast on the test set
The XGBoost model with the Fourier terms has performed very well, predicting for a forecasting window of 8 months ahead. For hourly data with multiple seasonalities that is a pretty impressive result.
For long term forecasts- most of the models have performed better than the baseline persistence model and the best model (XGBoost) gives a MAPE of 5.08% compared to the baseline error of 9.23% on the test set. The RMSE, R2, and MAE values are also considerably lower than the baseline model. For example, the difference in RMSE with the baseline model is almost 160 MW which is pretty significant. To help one understand this better, the average size of a gas-fired combined-cycle power plant is 500MW.
FB Prophet does a very good job of identifying the trend and multiple seasonalities of the data. It can be paired with XGBoost and a more robust long term forecast can be obtained.
Thanks! Please feel free to reach out to me if you have any questions or want to suggest any improvements or corrections.
Save energy, save the planet.