(Part 1)Time Series: Predicting Hourly Energy consumption of San Diego (short & long term forecasts) | Towards Data Science

Part 1 of this post will deal with the basics of energy (electricity) consumption, how to import, resample, and merge datasets gathered from different sources, EDA, and draw some basic inferences from San Diego’s energy consumption data. If you are only interested in the modeling part then please feel free to jump directly to Part 2 of this post.
towardsdatascience.com
In these posts (Part 1 and Part 2) we will cover both short term (an hour ahead and week ahead) and long term (months ahead) forecasting.
Links to Jupyter notebooks on nbviewer: 1. Data import and EDA (covered in this post)2. Building ML models and Predictions (covered in Part 2 post)
Github link for the entire project: Hourly_Energy_Consumption_Prediction.
Electrical utilities need to diligently plan ahead of time the allocation of generating units in their power plants to match their regional energy demand because if the demand is higher than the generation, it can cause several blackouts resulting in a huge loss to the economy; on the other hand, if the generation is higher than the demand the extra electricity will be wasted and it can also create an unnecessary load on the transmission lines.
So, it is very important for the utilities to have a forecast of the energy consumption to be able to allocate appropriate resources to meet their demand. A year, month, or day-ahead forecast can help the utilities plan for a larger time scale, but for smoother daily operations an hourly (or even better) forecast can prove very useful. For example, if the plant operators get a high energy forecast for the next hour, they can ramp up the energy supply by switching on more power plants.
In this project we will analyze past 5 years of hourly energy consumption data of SDGE utility to find trends in energy consumption around hour of the day, day of the week, season of the year, etc., and also to check if factors like outside temperature and solar installations in the region affect the energy consumption.
The developed prediction model can be utilized by the electrical utilities to effectively plan their energy generation operations and balance the demand with appropriate supply.
Instantaneous Load also called Demand, is measured in Watts (W, kW, MW, GW, etc.).
Energy consumed or produced over a certain period of time is measured in Watt-hours (Wh, kWh, MWh, GWh, etc.)
Since the data we will be using is a 1-hour interval data, both demand and energy during that interval will be the same. So, we will be using the energy and demand terms interchangeably in this post. Please refer to this link if you want to dive deeper. Note: The peak or maximum load decides the required capacity of a grid. For example, say the average load of a city is 10MW over a period of 365x24 hours, but the maximum peak load observed or that can be potentially observed at any moment within those 365x24 hours is say 20MW, then the built-in capacity of the city’s power plant infrastructure (including all power sources) should be at least 20MW. So, on average, the built-in power plant capacity of any region is much greater than the average expected demand of the region.
Enough with the terminology, let’s put the rest of ‘our energy’ into building some time series models.
Importing data:
Similarly, we can check the 2014–2017 data as well. Note: At the time of the project, data was available only till July 2019, but if you check now (May 2020), data is available on the website for all the months of 2019. The analysis shown in this post considers only the full years’ data from 2014–2018 but the remaining months of 2019 can be added easily using the same code.
The above code will give us the past 5 years of hourly energy consumption values for 4 of the CAISO utilities, namely PGE, SCE, SDGE, and VEA. It will also give us the CAISO total. Later, we will extract only the SDGE data for this project.
Below are some of the operations that were carried out on the above hourly1418 dataframe:
The ‘Dates’ column was already in datetime format, so we can directly use Datetime operations on the column.
Converting the energy consumption values to numeric; while converting to numeric, it was observed that some of the values in the columns were strings, so passing the errors=’coerce’ argument to the pd.to_numeric method. Also, after the conversion, the column was checked for null values.
Some values in the ‘Dates’ column data were in the format’201x-xx-xx yy:59:59.992' so they were converted to ‘201x-xx-xx yy:00:00’ by using dt.floor; this will made the Datetime rows in ‘Dates’ column uniform.
It was found that the dataframe only had 3 missing values, so by using errors = ‘coerce’ we didn’t lose much of our data, which justifies the use of ‘coerce’ method here. I just want to warn that errors = ‘coerce’ is not the best method to use every time and you have to check your data before using it, or else it can result in a lot of null values.
The original energy time series for the San Diego region can be observed below.
The above hourly1418 dataframe was saved as ‘hourly1418CA.csv’ file for future use. We will continue using only the SDGE energy consumption values.
hSDGE1418 = pd.read_csv('hourly1418CA.csv',usecols=['Dates','SDGE'], parse_dates=['Dates'])
43824 hours of data (Energy consumption in MWH = 106 Wh)
Mean: 2364.92 and Median: 2298.0
Min: 1437.08
Max: 4867.0
We can see how energy consumption has higher peaks during summer and has strong multi-seasonality patterns. What that means is, the energy consumption has daily, weekly as well as seasonal patterns. We will later also analyze the trend of the data.
Other date-time parameters were added to the dataframe as shown below:
non-working: the day is either a federal holiday or a weekendseason: November-May: winter and June-October: summer (as per SDGE link)
To see the impact of weather (specifically the temperature) on energy consumption, we’ll import the hourly temperature data from gov.noaa.ncdc. There are two or more weather stations in San Diego at which the weather data has been recorded. Since weather data at airports is generally more accurate and not biased by nearby buildings and industries we’ll use the San Diego International airport data for this project. (from this link, select San Diego County > San Diego International Airport and Add to cart, then you can download the required data).
The downloaded weather data has many columns but we will focus only on the ‘HourlyDryBulbTemperature’ which is the outside temperature in deg F.
The weather data is captured every 7 minutes and the energy consumption data is hourly data, so I had to resample the weather data in hourly intervals by calculating the average temperature for each hour before merging it with energy data.
hourly_weather = weatherdf_air.set_index('DATE').resample('H').mean()wehSDGE1418 = pd.merge(hSDGE1418, hourly_weather, how = 'outer', left_on = 'Dates', right_on= 'DATE')
It was observed that the energy consumption of San Diego has overall decreased over the years, and the decrease was more prominent over the daylight hours. So, we can test the theory that maybe solar capacity installed in the San Diego region has increased over the years, and thus more and more customers are using less energy when the sun is out, causing a total reduction in energy consumption. We can test this theory by importing the solar installations data in SDGE territory.
The solar installations data was imported from California DistributedGeneration Statistics site (direct data link). This dataset contains a lot of parameters but we will focus on,- ‘approval date’: which represents the date when the system was connected to the grid became active and,- ‘System Size AC’: which represents the total kW power in AC of the solar panels installed at a site.
To get more details about the data, you can check this workbook in my repo.
For our modeling purpose, we are not interested in how much PV capacity was installed in one day but rather we are interested in knowing how much total capacity of PV system is currently under operation in the region on any particular day. So, using a cumulative of the system size we want to see how the total solar capacity in the region increased overall.
SDGEPVM['cum_AC_kW'] = SDGEPVM['AC_kW'].cumsum()##Truncating the PV installation data to the min and max dates' ##limits of the 'weather+energy' data - 'wehSDGE1418' and storing it ##as SDGEPVTSDGEPVT = SDGEPVM.loc[wehSDGE1418.Date.min():wehSDGE1418.Date.max(), ['AC_kW','cum_AC_kW']].reset_index()
Basically, the idea is, if the customers have more PV at their end, they will rely less on the grid for the power supply (at least when the sun is out) thus decreasing the overall energy demand. And so we are considering PV installed at the customer sites (like residential housings, commercial buildings, etc.).
Let’s merge the PV dataset to our previously merged weather+energy ‘wehSDGE1418’ dataframe.
df = pd.merge(wehSDGE1418, SDGEPVT, left_on = 'Date', right_on = 'date_approved', how = 'left')# dropping duplicate date columnsdf.drop(['DATE', 'date_approved'], axis=1, inplace=True)
Since not necessarily every single day in 5 years (2014–18) solar panels were installed, the SDGEPVT dataframe didn’t have all the rows matching the energy dataframe wehSDGE1418 which resulted in missing values in the AC_kW and cum_AC_kW columns.
We can just put a 0 in place of NaN to indicate 0 installations for that day for the ‘AC_kW’ column.
And since the cumulative installed solar system size should remain the same until the next non-NaN value is encountered, using forward fill method to fill the ‘cum_AC_kW’ column.
df['cum_AC_kW'] = df['cum_AC_kW'].fillna(method='ffill')
Forward fill worked on all the rows except the first few rows because the data for SDGEPVT didn’t start at 2014–01–01 00:00:00 as our energy and weather data, so we’ll have to use a different method to fill in the first few missing values. So, basically the initial missing values should be equal to the first non-NaN (‘cum_AC_kW’ minus the ‘AC_kW’).
Plotting some graphs to get insights from the data. Some of the questions that can be answered are:
Checking how energy consumption varies across any particular day averaged over the entire period. Typical average daily load profile would be a curve peaking in the evening time because most people return home from work during the evening and turn on their lights, TV, AC, etc..
Plotting monthly average load profiles. San Diego has hot summers but the winters are not that cold so we can expect the load to be higher in summers because of the cooling load on the commercial and residential buildings.
Checking hourly vs weekday energy consumption map to get an overall idea of the consumption pattern during a typical week.
What is the effect of temperature on energy consumption?
Has the increasing number of solar installations behind the customer meter resulted in a decrease in total energy consumption of the entire SDGE territory?
Plotting average hourly load profile over the entire period 2014–18
Plotting average hourly load profile over the entire period 2014–18
2. Hourly vs weekday energy consumption to see how the consumption varies across any particular week
From SDGE’s website: “If customers can shift some of their energy use to lower-cost time periods outside 4 p.m. to 9 p.m., they can lower their electricity bills and make better use of cleaner, renewable energy sources, like wind and solar, when they are more available.”
3. Visualizing the distribution of energy values for different years
This peak shift to the left can be assumed to be caused by increasing renewable energy installations at customer sites (called behind-the-meter) and increasing participation of customers in demand response programs resulting in lower peak demand on the grid.
3.1. Plotting the same distribution but for 12 pm only
One of the reasons for this shift during daylight hours can be the addition of more renewables like solar behind the customer meters. We will see later that the solar installations in the SDGE have increased considerably over the last 5 years. There can be many factors causing this decrease including increased participation in energy efficiency and demand response programs, etc. but we will focus only on solar installations for this project.
4. Exploring energy and weather data together
As suspected before, we can see that energy consumption and temperature do kind of follow each other and seem to have some level of correlation between them.
More importantly, we can see that the highest energy consumption values occur at the highest temperatures. This can be the result of higher air conditioning loads at higher temperatures, required more in summer. In San Diego, the winters are not harsh but summers can get really hot, so people don’t use heaters much in the winter as they use the ACs in the summer.
for season in SDGE_t_pv['season'].unique():    corrcoef, pvalue = scipy.stats.pearsonr(SDGE_t_pv[SDGE_t_pv['season'] == season]['SDGE'], \                                            SDGE_t_pv[SDGE_t_pv['season'] == season]['HourlyDryBulbTemperature'])    print('pearson correlation coefficient and pvalue for '+season, corrcoef, pvalue)
pearson correlation coefficient and pvalue for winter: 0.32pearson correlation coefficient and pvalue for summer: 0.65
As observed in section 3.1 above, the energy consumption for daylight hours seems to have decreased from 2014 to 2018. Now, this can be due to stricter energy efficiency measures being mandated, or increase in incentives for different categories of ratepayers (residential, commercial, agriculture, etc.) to install self-generation resources like solar, wind, etc. and/or energy storage on their premises to avoid the higher energy demand. It was observed that the energy consumption decrease over the years is more dominant over the daylight hours, so we can test the theory that maybe solar capacity installed in the utility region has increased over the years, which it has as we will see below.
>> Energy consumption vs PV installed capacity: pearson correlation coefficient and pvalue for winter -0.515, 0.0>> Energy consumption vs PV installed capacity: pearson correlation coefficient and pvalue for summer -0.338, 0.0
The correlation seems to be significant for both the seasons and is much stronger for winter months. It’s a bit counterintuitive, but colder temperatures are actually better for solar panels than warm temperatures are. This relationship holds true across most types of solar panels — and has nothing to do with the installation process itself. Studies conducted in laboratories all over the world clearly show that increases in temperature have an inverse effect on the total solar output of the panels tested. In other words, the hotter it gets, the less efficient your solar PV system becomes.
Thus, all things being equal, your monthly utility bill savings will be less during June, July, August than they will be in December, February, and January. Ref: https://www.sunlineenergy.com/solar-pv-power-output-san-diegos-winter-months/
Also, “San Diego, another of the top-ranking US cities for sunshine, gets more sun in winter than in summer”. (Reference link)
We have seen the effects of temperature and PV installation capacity on the energy demand of San Diego city. The energy demand is significantly higher in the summer months than in winter, due to higher temperatures. And the installed solar panels capacity also seems to have a slight (reducing) effect on energy consumption over the years. And, the energy consumption has multiple seasonal patterns.
In Part 2 of this post, we will start from building simple short term forecasting models using traditional forecasting algorithms like SARIMAX and also linear regression. We will then also try our hands at long term forecasting using non-linear approaches like Random forest, XGBoost as well as an ensemble model (XGBoost + FB Prophet).
Thanks for reading. See you in Part 2.
Disclaimer: I chose San Diego out of all the utilities because, i. it serves a smaller region than other utilities so we can use temperature data from a single source without expecting much variation (actually San Diego has a wide variation in weather patterns, but we will neglect that for this project) and ii. I am an alumnus of UCSD.