Customer segmentation in utility retail using time series clustering
By: Alexander Bader
Tags: Utilities, Time Series Analysis, Time Series Forecasting, Python, Clustering

Here is an animated version showing these “monthly average weeks” or what we may call them. There are many interesting features to spot, such as shape changes of the daily curve with season or increasing “peakiness” in the winter months.

Averaged data: weekly and seasonal variability

Clustering — finding brothers and sisters

Now that we got some clean data, we can tackle the crux of today’s problem — finding clusters in the small sample set of meters we use here. A detailed explanation of how clustering algorithms work is somewhat beyond the scope of this article and better explained by more versed experts, so instead we’ll just have a look at how the scikit-learn implementation is used and what its parameters actually do.

Below you see a short function for obtaining clusters from time series kept in a Pandas dataframe as showed earlier. Note that this should work for any kind of time series data, as long as each time series is kept in a separate column and all data in one row corresponds to the same “timestamp”.

Before we can perform any clustering, we need to calculate the so-called “affinity matrix” from the raw data. This matrix essentially contains a similarity score for all possible pairs of time series in the data set. In the example code above, we provided for four different metrics which may be used to determine the affinity between two meters:

corr : affinity = 1 — c, where c is the Pearson correlation coefficient

: where is the Pearson correlation coefficient mape : mean absolute percentage error

: mean absolute percentage error eucl : Euclidean distance

: Euclidean distance cbd : compression-based dissimilarity

Sound familiar? It should, if you’ve had a look at the previous blog post in this series where we discussed how these are calculated and what they mean. You may notice that some of this code in get_clusters references functions defined in this older article.

Distance/affinity matrix of all meters

Note that we use the Pandas function corr() to calculate the affinity matrix - its original intent is to calculate the pairwise correlation coefficient for all columns in a dataframe, but it can be (ab)used to calculate an array of the same shape using different metrics. Interestingly, at least in my version of Pandas the diagonal values are automatically set to 1 which is incorrect for the metrics we use. So we have to manually fill these with the correct value each of the metrics would determine when comparing a time series with itself. On the left you can see what the mape affinity matrix looks like for the small sample set of electricity meters used in this article (lower value = higher similarity).

Now that we know the distances between all time series in our dataset we can start clustering them together. Scikit-learn’s AgglomerativeClustering follows a bottom-up approach starting out with each time series being its own cluster. Clusters are then recursively merged together until either a predefined number of clusters n_clusters is reached or until no more merging is possible with the given parameters.

The way clusters are merged together hereby depends on the linkage and distance_threshold parameters. Two single-point clusters may be merged if their distance is less than distance_threshold - easy enough - but how does the algorithm determine whether two multi-point clusters are closer than distance_threshold ? This is where linkage comes into play:

linkage="complete" : distance between two clusters is equal to the largest distance between any of their members

: distance between two clusters is equal to the largest distance between any of their members linkage="average" : distance between two clusters is equal to the average distance between all of their members

: distance between two clusters is equal to the average distance between all of their members linkage="single" : distance between two clusters is equal to the distance between their two closest members

The remaining parameters of AgglomerativeClustering just let the algorithm know that we will provide a precomputed affinity matrix and that we want it to compute some additional distances even for clusters which aren't merged, which is for the visualization below. We can execute the clustering and retrieve the labels defining all clusters like so

and plot the dendrogram using dendrogram from scipy.cluster.hierarchy . This looks as follows - the distance between clusters is indicated with horizontal lines and can be read off at the y-axis, with each vertical line at the bottom corresponding to one meter. Clusters are indicated with colors. You can see that had we chosen for example metric="mape" with threshold=0.25 , some more clusters would have been merged together.

Dendrogram: clustering result

Sanity check — does it make sense?

So how do we figure out whether these clusters make any sense? With a dataset as small as this, it’s best to just look at it. Here are plots of all clusters, the first image showing each meter’s “average July week” and the second showing the whole range of “monthly average weeks” (we can only really see the seasonality here). I guess we can agree that most clusters show rather similar daily/weekly variability and also similar seasonal variability.

Grouped consumption curves: average July week

Grouped consumption curves: full year

The last figure for today is the same affinity matrix as shown above, but now with the clusters marked with coloured boxes. From what I can tell this looks like a pretty good clustering — I certainly would have done a worse job trying to do this by eye!

So is this useful for us? Certainly. Combining clustering methods like this with metadata for all these electricity meters opens the door to using machine learning or other statistical methods to try and figure out which meter properties can tell us the most about what a meter’s electricity consumption is likely to look like.

Some of our further endeavours in this domain are going to be explained in future blog posts, so stay primed for the next edition of this series! I have a feeling that we may be looking at how to use state-of-the-art machine learning to create electricity consumption forecasts for a cluster of meters…

If you found this article interesting and would like to get into exploring novel data science approaches with utility data yourself, feel free to check out Gorilla’s job openings — at the moment we’re for example looking for a Python data engineer to work on things like these!