Anomaly Detection in Smart Meter Data | by Konstantin Golubev | Towards Data Science

Smart Electricity Meters are being installed in Switzerland, with the goal of 80% of all meters being replaced with the smart ones by 2027 (at least so it is stated here ). One of the challenges at the Energy Hackdays 2020 hackathon was to detect anomalies in the smart meter data. Here I want to describe a naive yet insightful approach to the problem. So insightful it is currently being field-tested by the company that presented the challenge, Clemap AG.
Developed together with Manuel Baez and Xue Wang.
Disclaimer: The approach has probably been known and utilized before but I could not find a reference. And it came as a revelation after two days of a hackathon.
The data is a time-series of readings from a single meter in kWh every 15 minutes in the span of approx. a year and a half.
As claimed in this paper, a lot of information about a household can be retrieved from the smart meter data. In our case, it was not the goal though. However, the fact that the meter is installed in an office rather than in a home was clear right away.
The graph above shows the average energy consumption for different days of the week. For example, the blue line shows the data averaged for all Mondays in the data set. Much less energy is consumed on the weekends (two lower lines), while Tuesdays and Wednesdays are the busy days. You can see the lunch break too!
At some point, I plotted (rather accidentally) the histogram of readings.
The insight it gave was that although the readings lie within the range of approx. 800 possible values somewhere between 2.00kWh and 10.00kWh, the number of unique values they actually take is not that large. In fact, turns out that there are just 233 unique readings in the dataset, and they lie between 2.31kWh and 9.48kWh.
An idea for an algorithm is then straightforward (and now, when I’m writing it, seems to be ridiculously straightforward) and goes as follows. Let’s declare a reading anomalous if hasn’t appeared often in the past. To do that we add a column with the frequency of each reading in the past. The code takes less than 20 lines (and can probably be shortened).
df['freq'] = 0app = cl.defaultdict(float)count = 0for index, row in df.iterrows():    val = row['kWh']    if pd.isna(val):       df.loc[index,'freq']= -1    else:        count += 1        if val in appr:            app[val] += 1        else:            app[val] = 1        df.loc[index,'freq'] = app[val]/count
Below is the chart of all readings, with anomalous marked with red.
In this pic, the brightness of red indicates how rare is the anomalous reading. The lightest red means that the reading’s frequency in the past is below 0.1%, and the brightest — below 0.25%. Absent readings’ frequency is set to -1, and hence they are also marked as anomalous.
Explainable and fast.
Tunable. In the meantime, the only parameter to tune is the threshold. However, one can make the memory limited to a period of time. That is, for example, declare a reading anomalous if hasn’t appeared often in the past year rather than in the whole history of the meter.
Adaptive. If the user behavior changes, the algorithm eventually adapts.
In case, the user behavior changes, the time it would take for the algorithm to adapt depends on how long it’s been running. The longer it has run the longer it will adapt.
It is important to understand what kind of anomalies it does not detect. For example, it hasn’t detected an anomalous day, on which the pattern of the energy consumption was usual but the values were approx. 1.5 higher. This day was found by a different approach: clustering of the days based on the day’s mean and std.
Jupyter Notebook on GitHub. Contains all the details, and other approaches.
The data was provided by Clemap AG and is available for download on Kaggle.
meterOS, a solution developed by a second team working on the same challenge at the hackathon. We joined forces by the end of the event.