Time Series Smoothing for better Forecasting
By: Marco Cerliani
Tags: Machine Learning, Neural Networks, Editors Pick, Data Science, Time Series Forecasting

In time series forecasting, the presence of dirty and messy data can hurt the final predictions. This is true, especially in this domain, because the temporal dependency plays a crucial role when dealing with temporal sequences.

Noise or outliers must be handled with care following ad-hoc solutions. In this situation, the tsmoothie package can help us save a lot of time in preparing time series for our analysis. Tsmoothie is a python library for time series smoothing and outlier detection that can handle multiple series in a vectorized way. It’s useful because it can provide the preprocess steps we needed, like denoising or outlier removal, preserving the temporal pattern present in our raw data.

In this post, we use these trinks to improve a forecasting task. More precisely, we try to forecast the daily power production of solar panels. In the end, we will expect to benefit from our denoising procedure and produce better predictions than the case without preprocessing.

THE DATA

A real dataset for our purpose is available on Kaggle. This data stores the daily power production of solar panels installed on the roof of a private house. The data are recorded since 2011 and present 3 different sources in a time series format:

The daily gas consumption of the house.

The daily power consumption of the house. Where a negative value indicates that solar power exceeds the local power consumption.

The daily value of the power meter on the DC to AC converter. This is the current cumulative solar power. We don’t need the cumulative but instead, we want the absolute daily value, for this reason, we operate a simple differentiation. This is our target to forecast.

As we can notice from the plot of the raw series, a lot of noise is present. This is normal for data registered by sensors. The situation is worse if our data sources suffer the external meteorological conditions or if the sensors are not of good quality and locate in not optimal positions.

Fortunately, we have the knowledge and the instruments to achieve good results for our forecasting task.

TIME SERIES SMOOTHING

The first step in our workflow consists of time series preprocessing. Our strategy is very intuitive and effective. We take the target time series (power production) and smooth it with a fantastic instrument: the Kalman Filter, a must to know for every data scientist.

Generally speaking, the great advantage of using the Kalman Filter, in time series tasks, is the possibility to use a state-space form to represent an unobserved component model. The scope of representing time series models in the state-space form is the availability of a set of general algorithms (including Kalman Filter), for the computation of the Gaussian likelihood, which can be numerically maximized to obtain the maximum likelihood estimation of the model’s parameters. It’s no accident that famous software use this representation to fit models like ARIMA. In our particular case, we use the Kalman Filter and a state-space representation to build an unobserved component model.

All that explained so far can sound tricky but I want to reassure you… Tsmoothie can easily build unobserved component models to operate a custom Kalman smoothing in a very simple and efficient way. At this stage, we can free our imagination and detect which components from level, trend, seasonality, long seasonality contributes to create the time series we are observing. A level and a long seasonality of 365 days sound good for us. We simply add a ‘confidence’ for each component assumption and we have finished.