Resampling and interpolation of bigdata timeseries in python, pyspark
By: David De Wilde
Tags: Interpolation, Spark, Big Data, Time Series Analysis, Pyspark

Introduction

Interpolation is a commonly applied transformation when it comes to time series analysis. For example, interpolating data from multiple sensors to a common time axis can greatly simplify further analysis. In pandas the ‘resample’ command provides this functionality for small to medium-sized datasets. (Py)Spark on the other hand doesn’t provide a built-in function to resample time series data using linear interpolation. On the internet, only a limited amount of approaches are suggested and those make use of pandasUDF or pythonUDF calls in spark, which has a severe performance impact. In this article we will describe and demonstrate a native PySpark implementation of linear interpolation for time series and compare the performance and scalability with 2 UDF based solutions.

Commonly used approaches

Pandas udf approach

A pandas UDF approach is suggested by Azarudeen Shahul in this blogpost. This method defines an pandasUDF that calls the pandas resample method. The main advantage is that the linear interpolation method doesn’t need to be implemented. However, for big data analysis the pandasUDF causes a performance bottleneck.

Python udf approach

Jessica Walkenhorst on the other hand proposes a python UDF based approach in this blogpost. This methodology is similar to the one proposed in the current blogpost (see below), but it uses pythonUDF’s instead of native spark functions for the creation of the time axis and for the linear interpolation, limiting the scalability.

Tempo

Databricks is currently developing a timeseries library on top of Apache Spark that also provides downsampling and interpolation: tempo. Though currently linear interpolation is not yet supported. Nonetheless, it might be added in the future and is worth keeping an eye on.

Koalas

Koalas currently doesn’t implement the resample method. Even for forward filling the documentation states that this should be avoided against large datasets.

Interpolation in native PySpark

The proposed PySpark interpolation method can be found in the code block below. As input it takes a dataframe with schema: “SensorId: bigint, Timestamp: timestamp, Value: double”. This dataframe contains the sensor values for different sensors at different timestamps. Next to that, the parameter resample_interval (=size of the resample interval in seconds) needs to be defined beforehand. The main steps in the interpolation pipeline are:

(line 4): Using a window function to add the previous timestamp and the previous value to the current row. As such we have all information from both start and end of an interval on a single row. (line 11): Round up the interval start timestamp and round down the interval end timestamp to the nearest interval border. Especially for the interval start time this is crucial for a correct working of the sequence function in step-4. (line 14): Filter out intervals that are not relevant. For example if we want to resample to 1 hour interval, in that case the interval: [00:05:00–00:08:25] is not relevant and should not be taken into account further. (line 16): Per interval create the timestamps at the correct interval size between the start and the end time. (line 18): The sequence command uses inclusive boundaries for both start and stop (as opposed to for example the python range function). This can possibly lead to duplicate values. This extra filter excludes the stop of the interval, effectively making the sequence function behave as: [PreviousTimestamp, NextTimestamp[ (line 20): Do the actual interpolation based on: PreviousTimestamp, NextTimestamp, PreviousValue, NextValue and Timestamp

The transformations described above to perform linear interpolation in PySpark only introduce 1 shuffle in our test scenario. This shuffle is introduced because of the window functions to get the previous timestamp and value. A full working example including creating a ‘virtual dataset’ can be found here: databricks notebook

Performance comparison

For the performance comparison 3 different algorithms were compared:

interpolate_pyspark: suggested native PySpark interpolation method

interpolate_pandasudf: approach suggested by Azarudeen Shahul

interpolate_pythonudf: approach suggested by Jessica Walkenhorst

For the first testcase a virtual dataset of 10 sensors with 1000 timepoints randomly distributed over a 1 week time interval was created. The data has been linearly resampled to intervals of 1s, 10s, 1min, 10min and 1hour. Afterwards a count (action) has been performed on the resulting dataframe to trigger the execution of the defined transformations. The tests were performed on a databricks cluster with 2 DS3_v2 azure worker nodes. The notebook can be found here: databricks notebook

Time interpolation for 10 sensors with 1000 readings distributed over 1 week. Interval size (1s, 10s, 1min, 10min, 1hour — logarithmic scale) versus duration of the interpolation.

This shows that the performance for all 3 methods is similar for bigger interval sizes, but for smaller interval sizes and thus larger datasets the udf based methods don’t scale well.

For 100 sensors instead of 10 sensor similar results were obtained:

Time interpolation for 100 sensors with 1000 readings distributed over 1 week. Interval size (1s, 10s, 1min, 10min, 1hour — logarithmic scale) versus duration of the interpolation.

For 100.000 sensors only the duration of the PySpark approach was calculated because the other methods took to long to calculate. With this large amount of sensors the resampling to 1s intervals for 1 week of data still completed quickly.

Time interpolation for 100.000 sensors with 1000 readings distributed over 1 week. Interval size (1s, 10s, 1min, 10min, 1hour — logarithmic scale) versus duration of the interpolation.

Conclusion

There is a gap in available libraries to support resampling and linear interpolation of timeseries data in (Py)Spark. Previous suggested pandas or python UDF based approaches to perform linear interpolation for timeseries have specific advantages (reuse of existing pandas libraries) for small to medium datasets. However, scaling those approaches to big data sets is challenging. To solve this we have proposed a (Py)Spark native approach with following characteristics:

limited complexity and amount of code

performant scaling to big data sets

Awaiting linear interpolation support of new libraries such as Tempo or Koalas, it provides a good alternative to do linear interpolation on big data sets using Spark.

All code can be found on my github

References