How to Synchronize Time Series Datasets in Python | by M Khorasani | Towards Data Science

One of the most painful things one could encounter in a data related career, is having to deal with unsynchronized time series datasets. Discrepancies can be due to a whole slew of reasons — daylight savings adjustments, inaccurate SCADA signals, and corrupted data to name a few. It is even common to find several discrepancies at various points within the same dataset which requires you to identify and correct each one individually. And while you’re at it, you may inadvertently offset another synchronous part. Luckily with the help of the novel ‘dynamic time warping’ technique we are able to apply quite literally a one size fits all solution to all of our unsynchronized datasets.
DTW for short is a technique that computes the optimal matching between two sequences of data. In other words, you are finding the shortest path from one dataset to another at any given time. The beauty of this method is that it allows you to apply as many corrections as needed to your dataset to ensure that each point is synchronized to the highest degree possible. You can even apply it to datasets of varying lengths. And the applications of DTW are endless, you can use it for both temporal and non-temporal data the likes of financial indicators, stock market indices, computational audio, and more. The only caveat is to ensure that your data has no null or missing values as that can throw a wrench into the works of DTW.
The distance measure used to find the shortest path between corresponding points can be any of the ones provided by Scipy’s distance measure module. While for most instances a Euclidean distance will do the trick, you may however want to experiment with others for good measure.
To implement our own version of DTW, we will be using the fastdtw library in Python. The novelty of this package is that it simplifies the complexity of the warping function, thereby reducing the complexity from O(n2) to O(n) which delivers a tangible difference at runtime.
Please proceed by firing up Anaconda or any Python IDE of your choice and installing fastdtw as shown below:
pip install fastdtw
Subsequently, import all of the required packages:
import numpy as npimport pandas as pdimport streamlit as stimport plotly.express as pxfrom sklearn.metrics import r2_score
Import your dataset and fill in any null values before running the synchronization:
df = pd.read_csv('dataset.csv')df['Power'] = pd.to_numeric(df['Power'],errors='coerce')df['Voltage'] = pd.to_numeric(df['Voltage'],errors='coerce')x = np.array(df['Power'].fillna(0))y = np.array(df['Voltage'].fillna(0))
Then proceed with executing the synchronization:
distance, path = fastdtw(x, y, dist=euclidean)
The result for the synchronized path will be similar to the following :
path = [(0, 0), (0, 1), (0, 2), (1, 3), (2, 4),...]
Each point from the reference dataset will be matched to one or more points in the target dataset, i.e. row 0 of the reference data can be matched with points 0, 1, or 2 of the target data.
Now that you have the warped path, you can proceed with creating a data frame with the synchronized results as follows:
result = []for i in range(0,len(path)):    result.append([df['DateTime'].iloc[path[i][0]],    df['Power'].iloc[path[i][0]],    df['Voltage'].iloc[path[i][1]]])df_sync = pd.DataFrame(data=result,columns=['DateTime','Power','Voltage']).dropna()df_sync = df_sync.drop_duplicates(subset=['DateTime'])df_sync = df_sync.sort_values(by='DateTime')df_sync = df_sync.reset_index(drop=True)df_sync.to_csv('C:/Users/.../synchronized_dataset.csv',index=False)
Finally, you can compute a correlation score using Sklearn’s r2_score module to compare the correlation or level of synchrony before and after synchronization:
correlation = r2_score(df['Power'],df['Voltage'])
To plot and visualize your synchronized data, we will be using Plotly and Streamlit — two of my favorite libraries for visualizing data and rendering it as an app.
You can use the function below to create a time series chart. Please ensure that your timestamps are in the correct format of dd-mm-yyyy hh:mm, or alternatively modify the function to adjust to your data.
To visualize and render your chart, run your script by typing the following commands in Anaconda prompt:
cd C:/Users/.../local_directorystreamlit run synchronization.py
And there you have a visualization of your data before and after synchronization:
Dynamic time warping is probably the single most effective solution for a prompt and convenient synchronization of time series data. While it is not flawless and does indeed have shortcomings such as poor matching of boundary conditions, it is the most comprehensive solution that I have been exposed to yet. You are by no means limited to linear data either and can indeed synchronize non-linear data with varying dimensions.
If you want to learn more about data visualization and Python, then feel free to check out the following (affiliate linked) courses: