Timestamp parsing for time-series data analysis with Pandas and Python
By: Sudeep Gowrishankar
Tags: Big Data, Data, Getting Started, Data Science, Python

Experiments with unique timestamps

1. List of timestamps with a standard format

ISO-8601 is a widely accepted international standard for time-related information exchange. In addition to timestamps that follow the ISO-8601 standard, a few others are also a “standard” format as far as Pandas is concerned. This means that there is some set of timestamp formats that Pandas can parse very efficiently. An exhaustive list of these is not available (as far as I know) but in general, timestamp formats with all parts of the date and ones that start with the year seem to fall under this category.

So now, let us see how these methods perform when given timestamps of a known standard format. The formats of the timestamps are consistent throughout each dataset. We test the performance with datasets of different sizes given to the applicable methods.

Comparing different methods when the timestamps are of a standard format | Image by author

The results show that Pandas.to_datetime significantly outperforms time.strptime in this instance. The pre-built lookup method also marginally outperforms the time.strptime method. However, it is still well short of the performance that Pandas delivers.

2. List of timestamps with a non-standard format

Now, if we run the same tests with datasets that have a non-standard timestamp format (e.g. 13–11–2000 04:50:32), we see some differences.

Comparing different methods when the timestamps are of a non-standard format | Image by author

We notice here that Pandas.to_datetime with a specified format performs the best and a plain time.strptime loop comes in second place. The pre-built lookup method spends too much time building the map and therefore, its performance suffers. Pandas.to_datetime without the infer option also takes a long time because of the repeated format-inference of each timestamp string.

We also see some curious behaviour with the results of Pandas.to_datetime with infer. We see that it performs exceptionally well until it hits a dataset size of close to 20000. And then it performs the same way as Pandas.to_datetime without infer. What is going on here?!

This behaviour happens to be a side-effect of the dataset used in these experiments but it illustrates an important point. The dataset used in these experiments is a list of timestamps that starts with 12:00AM on January 1, 2000 and progresses consistently with an interval of 1 second. The format of the timestamp used is dd-mm-yyyy hh:MM:ss. Therefore, when Pandas tries to infer the first timestamp in the list, there is ambiguity about the format. This is because the timestamp string 01–01–2000 00:01:00 could be either in the format dd-mm-yyyy hh:MM:ss or mm-dd-yyyy hh:MM:ss!

So, when we have a dataset that starts with an ambiguous timestamp but has an unambiguous timestamp towards the end of the list, Pandas may realize that its inference is incorrect when it reaches the end of the list. It would then fall back to the behaviour of inferring the datetime format for each timestamp string individually. This would cause the performance of the operation to be similar to the case when infer_datetime_format = False.

Experiments on datasets with duplicates

The datasets used thus far have had no duplicates in them. However, in the real world, we are often dealing with datasets that have repeated timestamps or multiple datasets from the same time period. In the industrial intelligence domain (in which I currently work), it is not uncommon to process scores of datasets together from the same time-range and therefore, there is a lot of duplicated timestamp strings between all of them.

In the following experiments, we’ll see how our choice of timestamp parsing may change based on how much duplication we have within our dataset.

For the following experiments, all datasets contained 1 million timestamp strings. During a test, different numbers of duplicates were infused into each dataset while keeping the dataset size fixed. The lowest number of duplicates infused was 0 (all unique), and the highest number of duplicates infused was 100 (each timestamp in the dataset had 99 other copies).

Experiment with timestamp strings of a standard format (and consistent throughout dataset)

Pandas performs well with or without duplicates when the timestamp format is standard | Image by author

We see here that Pandas.to_datetime is an easy choice when dealing with a standard format. And as expected, memoization and pre-built lookup mapping improves as the number of duplicates in a dataset increases.

Experiment with timestamp strings of a non-standard format (and consistent throughout dataset)

Memoization and pre-built lookups performing better than Pandas when the timestamp format is non-standard | Image by author

But when the format of the timestamps is not standard and there are some duplicates in the dataset, memoization and pre-built lookup mapping both perform significantly better. In fact, I recently used the pre-built lookup mapping method to parse a large collection of timestamp strings and it saved me over 8 hours!