Speeding Through Dates with Pandas | by Yaakov Bressler | Towards Data Science

Methods for increasing the speed of DateTime conversions by > 480X.
This article introduces datetime objects, how to manipulate them, and how to convert from string format to datetime. Several datetime conversion methods are then introduced and compared for optimizing the speed of conversion. A memoized approach is demonstrated to be over 480X faster in several scenarios — that’s very fast!
First, an introduction to dates, times, and datetimes in Python.
In Python, dates and time are stored in a single object aptly named a datetime object. These objects are accessed through Python’s datetime module. Thereby, datetime.datetime becomes perfectly sensible (if not a bit imprudent).
# Access today's date with the following:datetime.datetime.now()
Pro tip: A datetime object can contain the class of date or time or datetime.
datetime objects can be compared through simple computation.
The demonstrated computation is possible since all datetime objects contain date and time instances (even though we didn’t specify them in dt_a).
Attributes of date: year, month, day. Attributes of time: hour, minute, second, microsecond, and tzinfo.
Similarly, datetime objects can be computed with instances of just date or time classes — this is done by instantiating a timedelta object.
Take away is: datetime + timedelta = datetime.
Pro tip: timedelta objects contain most (but not all) of the same classes as datetime objects. Keep in mind to convert between time units when necessary. (E.g. adding 1 year would be done by adding 365 days.)
Many data objects (such as csv) store dates in the form of strings . These strings can be parsed with the strptime method from a datetime.datetime object.
Pro tip: dates come in all formats (notations), be sure to know which you’re working with so that your dates are correctly parsed.
Creating many datetime objects by repeatedly adding timedelta objects is a straightforward and computationally efficient method of creating many dates at once.
Pro tip: the code above is memory efficient but slightly computationally taxing since each date is computed from the first instantiated date — ch_new_yr. For more massive operations, consider optimizing by memoizing the previous date and adding a single day to it.
Big dates can be a lot — here’s how to avoid ending yours in pandamonium!
Create a function which will instantiate a big sample of dates data in string format. Note: the non-standard string format is implemented to create a computationally demanding scenario.
Pro tip: Instantiating data with a function allows us to easily recover the data in its original structure.
A function to return the speed of a function. This is getting meta.
Which method will turn out to be the best?
Note: Each item in the series is converted with this pandas function. ✔ Automated string format detection.
Note: Each item in the series is converted with this datetime using the apply method of the pandas series. ✘ Automated string format detection.
Note: A dictionary of unique dates is instantiated with keys as strings and values as their datetime format. The dictionary is then mapped to the pandas series. This technique dramatically increases performance by avoiding converting repeated dates.✔ Automated string format detection.
Note: Same memoization technique as above except using a datetime parser instead of pandas. ✘ Automated string format detection.
Run each function on a dataset and compare performance.
Note:
– Code for testing speed is available here.
– Computational speed will vary between machines but relative speeds should not.
– I incorporated early stopping into my timing program to discontinue any computation that took over 10 seconds.
How does performance compare when parsing 103 unique dates, each repeated 103 times?
Function performance becomes O(n4) after some threshold — hereafter termed PT.
PT for non-memoized functions is n unique.
PT for memoized functions is n unique ** 2.
Convergence point for memoized functions is about n unique ** 3.
The slower performance of the pandas functions are most likely due to the automated string format detection.
Fastest memoized datetime function is ~485X faster than the standard pandas function!
Fastest memoized datetime function is ~485X faster than the standard pandas function!
How does performance compare when parsing 104 unique dates, each repeated 102 times?
PT’s and convergence points are consistent with previous findings.
Fastest memoized datetime function is again~485X faster than the standard pandas function!
How does performance compare when parsing 102 unique dates, each repeated 104 times?
Linear convergence of functions vary, max at O(n4).
PT’s not achieved in this exercise.
Standard pandas function has similar performance after n unique ** 3 to other methods, perhaps owing to caching in the underlying Cython.
Add a random amount of seconds to each date — this puts more stress on the parsing process and will be more computationally taxing.
Note: I did not perform a full simulation in this scenario given the extreme time for computation. Also, datetime parsers don’t work with messy data since they require a specified parsing format
# 10**3 Unique Dates # 10**6 Overall{'function': 'f_pandas', 'speed': 160.412103}{'function': 'f_memoize_pd', 'speed': 0.495355}# 10**3 Unique Dates # 10**7 Overall{'function': 'f_pandas', 'speed': 1421.018452}{'function': 'f_memoize_pd', 'speed': 1.61473}
With 103 Unique Dates (106 Overall): memoized version is 325X faster than the standard approach.
With 103 Unique Dates (107 Overall): memoized version is 880X faster than the standard approach.
Magnitude of improvement should be expected to increase in relation to task complexity with the methods provided.
If your dates contain repeats, you should consider a memoization approach.
If your dates do not contain repeats (or contain fewer than 100 repeats), you should use the built in pandas method.
While pandas is super quick for most data conversions, it can be extremely slow for converting from string to datetime format. An alternative memoized approach is given and compared to standard methods, performing up to 480X faster (880X with noisy data).
Yaakov Bressler is a data scientist and theater producer focused on connecting people with what they want. Current focus is deep learning within dynamic creative optimization (DCO). Learn more at www.adsalience.com
NYC Python Meetup — meets Wednesday evenings at the Microsoft Building in Times Square.
Data Visualization Society — online community (with locally organized meetups) focused on data visualization.
DataCamp — best place to sharpen your data science skills. (Check out their newest track on data engineering.)