Dealing With Dates in Pandas — 6 Common Operations You Should Know | by Yong Cui | Towards Data Science

Besides texts and numbers, dates are a very common data type in our datasets. When we use Pandas to process dates, it’s definitely not the most straightforward task for most data scientists. When I began to use Pandas, I wasn’t familiar with date-related features, so every time when I dealt with dates, I had to search the solutions online. To save you such pain, here, I want to review 6 common operations with dates using Pandas.
When your dataset contains date information, you want to read it. For this tutorial, let’s create a CSV file called dates_text.csv, having the following data rows, as our starting point.
text_data = """date,category,balance01/01/2022,A,10002/02/2022,B,20003/12/2022,C,300"""with open("dates_text.csv", "w") as file:    file.write(text_data)
Let’s see what happens when we read this CSV file.
As shown above, the problem is that the date column is read as an object type instead of a date type, which prevents it from accessing any date-related functionalities in Pandas.
The easy solution is to ask Pandas to parse the date for us. As shown below, we specify a list object containing the date column name to the parse_dates parameter. As expected, the date column is now a kind of date type (i.e., datetime64[ns]). Please be noted that if you have multiple date columns, you can use parse_dates=[“date”, “another_date”].
It should be noted that Pandas integrates powerful date parsers such that many different kinds of dates can be parsed automatically. Thus, you usually just need to set the parse_date parameter.
You don’t always have a dataset that includes the dates as a single value. The following shows you a possible scenario where three columns that make up the complete date information.
text_data_cols = """y,m,d,category,balance2022,01,01,A,1002022,02,02,B,2002022,03,12,C,300"""with open("dates_text_cols.csv", "w") as file:    file.write(text_data_cols)
To parse dates in multiple columns, you can send a dictionary object to the parse_dates parameter.
In the United States, we typically use month before the day, and it’s the default format in Pandas too. Let’s inspect the DataFrame that we just read in the previous section. As shown, the date values are expressed as yyyy-mm-dd.
>>> df = pd.read_csv("dates_text.csv", parse_dates=["date"])>>> df        date category  balance0 2022-01-01        A      1001 2022-02-02        B      2002 2022-03-12        C      300
If you recall, it’s unclear whether our dates use month first or day first, the latter of which is used as a convention in many countries.
Raw Data:text_data = """date,category,balance01/01/2022,A,10002/02/2022,B,20003/12/2022,C,300"""
What if the raw dates do have days listed first? To read such dates properly, we can specify the dayfirst parameter, as shown below.
>>> df = pd.read_csv("dates_text.csv", parse_dates=["date"], dayfirst=True)>>> df        date category  balance0 2022-01-01        A      1001 2022-02-02        B      2002 2022-12-03        C      300
If you pay special attention to the third row, you’ll find out that 12 is now correctly listed as the month, as opposed to being parsed as the day previously.
Besides parse_dates and dayfirst, another related parameter that you can try when you find problems in parsing the date is set infer_datetime_format to True, which can help infer the default date format.
When you have a complicated dataset, you may need to implement your won date parser. Consider the following file with data using a special date format.
text_data_fmt = """date,category,balanceJan_01_2022,A,100Feb_02_2022,B,200Mar_12_2022,C,300"""with open("custom_dt_fmt.csv", "w") as file:    file.write(text_data_fmt)
When you attempt to parse the dates automatically, it doesn’t work.
In this case, we need to create a customized data parser. A possible solution is shown below.
The tricky thing here is to define the proper date format in the strptime method. You can find the complete formatting strings here. The data_parser takes a one-parameter function, in which the parameter refers to the column that you want to parse dates.
When you don’t need to deal with reading and parsing dates properly, you may still have to deal with dates during the process of preparing your datasets. In these cases, it’s useful to use the utility function of Pandas: to_datetime, and its function head is shown below (source).
pandas.to_datetime(arg, errors='raise', dayfirst=False, yearfirst=False, utc=None, format=None, exact=True, unit=None, infer_datetime_format=False, origin='unix', cache=True)
As you can see, this function takes a series of parameters that are pretty much everything you need to deal with dates. More important is that you can pass almost anything as the arg parameter. Some examples are given below.
You can see that to_datetime can work with a single string, a list of strings, or a Series object. Although the returned data types vary, they’re essentially the same thing or easily interchangeable. You can find more information here.
When we work with data that can use dates as a natural index, you can use Python’s built-in datetime module to create the index “manually”, as shown below.
However, this feature has already be considered by Pandas, and we should use the date_range function for that:
Sometimes, we can have a timestamp that consists of date and time. Consider the following example as our starting point.
As you can see, we now have the timestamp column that is in the datetime type. However, we want to separate the date from the time such that we can have other operations. To do that, you can use the dt accessor with Pandas, as shown below.
>>> df["date"] = df["timestamp"].dt.date>>> df["time"] = df["timestamp"].dt.time>>> df            timestamp category        date      time0 2022-01-01 09:01:00        A  2022-01-01  09:01:001 2022-01-02 08:55:44        B  2022-01-02  08:55:442 2022-05-01 22:01:00        C  2022-05-01  22:01:00
Actually, you can do more with the dt accessor, for instance, you can get the year, month, day, or even the week of the year! Some examples are shown below.
>>> df["timestamp"].dt.year0    20221    20222    2022Name: timestamp, dtype: int64>>> df["timestamp"].dt.month0    11    12    5Name: timestamp, dtype: int64>>> df["timestamp"].dt.weekday0    51    62    6Name: timestamp, dtype: int64
In this article, we reviewed 6 common operations related to processing dates in Pandas. There must be some aspects that I’ve overlooked here. If you have any questions, please feel free to leave a comment, and we can discuss additional features in a future article!
Thanks for reading this article. Stay connected by signing up my newsletter. Not a Medium member yet? Support my writing by using my membership link (at no extra cost for you, but a portion of your membership dues is redistributed to me by Medium as an incentive).