10 Tricks for Converting Numbers and Strings to Datetime in Pandas | by B. Chen | Towards Data Science

When doing data analysis, it is important to ensure correct data types. Otherwise, you may get unexpected results or errors. Datetime is a common data type in data science projects and the data is often saved as numbers or strings. During data analysis, you will likely need to explicitly convert them to a datetime type.
This article will discuss how to convert numbers and strings to a datetime type. More specifically, you will learn how to use the Pandas built-in methods to_datetime() and astype() to deal with the following common problems:
Converting numbers to datetimeConverting strings to datetimeHandling day first formatDealing with custom datetime formatHandling parse errorHandling missing valuesAssembling datetime from multiple columnsConverting multiple columns at onceParsing date column when reading a CSV fileDifference between astype() and to_datetime()
Converting numbers to datetime
Converting strings to datetime
Handling day first format
Dealing with custom datetime format
Handling parse error
Handling missing values
Assembling datetime from multiple columns
Converting multiple columns at once
Parsing date column when reading a CSV file
Difference between astype() and to_datetime()
Please check out the Notebook for the source code.
Pandas has 2 built-in methods astype() and to_datetime() that can be used to convert numbers to datetime. For instance, to convert numbers denote second to datetime:
df = pd.DataFrame({'date': [1470195805, 1480195805, 1490195805],                   'value': [2, 3, 4]})
When using to_datetime() , we need to call it from Pandas and set the argument unit='s':
>>> pd.to_datetime(df['date'], unit='s')0   2016-08-03 03:43:251   2016-11-26 21:30:052   2017-03-22 15:16:45Name: date, dtype: datetime64[ns]
When using astype() , we need to call it from a Series (the date column) and pass in 'datetime[s]':
>>> df['date'].astype('datetime64[s]')0   2016-08-03 03:43:251   2016-11-26 21:30:052   2017-03-22 15:16:45Name: date, dtype: datetime64[ns]
Similarly, we can convert numbers denote other units (D,s, ms, us, ns) to datetime, for instance, numbers denote the day
df = pd.DataFrame({'date': [1470, 1480, 1490],                   'value': [2, 3, 4]})>>> pd.to_datetime(df['date'], unit='D')0   1974-01-101   1974-01-202   1974-01-30Name: date, dtype: datetime64[ns]>>> df['date'].astype('datetime64[D]')0   1974-01-101   1974-01-202   1974-01-30Name: date, dtype: datetime64[ns]
Often, you’ll find that dates are represented as strings. In Pandas, strings are shown as object, it’s the internal Pandas lingo for the string.
>>> df = pd.DataFrame({'date':['3/10/2015','3/11/2015','3/12/2015'],                       'value': [2, 3, 4]})>>> df.dtypesdate     objectvalue     int64dtype: object
Both to_datetime() and astype() can be used to convert strings to datetime.
>>> pd.to_datetime(df['date'])0   2015-03-101   2015-03-112   2015-03-12Name: date, dtype: datetime64[ns]>>> df['date'].astype('datetime64')0   2015-03-101   2015-03-112   2015-03-12Name: date, dtype: datetime64[ns]
By default, to_datetime() will parse strings with month first (MM/DD, MM DD, or MM-DD) format, and this arrangement is relatively unique in the United State.
In most of the rest of the world, the day is written first (DD/MM, DD MM, or DD-MM). If you would like Pandas to consider day first instead of month, you can set the argument dayfirst to True.
df = pd.DataFrame({'date': ['3/10/2000', '3/11/2000', '3/12/2000'],                   'value': [2, 3, 4]})df['date'] = pd.to_datetime(df['date'], dayfirst=True)
Alternatively, you pass a custom format to the argument format.
By default, strings are parsed using the Pandas built-in parser from dateutil.parser.parse. Sometimes, your strings might be in a custom format, for example, YYYY-d-m HH:MM:SS. Pandas to_datetime() has an argument called format that allows you to pass a custom format:
df = pd.DataFrame({'date': ['2016-6-10 20:30:0',                             '2016-7-1 19:45:30',                             '2013-10-12 4:5:1'],                   'value': [2, 3, 4]})df['date'] = pd.to_datetime(df['date'], format="%Y-%d-%m %H:%M:%S")
If a date does not meet the timestamp limitations, we will get a ParseError when converting. For instance, an invalid string a/11/2000:
df = pd.DataFrame({'date': ['3/10/2000', 'a/11/2000', '3/12/2000'],                   'value': [2, 3, 4]})# Getting ParseErrordf['date'] = pd.to_datetime(df['date'])
to_datetime() has an argument called errors that allows you to ignore the error or force an invalid value to NaT.
df['date'] = pd.to_datetime(df['date'], errors='ignore')df
And to force an invalid value to NaT:
df['date'] = pd.to_datetime(df['date'], errors='coerce')
In Pandas, missing values are given the value NaN, short for “Not a Number”.
df = pd.DataFrame({'date': ['3/10/2000', np.nan, '3/12/2000'],                   'value': [2, 3, 4]})
When converting a column with missing values to datetime, both to_datetime() and astype() are changing Numpy’s NaN to Pandas’ NaT and this allows it to be a datetime.
>>> df['date'].astype('datetime64')0   2000-03-101          NaT2   2000-03-12Name: date, dtype: datetime64[ns]>>> pd.to_datetime(df['date'])0   2000-03-101          NaT2   2000-03-12Name: date, dtype: datetime64[ns]
Alternatively, we can replace Numpy NaN with another value (for example replacing NaN with '3/11/2000')
df = pd.DataFrame({'date': ['3/10/2000', np.nan, '3/12/2000'],                   'value': [2, 3, 4]})df['date'] = df['date'].fillna('3/11/2000').astype('datetime64[ns]')
To learn more about working with missing values
towardsdatascience.com
to_datetime() can be used to assemble a datetime from multiple columns as well. The keys (columns label) can be common abbreviations like [‘year’, ‘month’, ‘day’, ‘minute’, ‘second’, ‘ms’, ‘us’, ‘ns’]) or plurals of the same.
df = pd.DataFrame({'year': [2015, 2016],                   'month': [2, 3],                   'day': [4, 5],                   'hour': [10,11]                  })
To create a datetime column from a subset of columns
>>> pd.to_datetime(df[['month','day','year']])0   2015-02-041   2016-03-05dtype: datetime64[ns]
To create a datetime column from the entire DataFrame
>>> pd.to_datetime(df)0   2015-02-04 10:00:001   2016-03-05 11:00:00dtype: datetime64[ns]
So far, we have been converting data type one column at a time. There is a DataFrame method also called astype() allows us to convert multiple column data types at once. It is time-saving when you have a bunch of columns you want to change.
df = df.astype({    'date_start': 'datetime64',    'date_end': 'datetime64'})
If you want to set the data type for each column when reading a CSV file, you can use the argument parse_date when loading data with read_csv():
Note the data type datetime64 is not supported by dtype, and we should use parse_dates argument instead.
df = pd.read_csv(    'dataset.csv',    dtype={        # datetime64[ns] is not supported        'value': 'float16'    },    parse_dates=['date'])
To learn more about parsing date column with Pandas read_csv():
towardsdatascience.com
astype() is the common method to convert data type from one to other. The method is supported by both Pandas DataFrame and Series. If you need to convert a bunch of columns, the astype() should be the first choice as it:
can convert multiple columns at once
has the best performance (shown in the screenshot below)
However, astype() won’t work for a column with invalid data. For instance, an invalid date string a/11/2000. If we try to use astype() we would get a ParseError. As of Pandas 0.20.0, this error can be suppressed by setting the argument errors='ignore', but your original data will be returned untouched.
The Pandas to_datetime() function can handle these values more gracefully. Rather than fail, we can set the argument errors='coerce' to coerce invalid values to NaT.
In addition, it can be very difficult to use astype() when dealing with custom datetime format. The Pandas to_datetime() has an argument called format and offers more possibility in the way of custom conversion.
We have seen how we can convert a Pandas data column to a datetime type with astype() and to_datetime(). to_datetime() is the simplest way and offers error handling and more possibility in the way of custom conversion, while astype() has better performance and can convert multiple columns at once.
I hope this article will help you to save time in learning Pandas. I recommend you to check out the documentation for the astypes() and to_datetime() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
10 tricks to convert data to a numeric type in Pandas
Pandas json_normalize() you should know for flattening JSON
All Pandas cut() you should know for transforming numerical data into categorical data
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github