10 tricks for converting Data to a Numeric Type in Pandas | by B. Chen | Towards Data Science

When doing data analysis, it is important to ensure correct data types. Otherwise, you may get unexpected results or errors. In the case of Pandas, it will correctly infer data types in many cases and you can move on with your analysis without any further thought on the topic.
Despite how well pandas works, at some point in your data analysis process you will likely need to explicitly convert data from one type to another. This article will discuss how to change data to a numeric type. More specifically, you will learn how to use the Pandas built-in methods astype() and to_numeric() to deal with the following common problems:
Converting string/int to int/floatConverting float to intConverting a column of mixed data typesHandling missing valuesConverting a money column to floatConverting boolean to 0/1Converting multiple data columns at onceDefining data types when reading a CSV fileCreating a custom function to convert data typeastype() vs. to_numeric()
Converting string/int to int/float
Converting float to int
Converting a column of mixed data types
Handling missing values
Converting a money column to float
Converting boolean to 0/1
Converting multiple data columns at once
Defining data types when reading a CSV file
Creating a custom function to convert data type
astype() vs. to_numeric()
For demonstration, we create a dataset and will load it with a function:
import pandas as pdimport numpy as npdef load_df():   return pd.DataFrame({    'string_col': ['1','2','3','4'],    'int_col': [1,2,3,4],    'float_col': [1.1,1.2,1.3,4.7],    'mix_col': ['a', 2, 3, 4],    'missing_col': [1.0, 2, 3, np.nan],    'money_col': ['£1,000.00','£2,400.00','£2,400.00','£2,400.00'],    'boolean_col': [True, False, True, True],    'custom': ['Y', 'Y', 'N', 'N']  })df = load_df()
Please check out the Github repo for the source code.
Before we diving into change data types, let’s take a quick look at how to check data types. If we want to see all the data types in a DataFrame, we can use dtypes attribute:
>>> df.dtypesstring_col      objectint_col          int64float_col      float64mix_col         objectmissing_col    float64money_col       objectboolean_col       boolcustom          objectdtype: object
This attribute is also available in Series and we can use it to check data type on a specific column. For instance, check the data type of int_col:
>>> df.int_col.dtypesdtype('int64')
If we would like to explore data, the info() method may be more useful as it provides RangeIndex, total columns, non-null count, dtypes, and memory usage. This is a lot of valuable information that can help us to grasp more of an overall picture of the data.
>>> df.info()RangeIndex: 4 entries, 0 to 3Data columns (total 8 columns): #   Column       Non-Null Count  Dtype  ---  ------       --------------  -----   0   string_col   4 non-null      object  1   int_col      4 non-null      int64   2   float_col    4 non-null      float64 3   mix_col      4 non-null      object  4   missing_col  3 non-null      float64 5   money_col    4 non-null      object  6   boolean_col  4 non-null      bool    7   custom       4 non-null      object dtypes: bool(1), float64(2), int64(1), object(4)memory usage: 356.0+ bytes
The simplest way to convert a Pandas column to a different type is to use the Series’ method astype(). For instance, to convert strings to integers we can call it like:
# string to int>>> df['string_col'] = df['string_col'].astype('int')>>> df.dtypesstring_col       int64int_col        float64float_col      float64mix_col         objectmissing_col    float64money_col       objectboolean_col       boolcustom          objectdtype: object
We can see that it is using 64-bit integer numbers by default. In some situations, it can be more memory efficient to use shorter integer numbers when handling a large dataset. To do that, you can simply call astype('int8') , astype('int16') or astype('int32')
Similarly, if we want to convert the data type to float, we can call astype('float'). By default, it is using 64-bit floating-point numbers. We can use 'float128' for more precision or 'float16' for better memory efficiency.
# string to float>>> df['string_col'] = df['string_col'].astype('float')>>> df.dtypesstring_col     float64int_col          int64float_col      float64mix_col         objectmissing_col    float64money_col       objectboolean_col       boolcustom          objectdtype: object# For more precision>>> df['string_col'] = df['string_col'].astype('float128')# For more memory efficiency>>> df['string_col'] = df['string_col'].astype('float16')>>> df['string_col'] = df['string_col'].astype('float32')
If we want to convert a float column to integers, we can try using the astype() we used above.
df['float_col'] = df['float_col'].astype('int')
However, there is a bit of a gotcha. By displaying the DataFrame, we can see that the column gets converted to integers but rounded all the values down. It might be okay, but in most cases, I would imagine that is not. If we want to convert to integers and round the way that we would expect we can do round() first.
df['float_col'] = df['float_col'].round(0).astype('int')
Now, the number 4.7 gets rounded up to 5.
Let’s move on to a column of mixed strings and numbers. When running astype('int'), we get an ValueError.
# Getting ValueErrordf['mix_col'] = df['mix_col'].astype('int')
The error shows it’s the problem with the value 'a' as it cannot be converted to an integer. In order to get around this problem, we can use Pandas to_numeric() function with argument errors='coerce'.
df['mix_col'] = pd.to_numeric(df['mix_col'], errors='coerce')
But when checking the dtypes, you will find it get converted to float64.
>>> df['mix_col'].dtypesdtype('float64')
In some cases, you don’t want to output to be float values you want it to be integers, for instance converting an ID column. We can call astype('Int64'). Note it has a capital I and is different than Numpy 'int64'. What this does is change Numpy’s NaN to Pandas’ NA and this allows it to be an integer.
>>> df['mix_col'] =        pd.to_numeric(df['mix_col'], errors='coerce').astype('Int64')>>> df['mix_col'].dtypesInt64Dtype()
Alternatively, we can replace Numpy nan with another value (for example replacing NaN with 0) and call astype('int')
df['mix_col'] = pd.to_numeric(    df['mix_col'],     errors='coerce').fillna(0).astype('int')
Now we should be fully equipped with dealing with the missing values. In Pandas, missing values are given the value NaN, short for “Not a Number”. For technical reasons, these NaN values are always of the float64.
df.missing_col.dtypesdtype('float64')
When converting a column with missing values to integers, we will also get a ValueError because NaN cannot be converted to an integer.
To get around the error, we can call astype('Int64') as we did above (Note it is captial I, same as mentioned in the last section). What this does is change Numpy’s NaN to Pandas’ NA and this allows it to be an integer.
df['missing_col'] = df['missing_col'].astype('Int64')
Alternatively, we can replace Numpy NaN with another value (for example replacing NaN with 0) and call astype('int')
df['mix_col'] = df['missing_col'].fillna(0).astype('int')
If you want to learn more about handling missing values, you can check out:
towardsdatascience.com
Let’s move on to the money column. The problem is that if we are using the method above we’re going to get all NaN or NA values because they are all strings with symbols £ and ,, and they can’t be converted to numbers. So the first what we have to do is removing all invalid symbols.
>>> df['money_replace'] =        df['money_col'].str.replace('£', '').str.replace(',','')>>> df['money_replace'] = pd.to_numeric(df['money_replace'])>>> df['money_replace']0    1000.01    2400.02    2400.03    2400.0Name: money_replace, dtype: float64
We chain 2 replace() calls, one for £ and the other for ,, to replace them with an empty string.
If you are familiar with regular expression, we can also replace those symbols with a regular expression.
>>> df['money_regex'] =        df['money_col'].str.replace('[\£\,]', '', regex=True)>>> df['money_regex'] = pd.to_numeric(df['money_replace'])>>> df['money_regex']
replace('[\£\,]', '', regex=True) says we would like to replace £ and , with an empty string. The argument regex=True assumes the passed-in pattern is a regular expression (Note it defaults to True).
We have True/False, but you can imagine a case in which need these as 0 and 1 , for instance, if you are building a machine learning model and this is one of your input features, you’d need it to be numeric and you would use 0 and 1 to represent False and True. This is actually very simple, you can just call astype('int'):
df['boolean_col'] = df['boolean_col'].astype('int')
So far, we have been converting data type one column at a time. For instance
# Converting column string_col and int_col one at a timedf['string_col'] = df['string_col'].astype('float16')df['int_col'] = df['int_col'].astype('float16')
There is a DataFrame method also called astype() allows us to convert multiple column data types at once. It is time-saving when you have a bunch of columns you want to change.
df = df.astype({    'string_col': 'float16',    'int_col': 'float16'})
If you want to set the data type for each column when reading a CSV file, you can use the argument dtype when loading data with read_csv():
df = pd.read_csv(    'dataset.csv',     dtype={        'string_col': 'float16',        'int_col': 'float16'    })
The dtype argument takes a dictionary with the key representing the column and the value representing the data type. The difference between this and above is that this method does the converting during the reading process and can be time-saving and more memory efficient.
When data is a bit complex to convert, we can create a custom function and apply it to each value to convert to the appropriate data type.
For instance, the money_col column, here is a simple function we can use:
>>> def convert_money(value):       value = value.replace('£','').replace(',', '')       return float(value)>>> df['money_col'].apply(convert_money)0    1000.01    2400.02    2400.03    2400.0Name: money_col, dtype: float64
We can also use a lambda function:
df['money_col']    .apply(lambda v: v.replace('£','').replace(',',''))    .astype('float')
The simplest way to convert data type from one to the other is to use astype() method. The method is supported by both Pandas DataFrame and Series. If you already have a numeric data type (int8, int16, int32, int64, float16, float32, float64, float128, and boolean) you can also use astype() to:
convert it to another numeric data type (int to float, float to int, etc.)
use it to downcast to a smaller or upcast to a larger byte size
However, astype() won’t work for a column of mixed types. For instance, the mixed_col has a and missing_col has NaN. If we try to use astype() we would get a ValueError. As of Pandas 0.20.0, this error can be suppressed by setting the argument errors='ignore', but your original data will be returned untouched.
The Pandas to_numeric() function can handle these values more gracefully. Rather than fail, we can set the argument errors='coerce' to coerce invalid values to NaN:
pd.to_numeric(df['mixed_col'], errors='coerce')
We have seen how we can convert a Pandas data column to a numeric type with astype() and to_numeric(). astype() is the simplest way and offers more possibility in the way of conversion, while to_numeric() has more powerful functions for error handling.
I hope this article will help you to save time in learning Pandas. I recommend you to check out the documentation for the astypes() and to_numeric() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
Pandas json_normalize() you should know for flattening JSON
All Pandas cut() you should know for transforming numerical data into categorical data
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github