Data preprocessing with Python Pandas | by Angelica Lo Duca | Towards Data Science

This tutorial explains how to preprocess data using the pandas library. Preprocessing is the process of doing a pre-analysis of data, in order to transform them into a standard and normalized format.
Preprocessing involves the following aspects:
missing values
data standardization
data normalization
data binning
In this tutorial we deal only with missing values.
You can download the source code of this tutorial as a Jupyter notebook from my Github Data Science Repository.
In this tutorial we will use the dataset related to Hepatitis, which can be downloaded from this link.
Firstly, import data using the pandas library and convert them into a dataframe. Through the head(10) method we print only the first 10 rows of the dataset
import pandas as pddf = pd.read_csv('hepatitis.csv')df.head(10)
We note that the dataset presents some problems. For example, the column email is not available for all the rows. In some cases it presents the NaN value, which means that the value is missing.
In order to check whether our dataset contains missing values, we can use the function isna(), which returns if an cell of the dataset if NaN or not. Then we can count how many missing values there are for each column.
df.isna().sum()
which gives the following output:
age                 0sex                 0steroid             1antivirals          0fatigue             1malaise             1anorexia            1liver_big          10liver_firm         11spleen_palpable     5spiders             5ascites             5varices             5bilirubin           6alk_phosphate      29sgot                4albumin            16protime            67histology           0class               0dtype: int64
Now we can count the percentage of missing values for each column, simply by dividing the previous result by the length of the dataset (len(df)) and multiplying per 100.
df.isna().sum()/len(df)*100
which gives the following output:
age                 0.000000sex                 0.000000steroid             0.645161antivirals          0.000000fatigue             0.645161malaise             0.645161anorexia            0.645161liver_big           6.451613liver_firm          7.096774spleen_palpable     3.225806spiders             3.225806ascites             3.225806varices             3.225806bilirubin           3.870968alk_phosphate      18.709677sgot                2.580645albumin            10.322581protime            43.225806histology           0.000000class               0.000000dtype: float64
When dealing with missing values, different alternatives can be applied:
check the source, for example by contacting the data source to correct the missing values
drop missing values
replace the missing value with a value
leave the missing value as it is.
Dropping missing values can be one of the following alternatives:
remove rows having missing values
remove the whole column containing missing values We can use the dropna() by specifying the axis to be considered. If we set axis = 0 we drop the entire row, if we set axis = 1 we drop the whole column. If we apply the function df.dropna(axis=0) 80 rows of the dataset remain. If we apply the function df.dropna(axis=1), only the columns age, sex, antivirals, histology and class remain. However, removed values are not applied to the original dataframe, but only to the result. We can use the argument inplace=True in order to store changes in the original dataframe df (df.dropna(axis=1,inplace=True)).
df.dropna(axis=1)
As an alternative, we can specify only the column on which the dropping operation must be applied. In the following example, only missing rows related to the column liver_big are considered. This can be achieved through the subset parameter, which permits to specify the subset of columns where to apply the dropping operation.
df.dropna(subset=['liver_big'],axis=0,inplace=True)
Now we can check whether there are still missing values for the column indirizzo.
df.isna().sum()/len(df)*100
Another alternative involves the dropping of columns where a certain percentage of not-null values is available. This can be achieved through the thresh parameter. In the following example we keep only columns where there are at least the 80% of not null values.
df.dropna(thresh=0.8*len(df),axis=1,inplace=True)
A good strategy when dealing with missing values involves their replacement with another value. Usually, the following strategies are adopted:
for numerical values replace the missing value with the average value of the column
for categorial values replace the missing value with the most frequent value of the column
use other functions
In order to replace missing values, three functions can be used: fillna(), replace() and interpolate(). The fillna() function replaces all the NaN values with the value passed as argument. For example, for numerical values, all the NaN values in the numeric columns could be replaced with the average value. In order to list the type of a column, we can use the attribute dtypes as follows:
df.dtypes
which gives the following output:
age                  int64sex                 objectsteroid             objectantivirals            boolfatigue             objectmalaise             objectanorexia            objectliver_big           objectliver_firm          objectspleen_palpable     objectspiders             objectascites             objectvarices             objectbilirubin          float64alk_phosphate      float64sgot               float64albumin            float64histology             boolclass               objectdtype: object
Firstly, we select numeric columns.
import numpy as npnumeric = df.select_dtypes(include=np.number)numeric_columns = numeric.columns
Then, we fill the NaN values of numeric columns with the average value, given by the df.mean() function.
df[numeric_columns] = df[numeric_columns].fillna(df.mean())
Now, we can check whether the NaN values in numeric columns have been removed.
df.isna().sum()/len(df)*100
which gives the following output:
age                0.000000sex                0.000000steroid            0.689655antivirals         0.000000fatigue            0.000000malaise            0.000000anorexia           0.000000liver_big          0.000000liver_firm         0.689655spleen_palpable    0.689655spiders            0.689655ascites            0.689655varices            0.689655bilirubin          0.000000alk_phosphate      0.000000sgot               0.000000albumin            0.000000histology          0.000000class              0.000000dtype: float64
We note that in dtypes the categorial columns are described as objects. Thus we can select the object columns. We would like to consider only boolean columns. However the object type includes also the column class, which is a string. We select all the object columns, and then we remove from them the column class. Then we can convert the type of the result to bool.
boolean_columns = df.select_dtypes(include=np.object).columns.tolist()boolean_columns.remove('class')df[boolean_columns] = df[boolean_columns].astype('bool')
Now we can replace all the missing values for booleans with the most frequent value. We can use the mode() function to calculate the most frequent value. We use the fillna() function to replace missing values, but we could use also the replace(old_value,new_value) function.
df[boolean_columns].fillna(df.mode())
Now our dataset does not contain any missing value.
df.isna().sum()/len(df)*100
which gives the following output:
age                0.0sex                0.0steroid            0.0antivirals         0.0fatigue            0.0malaise            0.0anorexia           0.0liver_big          0.0liver_firm         0.0spleen_palpable    0.0spiders            0.0ascites            0.0varices            0.0bilirubin          0.0alk_phosphate      0.0sgot               0.0albumin            0.0histology          0.0class              0.0dtype: float64
Another solution to replace missing values involves the usage of other functions, such as linear interpolation. In this case, for example, we could replace a missing value over a column, with the interpolation between the previous and the next ones. This can be achieved through the use of the interpolate() function.
Since we have already managed all the missing values, we reload the dataset.
df = pd.read_csv('hepatitis.csv')df.isna().sum()/len(df)*100
We select only numeric columns.
numeric = df.select_dtypes(include=np.number)numeric_columns = numeric.columnsdf.head(10)
Now we can apply the interpolate() function to numeric columns, by setting also the limit direction to forward. This means that the linear interpolation is applied starting from the first row until the last one.
df[numeric_columns] = df[numeric_columns].interpolate(method ='linear', limit_direction ='forward')
For example, in line 6 the column bilirubin, which was NaN before the interpolation, now assumes the value 0.95, which is the interpolation between 0.90 (line 4) and 1.00 (line 6).
df.head(10)
In this tutorial we have seen one of the aspects of data preprocessing, which is dealing with missing data. Missing data can alter the data analysis process, thus they must be managed.
Three strategies can be used to deal with missing data:
drop missing data: this can be done when the dataset has a small number of missing datareplace missing data with other values, such as the mean or the most frequent valueleave missing data as they are.
drop missing data: this can be done when the dataset has a small number of missing data
replace missing data with other values, such as the mean or the most frequent value
leave missing data as they are.
If you would like to learn about the other aspects of data preprocessing, such as data standardization and data normalization, stay tuned...
If you wanted to be updated on my research and other activities, you can follow me on Twitter, Youtube and and Github.