Data Wrangling With Pandas. My journey into data science has been... | by Mohammed Raza | Towards Data Science

My journey into data science has been possible by the vast resources of the internet. The Journal of Data Science defines it as almost everything that has something to do with data. In a job, this translates to using data to have an impact on the organization by adding value. Most commonly it is to use and apply the data to solve complex business problems. One of the most common steps taken in data science work is data wrangling. The following is a concise guide on how to go about exploring, manipulating and reshaping data in python using the pandas library.
We will explore a breast cancer data set (credits: UCI) and use pandas to clean, reshape, massage and give us a clean data set, all of this will help dramatically increase the quality of our data. Note: Data quality is KEY for optimal performance with machine learning algorithms.
If you want to follow along take a look at the GitHub repo page, try and experiment around with the dataset along with the python code.
The following pandas functionalities will be covered:
Data exploration — columns, unique values in a column, describe, duplicatesDealing with missing values — quantifying missing values per column, filling & dropping missing valuesReshaping data — one hot encoding, pivot tables, joins, grouping and aggregatingFiltering dataOther — Making descriptive columns, element-wise conditional operations
Data exploration — columns, unique values in a column, describe, duplicates
Dealing with missing values — quantifying missing values per column, filling & dropping missing values
Reshaping data — one hot encoding, pivot tables, joins, grouping and aggregating
Filtering data
Other — Making descriptive columns, element-wise conditional operations
Let us begin by reading in our dataset (csv file) into pandas and displaying the column names along with their data types. Also take a moment to view the entire dataset.
import pandas as pdfilename = 'data/breast_cancer_data.csv'df = pd.read_csv(filename)df.dtypespatient_id                 int64clump_thickness          float64cell_size_uniformity     float64cell_shape_uniformity      int64marginal_adhesion          int64single_ep_cell_size        int64bare_nuclei               objectbland_chromatin          float64normal_nucleoli          float64mitoses                    int64class                     objectdoctor_name               objectdtype: object
In the data we have the following columns as described by the source — Patient ID: id number, Clump Thickness: 1–10, Uniformity of Cell Size: 1–10, Uniformity of Cell Shape: 1–10, Marginal Adhesion: 1–10, Single Epithelial Cell Size: 1–10, Bare Nuclei: 1–10, Bland Chromatin: 1–10, Normal Nucleoli: 1–10, Mitoses: 1–10, Class: malignant or benign, Doctor name: 4 different doctors.
Based on this, we can assume that patient_id is a unique identifier, class is going to tell us whether the tumor is malignant (cancerous) or benign (not cancerous). The remaining columns are numeric medical descriptions of the tumor, except for the doctor_name which is a categorical feature.
Things to keep in mind — If our goal is to predict wether a tumor is cancerous or not based on the remaining features, we will have to one hot encode the categorical data and clean up the numerical data.
From our first output we see that bare_nuclei was read as an object data type although the description is numeric. Therefore we will need to change this.
To verify that our data matches up with the source we can use the describe option in pandas:
df.describe()
This neatly summarizes some statistical data for all numerical columns. It seems that all. For categorical data we can hand this by grouping together values:
df.groupby(by =[‘class’, ‘doctor_name’]).size()
With every dataset it is vital to evaluate the missing values. How many are there? Is it an error? Are there too many missing values? Does a missing value have a meaning relative to its context?
We can sum up the total missing values using the following:
df.isna().sum()
Now that we have identified our missing values, we have a few options. We can fill them in with a certain value (zero, mean/max/median by column, string) or drop them by row. Since there are few missing values, we can drop the rows to avoid skewing the data in further analysis.
df = df.dropna(axis = 0, how = 'any')
This allows us to drop rows with any missing values in them.
To view repeating rows we can start off by looking at the number of unique values in each column.
df.nunique()
We see here that although there are 699 rows, there are only 645 unique patient_id’s. This could mean that some patient appear more than once in the dataset. To isolate these patients and view their data, we use the following:
df[df.duplicated(subset = 'patient_id', keep =False)].sort_values('patient_id')
This line displays all the duplicated patient_id’s in order. The number of times a patient shows up in the dataset can also be viewed.
repeat_patients = df.groupby(by = 'patient_id').size().sort_values(ascending =False)
This shows that one patient shows up in the data 6 times!
If we want to remove patients that show up more that 2 times in the data set.
filtered_patients = repeat_patients[repeat_patients > 2].to_frame().reset_index()filtered_df = df[~df.patient_id.isin(filtered_patients.patient_id)]
If we did not have the tilde (“~”) we would get all individuals that repeat more than twice. By adding a tilde the pandas boolean series is reversed and thus the resulting data frame is of those that do NOT repeat more than twice.
The dataset has elements of categorical data in the “doctor_name” column. To feed this data into a machine learning pipeline, we will need to convert it into a one hot encoded column. This can be done with a sci-kit learn package, however we will do it in pandas to demonstrate the pivoting and merging functionality. Start off by creating a new dataframe with the categorical data.
categorical_df = df[['patient_id','doctor_name']]categorical_df['doctor_count'] = 1 
We add a column an extra column to identify which doctor a patient deals with. Pivot this table so that we only have numerical values in the cells and the columns become the doctors’ name. Then fill in the empty cells with 0.
doctors_one_hot_encoded = pd.pivot_table( categorical_df,                                  index = categorical_df.index,                                   columns = ['doctor_name'],                                    values = ['doctor_count'] )doctors_one_hot_encoded = doctors_one_hot_encoded.fillna(0)
Then drop the multiIndex columns:
doctors_one_hot_encoded.columns = doctors_one_hot_encoded.columns.droplevel()
The ‘doctors_one_hot_encoded’ dataframe looks like this.
We can now join this back to our main table. Typically a left join in pandas looks like this:
leftJoin_df = pd.merge(df1, df2, on ='col_name', how='left')
However we are joining on the index so we pass the “left_index” and “right_index” option to specify that the join key is the index of both tables
combined_df = pd.merge(df, one_hot_encoded, left_index = True,right_index =True, how =’left’)
The resulting left join looks like this.
We can drop the column that we no longer need by the following
combined_df = combined_df.drop(columns=['doctor_name']
Another key component in data wrangling is having the ability to conduct row-wise or column wise operations. Examples of this are; rename elements within a column based on its value and create a new column that yields a specific value based on multiple attributes within the row.
For this example lets create a new column that categorizes a patients cell as normal or abnormal based on its attributes. We first define our function and the operation that it will be doing.
def celltypelabel(x):    if ((x['cell_size_uniformity'] > 5) &      (x['cell_shape_uniformity'] > 5)):                return('normal')    else:        return('abnormal')
Then we use the pandas apply function to run the celltypelabel(x) function on the dataframe.
combined_df['cell_type_label'] = combined_df.apply(lambda x: celltypelabel(x), axis=1)
Although some of these data manipulation steps can be done in SAS and excel. Doing it in python not only allows you to connect the data to vast open source resources in computer vision, machine and deep learning, but also for ETL automation purposes and more.
Thank you for reading.
— Mohammad Raza