Simple Guide to Data Cleaning with Python | by June Tao Ching | Towards Data Science

Data cleaning is the most crucial step in any project, if we do not take care of it properly, it might lead us to a completely different conclusion. Often than not, we might spend half of the time cleaning our data in most projects.
In this article, I am going to share some Python functions that can help us in data cleaning especially in:
Dropping unused column
Removing duplicate
Mapping of data
Dealing with null data
We are going to use pandas in this project, let’s install the package if we do not have it.
conda install pandas
I have modified the famous titanic dataset from Kaggle for demo purpose, you can download the dataset here.
Let’s import the package and read the dataset.
df.head() will display the first 5 rows of the dataframe, you can quickly take a glance at the dataset by using this function.
Based on our observation, there is an invalid/null Unnamed: 13 column that we do not need. We can drop it by using the function below.
If you wish to drop multiple columns, an array is needed to pass in columns parameter.
inplace = True will do the operation directly on the dataframe itself, by default it will create another copy and you have to assign it to the dataframe again like df = df.drop(columns="Unnamed: 13").
Let’s check for duplicates in this dataset by using this function.
df[df.duplicated(keep=False)]
keep allows a few parameters to check on duplicates.
first : Mark duplicates as True except for the first occurrence.
last : Mark duplicates as True except for the last occurrence.
False : Mark all duplicates as True.
In this case, I would like to show all duplicates, so False is passed as the parameter. Now we have seen there are duplicates in this dataset, I would like to remove them and keep the first occurrence. The following function is used to keep the first occurrence.
df = df.drop_duplicates(keep="first")
We can use len(df) or df[df.duplicated(keep=False)] to check if the duplicates are removed. df[df.duplicated(keep=False)] will return null if the duplicates are removed.
With the help of df["Sex"].unique and df["Sex"].hist(), we found out that there are other values such as m, m and F exist in this column. This might due to the wrong input from the data source and we have to assume these values are correct and map to male or female.
df["Sex"] = df["Sex"].map({    "male": "male",    "m": "male",    "m ": "male",    "M": "male",    "F": "female",    "f": "female",    "female": "female"})
The function above is used to map those values to male or female.
Note: please make sure the default values male and female are included in the mapping, else it will become nan after you perform the mapping.
In this column, there are 3 missing values: -, na and NaN. Pandas doesn’t recognize - and na as null. We have to replace them with null before dealing with them.
replace() is used to replace - and na to null.
If we found out this while reading the data, we actually can handle this missing value by passing the missing values to thena_values parameter. The results are the same.
Now we have replaced them with null values, how are we going to do with those missing value?
Solution 1: Drop the observation (row) / feature (column)
If we are sure about the missing data are not useful or the missing data are only a small portion of the data, we can drop the rows that contain missing values.
In statistics, this method is called listwise deletion, it is a method for handling missing data. In this method, an entire record is excluded from analysis if any single value is missing.
If we are confident that this feature (column) does not provide useful information or the missing value percentage is high, we can drop the entire column. This is very useful when doing statistical analysis, since filling in the missing values may yield unexpected or biased results.
Solution 2: Impute the missing values
It means to calculate the missing values based on other data. For example, we can compute missing values for age with the date of birth.
In this case, we do not have date of birth, we can replace the missing values with the mean or median (mode for categorical value) of the data.
Note: Mean is most useful when the data is not skewed, while the median is more robust, not sensitive to outliers, and thus used when data is skewed.
Let’s use median to replace the missing value in this case.
df["Age"].median is used to compute the median of the data while fillna is used to replace the missing value with the median.
Now you have learnt how to clean your data in Python with pandas. I hope this article is useful to you. Do drop me a comment if I made any mistakes or typos.
You can view the complete Jupyter notebook in my Github. Cheers!
If you enjoyed reading this piece, you might also enjoy these:
towardsdatascience.com
towardsdatascience.com
You can find links to my other works on Medium and follow me here. Thanks for reading!