Data Science: Data Cleansing And Visualization For Beginners Using Python
By: Rajesh Ramachander
Tags: Data Imputation, Data Science, Data Visualization, Pandas, Data Exploration

In this article, I am discussing an educational project work in the fascinating field of Data Science while trying to learn the ropes of Data Science. This write-up intends to share the project journey with the larger world and the outcome.

Data science is a discipline that is both artistic and scientific simultaneously. A typical project journey in Data Science involves extracting and gathering insightful knowledge from data that can either be structured or unstructured. The entire tour commences with data gathering and ends with exploring the data entirely for deriving business value. The cleansing of the data, selecting the right algorithm to use on the data, and finally devising a machine learning function is the objective in this journey. The machine learning function derived is the outcome of this art that would solve the business problems creatively.

I will be focussing exclusively on the Data cleansing, imputation, exploration, and visualization of the data. I presume the reader has a basic knowledge of Python or even any equivalent language such as Java or C or Cplusplus to follow the code snippets. The coding was done in Python and executed using Jupyter notebook. I will describe the steps we undertook in this project journey, forming this write-up’s crux.

Import Libraries

We began by importing the libraries that are needed to preprocess, impute, and render the data. The Python libraries that we used are Numpy, random, re, Mat- plotlib, Seaborn, and Pandas. Numpy for everything mathematical, random for random numbers, re for regular expression, Pandas for importing and managing the datasets, Matplotlib.pyplot, and Seaborn for drawing figures. Import the libraries with a shortcut alias as below.

Loading the data set

The dataset provided was e-commerce data to explore. The data set was loaded using Pandas.

The ‘info’ method gets a summary of the dataset object.

The shape of the dataset was determined to be 10000 rows and 14 columns. The describe specifies the basic statistics of the dataset.

The ‘nunique’ method gives the number of unique elements in the dataset.

Initial exploration of the data set

This step involved exploring the various facets of the loaded data. This step helps in understanding the data set columns and also the contents.

Interpreting and transforming the data set

In a real-world scenario, the data information that one starts with could be either raw or unsuitable for Machine Learning purposes. We will need to transform the incoming data suitably.

We wanted to drop any duplicate rows in the data set using the ‘duplicate’ method. However, as you would note below, the data set we received did not contain any duplicates.

Impute the data

While looking for invalid values in the data set, we determined that the data set was clean. The question placed on us in the project was to introduce errors at the rate of 10% overall if the data set supplied was clean. So given this ask, we decided to introduced errors into a data set column forcibly. We consciously choose to submit the errors in the

‘Purchase Price’ column as this has the maximum impact on the dataset outcome. Thus, about 10% of the ‘Purchase Price’ data randomly set with ‘numpy — NaN.’

We could have used the Imputer class from the scikit-learn library to fill in missing values with the data (mean, median, most frequent). However, to keep experimenting with hand made code; instead, I wrote a re-usable data frame impute class named ‘DataFrameWithImputor’ that has the following capabilities.

Be instantiated with a data frame as a parameter in the constructor. Introduce errors to any numeric column of a data frame at a specified error rate. Introduce errors across the dataframe in any cell of the dataframe at a specified error rate. Impute error values in a column of the data set. Find empty strings in rows of the data set. Get the’ nan’ count in the data set. Possess an ability to describe the entire data set in the imputed object. Have an ability to express any column of the dataset in the impute object. Perform forward fill on the entire data set. Perform backward fill on the entire data set.

Shown below is the impute class.

The un-imputed data set was checked for any Nan or missing strings for one final time before introducing errors.

A helper function ‘do_impute’ was defined to introduce errors in a column of the data set and impute the data set column afterwords. This function would take a condition parameter to perform imputation.

To introduce error, random cells in the ‘Purchase Price’ column is set to ‘NaN.’ Once set, there are several ways to fill up missing values or ‘NaN’:

We can remove the missing value rows itself from the data set. However, in this case, the error percentage is low at just 10%, so this method was not needed. Fill in the null cell in the data set column with a constant value. Filling the invalid section with mean and median values Fill the nulls with a random value. Filling null using data frame backfill and forward fill

The above mentioned are some common strategies applied to impute the data set. However, there are no limits to designing a radically different approach to the data set imputation itself.

Each of the imputed outcomes was studied separately — the fill (backfill and forward fill) and constant value imputations outcome shown below.

The median and random value imputations are in the code below.

Then the mean imputed outcome is visually compared with an un-imputed or clean data column as below.

From the above techniques, mean imputation was found closer to the un- imputed clean data, thus preferred. Other choices such as fill(forward and backward) also seemed to produce data set column qualitatively very close to clean data from the study above. However, the mean imputation was preferred as it gives a consistent result and a more widespread impute technique.

The data frame adopted for further visualization was the mean imputed data set.

Exploring and Analysing the data

A cleaned up and structured data is suitable for analyzing and finding exemplars using visualization.

Find relationship between Job designation and purchase amount?

2. How does purchase value depend on the Internet Browser used and Job (Profession) of the purchaser?

3. What are the patterns, if any, on the purchases based on Location (State) and time of purchase (AM or PM)?

4. How does purchase depend on ‘CC’ provider and time of purchase ‘AM or PM’?

5. What are top 5 Location(State) for purchases?

We plotted a sub-plot as below.

We can similarly repeat this subplot to view the top 5 credit cards, the top 5 email providers, and the top 5 languages involved in purchases. There are many other visualizations techniques beyond what I have described in this article, with each one capable of giving unique insights into the dataset.

Acknowledgment

I acknowledge my fellow project collaborators below, without whose contribution this project would not have been so exciting.

The Team

Rajesh Ramachander: linkedin.com/in/rramachander/ Ranjith Gnana Suthakar Alphonse Raj: linkedin.com/in/ranjith-alphonseraj-21666323/ Yashaswi Gurumurthy: linkedin.com/mwlite/in/yashaswi-gurumurthy-020521113 Praveen Manohar G: linkedin.com/in/praveen-manohar-g-9006a232 Rahul A linkedin.com/in/rahulayyappan

Closing Words

The Github link to the codebase is https://github.com/RajeshRamachander/ecom/blob/master/ecom_eda.ipynb.

We had fun and many learnings while doing some of these fundamental steps required to work through a large data set, clean, impute, and visualize the data for further work. We finished the project here, and of course, the real journey does not end here as it will progress into modeling, training, and testing phases. For us, this is only the beginning of a long trip. Every data science project that has a better and cleaner data will generate awe-inspiring results!