Data analysis and visualization in Python | Towards Data Science

Many a time, I have seen beginners in data science skip exploratory data analysis (EDA) and jump straight into building a hypothesis function or model. In my opinion, this should not be the case. We should first perform an EDA as it will connect us with the dataset at an emotional level and yes, of course, will help in building good hypothesis function.
EDA is a very crucial step. It gives us a glimpse of what our data set is all about, its uniqueness, its anomalies and finally it summarizes the main characteristics of the dataset for us. In this post, I will share a very basic guide for performing EDA.
Step 1: Import your data set and have a good look at the data.
In order to perform EDA, we will require the following python packages.
Packages to import:
Once we have imported the packages successfully, we will move on to importing our dataset. You must be aware of read_csv() tool from pandas for reading csv files.
Import the dataset:
For the purpose of this tutorial, I have used Loan Prediction dataset from Analytics Vidhya. If you wish to code along, here is the link.
The dataset has been successfully imported. Let’s have a look at the Train dataset.
Train.head()
head() gives us a glimpse of the dataset. It can be considered similar to select * from database_table limit 5 in SQL. Let’s go ahead and explore a little bit more about the different fields in the Train dataset. info() gives uses all the relevant information on the dataset. If your dataset has more numerical variables, consider using describe() too to summarize data along mean, median, standard variance, variance etc.
Train.info()
We observe that there are 614 records and 13 columns in the dataset. Train dataset has Loan_ID, Gender, Married, Dependents, Education, Self_Employed, Property_Area and Loan_status as object types. Object type in pandas is similar to strings. ApplicantIncome field is of integer type. The other three fields namely CoapplicantIncome, Loan_Amount_Term and Credit_History are floating point types.
Step 2: Now let’s try to classify these columns as Categorical, Ordinal or Numerical/Continuous.
Categorical Variables: Categorical variables are those data fields that can be divided into definite groups. In this case, Gender(Male OR Female), Married(Yes Or No), Education(Graduate Or Not Graduate), Self_Employed(Yes Or No), Loan_Status(Y Or N) are categorical variables.
Ordinal Variables: Ordinal variables are the ones that can be divided into groups, but these groups have some kind of order. Like, high, medium, low. Dependents field can be considered ordinal since the data can be clearly divided into 4 categories : 0, 1, 2, 3+ and there is a definite ordering also. Same is the case with Property_Area (Urban, Semi-urban Or Rural).
Numerical or Continuous Variables: Numerical variables are those that can take up any value within a given range. For example, applicantIncome, CoapplicantIncome, Loan_Term, Loan_Amount, Credit_History.(I assumed that credit history can be anything between 0 and 1, but for this, it seems more like a categorical variable.)
Good Job! You may pat yourself for this, as you now know to identify different types of variables. Going ahead, we will perform univariate, bivariate and multivariate analysis one by one.
Step 3: Now we are all set to perform Univariate Analysis.
Univariate analysis involves analysis of one variable at a time. Let’s say “Gender” then we will analyze only the “Gender” field in the dataset. The analysis is usually summarized in the form of count. For visualization, we have many options such as frequency tables, bar graphs, pie charts, histograms etc. Since we are beginners, we will stick to bar charts.
Here is the basic syntax for plot() in pandas.
pandas.DataFrame.plot(kind='{bar,barh,pie,box,line,...}',figsize=(x,y), use_index={True,False},title= Name_of_plot, fontsize={integer},colormap={colors_from_matplotlib})
Let’s start with categorical variables first.
Train.Gender.value_counts(normalize=True)
This piece of code will take gender field from the Train dataset and perform groupwise count on it. Values have been normalized as it will help in visualizing percentage.
.plot(kind = 'bar', title = "Gender")
plot() tool from pandas will help in plotting a chart of a specific kind. The arguments such as kind = ‘bar’, means we want a bar chart. Feel free to choose pie, hist,line, etc. based on your requirement. title = ‘Gender’, this is quite evident, it's the name of the plot. There are other arguments too such as figsize=(x,y) etc.
We can plot all the categorical variables together using plt.subplot() and give some space between them using plt.tight_layout().
Insights :
80% of loan applicants are male in the training dataset.
Nearly 70% are married
About 75% of loan applicants are graduates
Nearly 85–90% loan applicants are self-employed
The loan has been approved for more than 65% of applicants.
Now let’s move to ordinal variables.
Insights :
Almost 58% of the applicants have no dependents.
Highest number of applicants are from Semi Urban areas, followed by urban areas.
Visualization for numerical variables will be a bit different from the ordinal and categorical variables. You may create bar plots by first creating bins, but a better plot will be a distribution, dotted line or box plot, as it will help us in identifying outliers.
Insights:
85% of applicants have a credit history of 1
Nearly 85% of loans are taken for 360 days.
The applicantIncome is mostly between 10000–40000 with some outliers.
CoapplicantIncome is lesser than applicantIncome and is within the 5000–15000, again with some outliers.
Loan Amount is mostly concentrated between 250–500.
We might have to remove outliers from applicantIncome and CoapplicantIncome. But that forms part of the data preparation stage.
Well Done So far !!
Step 4: Now let’s find some relationship between two variables, particularly between the target variable “Loan_Status” and a predictor variable from the dataset. Formally, this is known as bivariate analysis.
Bivariate Analysis: Bivariate analysis is finding some kind of empirical relationship between two variables. Let’s say ApplicantIncome and Loan_Status.
Before performing any kind of analysis, let’s create an hypothesis.This hypothesis will act as a guiding light, where to look and analyse.
I have come up with the following hypothesis after looking at the results of univariate analysis. You may have your own completely different hypothesis.
Applicants with higher income might have more chances of getting their loans approved.
Applicants with less number of dependents higher coapplicantIncome might have more chances of getting loan approvals.
Applicants who are graduates, tend to earn more and hence have higher loan approval rates.
Applicants who are married, might seem more responsible hence higher loan approval chances.
Applicants who are not self-employed, might have a higher chances of loan approval as they tend to have constant source of income. There is less uncertainty, I would say.
Candidates with property in urban areas might have higher chances of loan approval, since the cost of collateral would be high.
Good credit history should definitely correlate with loan approval.
For Gender, I don’t have any specific thing in my mind but let’s say women tend to be more responsible and hence high approval rates. (P.S. No Hate).
Now, let’s check if this hypothesis, is correct or not for this dataset.
For visualization, we will be using seaborn.countplot(). It can be considered similar to the histogram for categorical variables.
The basic syntax for sns.countplot() is as follows :
seaborn.countplot(x = 'x_axis_values', y_axis = 'y_axis_values',hue='data_field_on_which_colour_of_bars_depend', data=dataset)
Here, I have used the minimum possible arguments, you may use color, saturation etc. for amplifying your plots.
sns.set(rc={'figure.figsize':(11.7,8.27)})
sns.set() is used for setting the size of output figure.
sns.countplot(x="Gender", hue='Loan_Status', data=Train)
sns.countplot() will plot gender field counts, with bars colored based on loan_status values.
Insights :
There is not a substantial difference between male and female approval rates.
Married applicants have a slightly higher chances of loan approval.
Graduates have higher chance of loan approval compared to non-graduates.
There is no substantial difference in the loan approval rates for self_employed vs not self_employed.
Applicants with no dependents or 2 dependents have higher chances of approval. But this does not correlate well.
Applicants with properties in semi-urban areas have higher loan approval rates.
Step 5 : Let’s move on to analyzing more than two variables now. Yay !! You guessed it right, we call it “Multivariate analysis”. You should first create an hypothesis like in step 3 and act in that direction.
Here is an analysis of Gender, applicantIncome and Loan_Status.
Since applicantIncome is a continuous field, I have first created 12 bins using np.linspace() function with intervals in the range of min to max ApplicantIncome.
bins = np.linspace(Train.ApplicantIncome.min(), Train.ApplicantIncome.max(),12)
FacetGrid() is used to for plotting conditional relationships between multiple variables. Here, we have Gender group on x-axis and value counts on y-axis and Loan_Status as hue.
graph = sns.FacetGrid(Train, col="Gender", hue="Loan_Status", palette="Set2", col_wrap=2)
Next, we have mapped FacetGrid() plot with bins for ApplicantIncome.
graph.map(plt.hist, 'ApplicantIncome', bins=bins, ec="k")
Now everything together.
Let’s go ahead and do it for all other possible combinations.
Insights :
Females with income higher than 7000 has higher chances of loan approval
Females seem to loan lesser amount than men
The coapplicant income for female candidates is less compared to males. However, it does not reflect much on the loan_status.
This is quite intuitive. You understand the concept right. Now feel free to try more such plots with other predictors such as married, self_employed, property_area etc.
Finding a correlation between numerical variables in the dataset.
correlation_mat = Train.corr()
Let’s visualize the data in this correlation matrix using a heat map.
We don’t need the entire heat map. Why not delete the upper half, since its repetitive. A mask can be used to perform the task.
Insights :
There is a positive correlation between ApplicantIncome and LoanAmount, CoapplicantIncome and LoanAmount.
In a nutshell...
Exploring and knowing your datasets is a very essential step. It not only helps in finding anomalies, uniqueness and pattern in the dataset but also helps us in building better hypothesis functions. If you wish to see the entire code, here is the link to my jupyter notebook.