EDA exploratory data analysis  R Analytics python | Towards Data Science

While working on a Data Science or Machine Learning project or assignment, we have all felt the urge to fast-forward to the model building and prediction stage. However, as our mentors and experts reiterate, the initial Exploratory Data Analysis (EDA) phase is crucial to get better outcomes from the later stages of training a model and predictions. The estimates of time spent on EDA and data preparation range anywhere from 60–80% of the total time spent on a Data Science project.
Given this, are there packages in R that enable us to speed up our EDA while still doing a thorough job of it? Well, looks like there is !
In this Article, I would like to showcase the capabilities of the R package SmartEDA in making our lives easier in EDA. With just a single line of code using the methods in SmartEDA, we can get each of the following:
Much richer information
Better formatted output for readability
Comprehensive multi-point summary for each feature in the dateset
Visualize uni-variate and bi-variate analysis
All the above features of the SmartEDA come in very handy especially when handling a dataset with large number of features. Try using the traditional str()and summary()` functions with a dataset having over 10 features !
Let us now take SmartEDA for a test drive..
We will use the Titanic dataset from Kaggle. You can download it from here. This contains data on the passengers of the ill-fated Titanic voyage and if they survived the disaster or not.
#install.packages("SmartEDA")library(SmartEDA)data = read.csv("titanic.csv")
Here is a sample of the first few rows of the dataset.
head(data)
The dataset has passenger information including the class of travel, Name, Gender, Age, Number of siblings and spouses onboard, Number of Parents & Children onboard, Fare and Port of embarkation etc.
The ExpData() method gives a top level meta data information about the dataset.
ExpData(data, type=1)
The same method with type = 2 argument gives a neat summary of each feature in the dataset that includes variable type, % missing cases and number of unique values for each feature.
Awesome ! Right? Wait.. it gets better.
Summary of Continuous Features
The ExpNumStat() method generates a comprehensive multi-point statistical summary of all the continuous features in the dataset with a wealth of information including number of negative, zero and positive values, number of missing values and %, min, max, mean, median, standard deviation, coefficient of variation, IQR, Skew, Kurtosis and number of outliers... Whew !!
Also, ExpNumStat()automatically identifies the continuous features in the dataset. Here, below we have taken only a subset of all the summary points available for output. You can view the full output by removing the column index numbers and specify only the column index for the statistic of interest to you.
ExpNumStat(data,by ="A",Outlier=TRUE,round= 2)[,c(1,2,4:6,9:10,12:15,18,23)]
ExpNumStat(data, by = "A", Outlier = TRUE, round = 2)[,c(1,2,9,16:22)]
Further, setting the by argument to G or GA, we can get the summary split by ‘group’ and ‘group plus all’ by specifying a categorical feature in the group argument. This is especially useful in classification problems like the one we have here, where we can get a summary of the continuous features for each level of the categorical response variable (“Survived” in this dataset).
ExpNumStat(data[,-1],by = "GA",Outlier=TRUE,round= 2,gp = "Survived")[,c(1,2,4:6,9:10,12:15,18,23)]
In this case, we can clearly see that the Fare feature plays a big role in whether the passenger survived or not.
Summary of Categorical Features
What about summary of Categorical features? The ExpCatStat() method does the job here.
ExpCatStat(data, Target = "Survived")
The single line of code above not only provides the Chi-squared statistic and p-value between each categorical feature and the target feature, but also provides the information value(IV) for each feature and a useful labelling of its degree of association with the target feature and its predictive power. Talk about going beyond the call of duty !!
Visualization
With SmartEDA,it is not all output in numbers. It delivers in visually exploring data as well be it uni-variate or bi-variate analysis throughExpNumViz and ExpCatViz methods.
Bi-Variate Analysis : Between Pairs of Continuous Features
Passing the scatter = TRUE argument to ExpNumViz() gives a scatter plot between all the pairs of continuous features of the dateset. Here we plot Pclass vs Fare and Age vs SibSp (number of siblings/spouse on board) by selecting the index number of the plots.
ExpNumViz(data[,-1], scatter = TRUE)[4:5]
Passing just the dataset to the method plots the density distribution of each of the continuous features.
Bi-Variate Analysis : Continuous Features vs Categorical Response Feature
Again, all it takes is a single line of code to generate box plots showing the distribution for various levels of the categorical response feature (“survived” encoded by 1 and not survived encoded by 0”). We sample two of them below.
ExpNumViz(data = data,target = "Survived")[4:5]
Categorical Features: Uni-variate Analysis
ExpCatViz without any additional parameters other than the dataset, plots the % bar plots of the count of levels in each categorical feature. We have sampled a couple below.
ExpCatViz(data)
Categorical Features: Bi-Variate Analysis with Target Feature
The ExpCatViz method takes 2 additional arguments target and margin and generates barplots for all categorical features in the dataset vs each level of the target categorical feature “Survived”.
ExpCatViz(data, margin = 2, target = "Survived",           col = c("red","green"))[c(1,3:5)]
The above two plots possibly point to the tragic dilemma that was faced by passengers travelling in a larger family group with more siblings, with parents and spouse, that of being unable to leave behind family. Look at the disparity in survival rates !
So, to summarize, we were able to do substantial EDA number crunching as well as visualization with barely half a dozen lines of code. Please give your feedback and comments as well as which is your favourite R package for EDA and why.
Thanks for reading !
No rights reserved
 by the author.