Effortless Exploratory Data Analysis (EDA) | by Raj Sangani | Towards Data Science

Before we jump to model building, understanding the data at hand is essential. Analysing the data alone can give us valuable insights to solve our problems. Moreover understanding data is very useful to determine which features would help our model, which features can be done away with , how can we deal with missing values & correlations and so on. Building a model after considering all these factors will ensure that it is robust & can be generalised well.
Since the data science pipeline from EDA to model prototyping is somewhat structured (for simple ML tasks at least ) we can use tools like PyCaret to automate EDA, model building and tuning hyper-parameters. It has a ton of other features like bagging , boosting, stacking models, generating polynomial features from data , transforming categorical data with high cardinalities and many more, all of which I will talk about in a later article. Today our focus will be on EDA. It is worthwhile to note that PyCaret uses pandas-profiling under the hood to perform EDA.
I have used the popular California Housing Prices dataset (1990) here. Let’s try and analyse what features affect housing prices within a block. I would strongly recommend users to try this code on Google Colab or Jupyter unless your device has enough RAM.
Installing PyCaret and making it Colab ready.
Installing PyCaret and making it Colab ready.
!pip install pycaretfrom pycaret.utils import enable_colabenable_colab()import pandas as pdfrom pycaret.regression import *dataset=pd.read_csv('/content/housing.csv') #Read the Csv File
2. Generate the EDA report !
exp1 = setup(data = dataset, target = 'median_house_value', session_id=123 , profile=True)
Make sure that the profile parameter is set to True
Firstly it’s remarkable that the report was generated in only 18.99 seconds. Plus we have this really cool navbar to jump to sections within the report.
Quickly head over to sample which basically shows us the head of the dataset.
The overview section tells us about the dataset structure, the number of missing cells , duplicate cells, size taken up by the dataset and the number of variables along with their types.
Let’s observe a numeric feature in detail (Longitude) under the Variables section.
Once we click on toggle details under the variable name we have tons of data, the quantile statistics, standard deviations , distinct values within the column. This is just the beginning! Once we switch over to the histogram tab we can see the distribution of the variable.
We also have the Common Values and Extreme Values tabs here that show us the frequency of the values in the column & also the Minimum and Maximum values in the column.
On switching to the categories tab we get some amazing plots , especially the all important pie chart. We can also see the distinct values in our columns & their counts through charts.
Under the interactions tab we can see how two variables are correlated. Feel free to play around with the features. As an example we can see how median_income is correlated with median_house_value.
We can see how all numeric features are correlated & also toggle the correlation types. The report also describes how that specific type of correlation is calculated.
This section tells us about the missing values in our columns. My favourite feature is the Matrix depiction of missing values since it shows us the rows that contain missing values ! In the diagram below, the white horizontal lines indicate the missing values.
This EDA report is super helpful and covers almost all the basic requirements we need for simple EDA tasks. It does miss out on some important features like One Way Anova Test & Chi -Square Tests but you can’t have it all in 19 seconds and one line of code!
Although this is very handy, I feel that EDA is not a sclerotic process and often for tough problems creative techniques that are not part of this library at the moment can be very insightful.
Check out my GitHub for some other projects and the entire code. You can contact me on my website. Thank you for your time!