A better EDA with Pandas-profiling | by Thomas Gey | Towards Data Science

Mostly in Data Science, data we collect aren’t as clean and complete as available datasets on internet, designed and prepared for the application of machine learning models. Mostly, data comes from external sources, is not well structured or is simply incomplete, and it’s your job to make this data welcoming and usable!
While data scientists are aware of the importance of data quality, this is not necessarily the case for business leaders. They focus more on adopting technology as early as possible to maintain a competitive advantage and often have little understanding of the practical aspects. This means that people use everything in their datasets and hope for the best, under the pretext that quantity can compensate for quality.
Because of this difference of mindset, a data scientist spends 80% of his time preparing data sets before modelling.
Why 80% of his time? There’s missing values, unbalanced data, column names without any meaning, deduped data, outdated data and other reasons. Awareness of the importance of data is recent. As a result, even if the trend is reversed, the data is rarely easily accessible and well structured.
medium.com
The consequence of a bad dataset is simple and logic: predictive models made with these datasets will have low accuracy and low efficiency. Train a model with incorrect data will create a bias and your results will be far from reality. A bad model is a model unable to be deployed, so its a loss of money for the company developing the model. A car built with rusted pieces will not drive correctly or at worst, not drive at all... and nobody wants to use or buy a rusted car!
So, your dataset is an essential key to your project. Its technical and economic success depends on the quality of data you selected. During a machine learning project, skipping this data evaluation step can lead to a considerable waste of time and you will have to start your project over again from the beginning. It seems logic now data scientists spend so much time investigating and preparing datasets!
becominghuman.ai
Low quality data will only lead to low quality results.
To ensure that our datasets are useful, a good practice is EDA, Exploratory Data Analysis.
An EDA is a way to familiarize yourself with the dataset. Through this reflection work, it is the assurance of working with interesting, coherent and cleaned data. This step is very visual and is based on summary statistics and graphical representations.
By an EDA, data scientists can find which features are important or the correlation between them. On the other hand, an EDA allows you to discover erroneous or missing values, detect anomalies or reject a hypothesis. The selection of feature variables will be later used for machine learning.
In general, an Exploratory Data Analysis is followed by a feature engineering/data augmentation step where you work on the initial data to bring them additional value
towardsdatascience.com
For this example, I took a dataset that is perfect for EDA, FIFA 19 complete player dataset. Data contains multiple data types, missing values, and many metrics are applicable. Several complete analyses on this dataset are available here. I use JupyterLab as IDE because of its flexibility and its user-friendly interface.
Let’s start by importing the data from the CSV file using Pandas library:
Pretty quick to load. Now an overview with data.sample(5), a method that selects randomly 5 rows. It’s better to use .sample() than .head() if you don’t know how your data is sorted.
Let’s get some descriptive statistics with .describe(). This method “summarizes the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values”.
Now we have descriptive statistics, we’re gonna check missing values. We’ll just print 10 ordered features with more than 10% missing values:
As you can see, doing an EDA on a notebook is convenient and efficient for a data scientist. Nevertheless, the visual remains summary and is print by metrics. Let’s see with Pandas-profiling now how we can print this information dynamically and with fewer efforts.
It is by seeing that we learn
Github description: “Generates profile reports from a pandas Dataframe”
Pandas-profiling brings all the bricks together to a complete EDA: Most frequent values, missing values, correlations, quantile and descriptive statistics, data length and more. Thanks to these metrics, you’ll quickly see the distribution and disparity of your data. That information is essential to know if data is useful for future usages.
github.com
Pandas-profiling presents different metrics in a structured way through an HTML report. Thanks to its interaction, it is easy to switch from one feature to another and access its metrics.
Let’s see how to use it:
pip install pandas-profiling
Using our previous FIFA Dataset:
import pandas as pdimport pandas_profilingdata_fifa = pd.read_csv('fifa.csv')profile = data_fifa.profile_report(title='Pandas Profiling Report')profile.to_file(output_file="fifa_pandas_profiling.html")
In the code above, we only import pandas and pandas-profiling dependencies, read our CSV file and call the profile_report() method because pandas_profiling extends the pandas DataFrame with data_fifa.profile_report() . Then, we export our ProfileReport object as an HTML file with .to_file() .
And voilà! Here, our HTML file is located at the root of the folder :
The image above presents the “overview” part of the report. This part briefly presents information on the type of variables, missing values, or the size of the dataset.
Pandas-profiling uses matplotlib library for graphs and jinja2 as the template engine for its interface.
As a freelance, when I have to work on a new dataset for a customer, I always produce first a pandas-profiling, it helps me to soak up the dataset. This practice allows me to quantify the processing time of the dataset. How many features seem to be correct? How many contain missing values? What percentage for each feature? Which variable depends on another?
Also, this report can serve as an interface to give a global aspect of data health to a client. Instead of presenting your analysis on your Jupyter notebook, graph after graph with code between, the report centralizes the metrics by feature, and has a more user-friendly interface. My clients like to have a complete follow-up of the missions I carry out for them and would like to know my progress regularly. I very often use this report to give a health status of the data. This step is followed by the second step of more in-depth data analysis and visualization.
Big data’s potential just keeps growing. Taking full advantage means companies must incorporate analytics and predictive systems into their strategic vision and use it to make better, faster decisions. Often, the data we retrieve contains erroneous or missing data. To be truly effective, these data must be analyzed and processed. That’s the role of the data scientist. With pandas-profiling, data scientists are able to produce quick exploratory data analysis reports with fewer efforts. The report is clear, easy to use and readable by anybody with basic statistic skills.
With this global understanding of what you have in front of you, you will have some food for thought to go further in your analysis, start processing the data or looking for external data sources to enhance your quality data. It’s crucial to pre-process data and improve its quality because the quality of our predictive machine learning model is directly correlated with the quality of the data that we feed into our model.
I hope you like this little introduction to Pandas-profiling! This is the first article I’m writing on medium, I’m interested in any comments and improvements!