Better, Faster, Stronger Python Exploratory Data Analysis (EDA) | by Philippe Bouaziz,Data Scientist at Africa4Data,PhD | Towards Data Science

Exploratory data analysis (EDA) is a fundamental step in the understanding of most data science projects whether your developing machine learning models or business analytics. This process consists of handling missing data, coding data types (continuous, categorical), calculate associations-correlations between features, shaping data structures, building design charts (histogram, boxplot, time series, etc..). Due to the high quantity of data, finding tricks for faster analysis using automatizations library is a key advantage for becoming a unicorn data scientist. In this article, we will review the 4 most successful open source short python code lines which can be combined for making a first-class EDA. For this article, we will be analysing the sample chocolate bar rating dataset you can find here: chocolate-bar.
Pandas UI
Pandas UI
Pandas_ui is an efficient open-source alternative to bamboolib. This package reduces data pre-processing time creating useful visualizations combining python libraries such as plotly, pandas, qgrid, ipywidgets, and pandas profiling. With this package, you can delete columns, encode categorical features, merged features, filter-sort rows, groupby features, data types, unique and missing values count, replace values, quantiles statistics(Q1 median, Q3, maximum, range, interquartile range), descriptive statistics(mean, mode, standard deviation, sum, median absolute deviation), outliers, visualizations (histograms), correlations (Pearson matrices). Ultimately focusing on your machine learning goals saving time on the data explorations step. For using this package a jupyter notebook( or jupyter lab) on your computer with python 3.7 is required. Refer to this github for more details.
pip install pandas_uijupyter nbextension enable — py qgrid — sys-prefixjupyter nbextension enable — py widgetsnbextension — sys-prefixfrom pandas_ui import *pdf =pandas_ui(“../input/chocolate-bar-2020/chocolate.csv”)pdf.to_file(output_file=”pandas_ui1.html”)get_df() # to get the data frame#get_meltdf() or get_pivotdf() # to get melt or pivot dataframes if you have created any.
2. Pandas profiling
Pandas_profiling is an open-source one-line code that creates a complete HTML analytic report from a csv dataset for fast and accurate data analysis. With this package, you can obtain easily data types, unique and missing values(count, heatmap, and dendrogram), quantiles(Q1 median, Q3, maximum, range, interquartile range), descriptive statistics(mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness), outliers, visualizations such as histograms, correlations (Spearman, Pearson and Kendall matrices), text analysis (Uppercase, Space, scripts and ASCII blocks), file and image analysis (dimensions, truncated images scan). For using this package a jupyter notebook(or jupyter lab) on your computer with python 3.7 is required.
Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great but a little basic for serious exploratory data analysis. pandas_profiling extends the pandas DataFrame with df.profile_report() for quick data analysis.
pip install pandas-profilingorconda install -c anaconda pandas-profilingfrom pandas_profiling import ProfileReportdf = pd.read_csv(‘../input/chocolate-bar-2020/chocolate.csv’)pr = ProfileReport(df)pr.to_file(output_file=”pandas_profiling1.html”)pr
Refer to this GitHub for more details.
3. Sweetviz
Sweetviz is a great two-line code that creates high-density visualizations HTML report. With this package you can select features easily by the target value visualization against other features (A vs B for target feature y), comparing two datasets (train vas test). Also, you can obtain missing values, quantile and descriptive statistics, histograms, uncertainty correlations (categorical- categorical), correlation ratio (for categorical-numerical). For using this package a jupyter notebook( or jupyter lab) on your computer with python 3.7 is required.
After installation of Sweetviz (using pip install sweetviz), simply load the panda’s data frames as you normally would, then call either analyze(), compare(), or compare_intra() depending on your need (more on that below). The full documentation can be found on GitHub. For now, let’s start with the case at hand, loading it as so:
df = pd.read_csv(‘../input/chocolate-bar-2020/chocolate.csv’)from sklearn.model_selection import train_test_splittrain, test = train_test_split(df, test_size=0.3)!pip install sweetvizimport sweetviz as svsweetviz_report = sv.analyze([df,”data”],target_feat=’rating’)sweetviz_report.show_html(‘viz.html’)df1 = sv.compare(train, test)df1.show_html(‘Compare.html’)
We now have 2 data-frames (train and test), and we would like to analyse the target value “Rating”. Here the interactive report more in details : here
Squares represent categorical-featured-related variables correlations. Note that the trivial diagonal is left empty, for clarity.
4. Autoviz
Autoviz is an open-source one-line code that creates a complete HTML analytic report from a csv dataset for small and big datasets. With this package, you can obtain visualizations including bar charts, histograms, correlations heatmaps, quantile, and descriptive statistics. Autoviz offers a one-click engine Auto Viz.io in which you can upload your data online getting a completely free report directly send to your email.
AutoViz can be implemented in 4 simple steps:
!pip install autovizfrom autoviz.AutoViz_Class import AutoViz_ClassAV = AutoViz_Class()dft = AV.AutoViz(filename = “”, sep= ‘,’ , depVar=’rating’, dfte= df, header=0, verbose=2, lowess=False, chart_format=”svg”, max_rows_analyzed=2500, max_cols_analyzed= 21)dft.to_file(output_file=”autoviz_profiling.html”)
If you have some spare time I’d recommend, you’ll read this:
https://medium.com/swlh/eda-exploratory-data-analysis-e0f453d97894
Sum Up
Refer to this link Chocolate bar rating EDA for a complete EDA analysis of chocolate bar rating using these tools and other well-known visualizations.
This brief overview is a reminder of the importance of Exploratory Data Analysis using python in data science. This post has for the scope to covered 4 essential Python EDA tools for making a complete exploration workflow, as well as useful documentation.
I hope you enjoy it, keep exploring!!!