Pandas Profiling & Sweetviz. Python EDA tools | Towards Data Science

Data cleaning and Exploratory Data Analysis go hand-in-hand — with a better understanding of the data, can one be better positioned to spot errors or outliers for mitigation.
Most of us do EDA through pandas functions, coupled with visualizations using matplotlib to seaborn. Occasionally, we define functions to do 1) automated and 2) customized EDA of datasets (e.g. doing EDA on multiple, large datasets before merging them). While working on correlation matrices for a pet project, I came across Sweetviz & Pandas profiling and incorporated them into my EDA workflow. Here are some of the observations:
Compatibility & Installation: Weary of potential compatibility issues, a quick google/ StackOverflow search surfaced issues between pandas and pandas profiling versions. The versions that worked for me are pandas (1.0.1) and pandas profiling (2.4.0). If the default installation of pandas profiling and upgrade is not working as it should via the anaconda navigator, try:
conda install -c conda-forge/label/cf202003 pandas-profiling
Output: There are two alternatives for viewing of the output — 1. interactive widget within the jupyter notebook itself or 2. HTML format (allows viewing external to the notebook). HTML format is essentially the same in content and layout.
Pandas profiling starts with an overview of the dataset, followed by the statistical information of the variables (in alphabetical order). Correlations matrices are located under the correlations tab. Missing values (if any) are marked by empty white space in diagrams — similar to the missingno package.
Compatibility & Installation: Installation is available with pip install and is fairly straightforward. It supports Python 3.6+ and Pandas 0.25.3+, though support for Google Colab is not available yet.
Output: Sweetviz generates an HTML report by default. It is meant for the quick visualizations of predictor variables to the target variable. Sweetviz output could be thought of as a left-right panel display. The left panel at the top shows top-level dataset characteristics (e.g. number of observations in dataset, number of features(variables), duplicates (if any)), followed by an in-depth summary statistics for the respective variables.
The right panel shows a more in-depth analysis of the inter-variable relations (Pearson-correlation) and distribution of values. Grouping of variables by numerical and categorical types can be found under this space (uncertain if sweetviz can auto-identify categorical features though).
To navigate among variables, click on the variable, and a red bounding box would appear to “lock” the right panel display in place. Unclick again to move on to other variables.
An interesting feature of sweetviz is its summary associate diagram, where it differentiates between categorical and numerical variables. It served as a quick sanity check in counter-checking if the variables have been properly encoded during my personal project. For larger datasets, this could be a quick visual aid to screen through the variables.
It depends on the needs of the reader and personal preference. As it was my second foray into both libraries, I spent most of the time comparing my own EDA insights with the output of the two libraries. Personally, I’d still prefer to do my own EDA through self-defined functions and then use either library for quick counterchecking. Both pandas profiling and sweetviz automates and sped up the EDA process to a certain degree (less the initial installation troubleshooting) and provide quick insight into the characteristics of the dataset.
For beginners, it would be good to be grounded in pandas functions for EDA before trying pandas profiling and/or sweetviz. Imho, it is more important to be equipped with the fundamental statistical knowledge to interpret the output well so as to utilize these libraries to their full potential.
I hope this post has been an informative read. Til next time, cheers.