Bringing Customization to Pandas Profiling | by Ian Eaves | Towards Data Science

If you’ve previously used pandas-profiling, you might have observed that column summaries are unique to the data types of each feature in your data. However, until recently it wasn’t possible to customize those summaries, so, if you wanted to automatically compute the average surface area of a sequence of shapely geometries, or the set of domain names in a sequence of email addresses, you were out of luck — until now.
The recently completed migration of pandas-profiling to the visions type system brings fully customizable type detection logic, summary algorithms, and is the first step towards end-to-end report customization including customized renderings. Over the rest of blog post I’m going to show you how to get started with visions and easily customize your data summaries using pandas-profiling.
The default pandas-profiling type system consists of four basic data types:
Categories
DateTimes
Boolean
Numeric
Technically, there is a fifth catch-all type called Unsupported for anything that doesn’t fit in the other four (these are usually complex objects like lists). If you want to inspect it’s implementation check out the ProfilingTypeSet object found in pandas_profiling.model.typeset.
NOTE: You’ll need to install pydot and pygraphviz to reproduce these plots
These typesets can be as complicated or simple as your usecase calls for. Want to distinguish between Integers and Floats? URLs from Strings? Images from Files? Something else specific to your data? You can find most of these types now available in pandas-profiling with many more available in visions, and if not available, writing your own is usually just a few lines of code. A full walk-through of types and type systems can be found here.
To get started let’s download a copy of the Titanic dataset and convert the Survived field over to a boolean.
Now that we have some data we can use pandas-profiling to create a profile report.
No surprises here, just the standard pandas-profiling workflow. In addition to the normal html report we can directly inspect the summarization values computed by pandas-profiling through the description_set attribute on the report. Let's check out our summary of the Survived attribute.
{'n_distinct': 2, 'p_distinct': 0.002244668911335578, 'is_unique': False, 'n_unique': 0, 'p_unique': 0.0, 'type': Boolean, 'hashable': True, 'value_counts_without_nan': False    549 True     342 Name: Survived, dtype: int64, 'n_missing': 0, 'n': 891, 'p_missing': 0.0, 'count': 891, 'memory_size': 1019}
There’s a lot of useful information here: memory usage, whether series values are hashable, missing value counts, etc... By default, every Boolean sequence receives an identical summary — and now, thanks to summarizers, we can customize these results!
Under the hood, pandas-profiling uses a PandasProfilingSummarizer object to generate summaries for the profile report. These summarizers map data types, like Boolean, to a list of summary functions used to create the final summary.
You can reproduce the summary above directly from this summarizer like so:
{'n_distinct': 2, 'p_distinct': 0.002244668911335578, 'is_unique': False, 'n_unique': 0, 'p_unique': 0.0, 'type': Boolean, 'hashable': True, 'value_counts_without_nan': False    549 True     342 Name: Survived, dtype: int64, 'n_missing': 0, 'n': 891, 'p_missing': 0.0, 'count': 891, 'memory_size': 1019}
To make a customized summary we need to specify a sequence of function(s) to call for each data type in our typeset. Types support arithmetic in much the you would naturally expect such that adding two types together, like Boolean + Unsupported, yields a typeset containing Boolean and Unsupported.
Combining these together, let’s write a summarizer that computes length of each series for all data types and additionally, the probability of being True for boolean.
Although this may look complicated, at heart it’s just keeping track of a bunch of functions to call on series of different data types. Behind the scenes, the summarizer object took care of composing the summary functions together to produce a final result tacking on a “type” value to indicate which summarization was used.
{'type': Boolean, 'length': 891, 'probability_true': 0.3838383838383838}
{'type': Unsupported, 'length': 891}
Although summarizers are now fully customizable, the associated html renderings haven’t been updated to expose customized statistics so be on the lookout for additional updates in the future.
Although you can now customize summaries in pandas-profiling pretty much to your hearts content, in most cases you’ll be modifying or adding to the default summarizer rather than building one entirely from scratch. This is also pretty easy, just append any new functionality you want onto it’s associated type! Let’s update the default PandasProfilingSummarizier to include the new_boolean_summary function and run a report with the new functionality
{'n_distinct': 2, 'p_distinct': 0.002244668911335578, 'is_unique': False, 'n_unique': 0, 'p_unique': 0.0, 'type': Boolean, 'hashable': True, 'value_counts_without_nan': False    549 True     342 Name: Survived, dtype: int64, 'n_missing': 0, 'n': 891, 'p_missing': 0.0, 'count': 891, 'memory_size': 1019, 'probability_true': 0.3838383838383838}
Voila, just like that we were able to insert the probability_true calculation into our profile report.
So far we’ve been quite happy with the flexibility afforded by visions to define advanced data types and support user defined functionality. We’re excited to begin rolling out more of these capabilities soon, including support for customized renderings, and are eager to gather feedback. How have you been using customized types and summaries or do you have interesting use-cases worth adding into pandas-profiling? If so, please comment below and share!