Package: popmon
popmonis a package that allows one to check the stability of a dataset.popmonworks with bothpandasandspark datasets.popmoncreates histograms of features binned in time-slices,
and compares the stability of theprofilesand distributions of
those histograms usingstatistical tests, both over time and with respect to a reference.
It works with numerical, ordinal, categorical features, and the histograms can be higher-dimensional, e.g. it can also track correlations between any two features.popmoncanautomatically flagand alert onchanges observed over time, such
as trends, shifts, peaks, outliers, anomalies, changing correlations, etc,
using monitoring business rules.AnnouncementsSpark 3.0With Spark 3.0, based on Scala 2.12, make sure to pick up the correcthistogrammarjar files:spark=SparkSession.builder.config("spark.jars.packages","io.github.histogrammar:histogrammar_2.12:1.0.20,io.github.histogrammar:histogrammar-sparksql_2.12:1.0.20",).getOrCreate()For Spark 2.X compiled against scala 2.11, in the string above simply replace 2.12 with 2.11.ExamplesFlight Delays and Cancellations Kaggle dataSynthetic data (code example below)DocumentationThe entirepopmondocumentation including tutorials can be found atread-the-docs.NotebooksTutorialColab linkBasic tutorialDetailed example (featuring configuration, Apache Spark and more)Incremental datasets (online analysis)Report interpretation (step-by-step guide)Check it outThepopmonlibrary requires Python 3.6+ and is pip friendly. To get started, simply do:$pipinstallpopmonor check out the code from our GitHub repository:$gitclonehttps://github.com/ing-bank/popmon.git$pipinstall-epopmonwhere in this example the code is installed in edit mode (option -e).You can now use the package in Python with:importpopmonCongratulations, you are now ready to use the popmon library!Quick runAs a quick example, you can do:importpandasaspdimportpopmonfrompopmonimportresources# open synthetic datadf=pd.read_csv(resources.data("test.csv.gz"),parse_dates=["date"])df.head()# generate stability report using automatic binning of all encountered features# (importing popmon automatically adds this functionality to a dataframe)report=df.pm_stability_report(time_axis="date",features=["date:age","date:gender"])# to show the output of the report in a Jupyter notebook you can simply run:report# or save the report to filereport.to_file("monitoring_report.html")To specify your own binning specifications and features you want to report on, you do:# time-axis specifications alone; all other features are auto-binned.report=df.pm_stability_report(time_axis="date",time_width="1w",time_offset="2020-1-6")# histogram selections. Here 'date' is the first axis of each histogram.features=["date:isActive","date:age","date:eyeColor","date:gender","date:latitude","date:longitude","date:isActive:age",]# Specify your own binning specifications for individual features or combinations thereof.# This bin specification uses open-ended ("sparse") histograms; unspecified features get# auto-binned. The time-axis binning, when specified here, needs to be in nanoseconds.bin_specs={"longitude":{"bin_width":5.0,"bin_offset":0.0},"latitude":{"bin_width":5.0,"bin_offset":0.0},"age":{"bin_width":10.0,"bin_offset":0.0},"date":{"bin_width":pd.Timedelta("4w").value,"bin_offset":pd.Timestamp("2015-1-1").value,},}# generate stability reportreport=df.pm_stability_report(features=features,bin_specs=bin_specs,time_axis=True)These examples also work with spark dataframes.
You can see the output of such example notebook codehere.
For all available examples, please see thetutorialsat read-the-docs.Pipelines for monitoring dataset shiftAdvanced users can leverage popmon’s modular data pipeline to customize their workflow.
Visualization of the pipeline can be useful when debugging, or for didactic purposes.
There is ascriptincluded with the package that you can use.
The plotting is configurable, and depending on the options you will obtain a result that can be used for understanding the data flow, the high-level components and the (re)use of datasets.Example pipeline visualization (click to enlarge)Reports and integrationsThe data shift computations that popmon performs, are by default displayed in a self-contained HTML report.
This format is favourable in many real-world environments, where access may be restricted.
Moreover, reports can be easily shared with others.Access to the datastore means that its possible to integrate popmon in almost any workflow.
To give an example, one could store the histogram data in a PostgreSQL database and load that from Grafana and benefit from their visualisation and alert handling features (e.g. send an email or slack message upon alert).
This may be interesting to teams that are already invested in particular choice of dashboarding tool.Possible integrations are:GrafanaKibanaResources on how to integrate popmon are available in theexamples directory.
Contributions of additional or improved integrations are welcome!Comparison and profile extensionsExternal libraries or custom functionality can be easily added toProfilesandComparisons.
If you developed an extension that could be generically used, then please consider contributing it to the package.Popmon currently integrates:DiptestA Python/C++ implementation of Hartigan & Hartigan’s dip test for unimodality.
The dip test tests for multimodality in a sample by taking the maximum difference, over all sample points, between the empirical distribution function, and the unimodal distribution function that minimizes that maximum difference.
Other than unimodality, it makes no further assumptions about the form of the null distribution.To enable this extension install diptest usingpip install diptestorpip install popmon[diptest].ResourcesPresentationsTitleHostDateSpeakerpopmon: Analysis Package for Dataset Shift DetectionSciPy Conference 2022July 13, 2022Simon BrugmanPopmon - population monitoring made easyBig Data Technology Warsaw Summit 2021February 25, 2021Simon BrugmanPopmon - population monitoring made easyData Lunch @ EnecoOctober 29, 2020Max Baak, Simon BrugmanPopmon - population monitoring made easyData Science Summit 2020October 16, 2020Max BaakPopulation Shift Monitoring Made Easy: the popmon packageOnline Data Science Meetup @ ING WBAAJuly 8 2020Tomas SostakPopmon: Population Shift Monitoring Made EasyPyData Fest Amsterdam 2020June 16, 2020Tomas SostakPopmon: Population Shift Monitoring Made EasyAmundsen Community MeetupJune 4, 2020Max BaakArticlesTitleDateAuthorPOPMON v1.0.0: The Dataset-Shift PokémonAug 3, 2022Pradyot PatilMonitoring Model Drift with PythonApril 16, 2022Jeanine SchoonemannThe Statistics Underlying the Popmon HoodApril 15, 2022Jurriaan Nagelkerke and Jeanine Schoonemannpopmon: code breakfast sessionNovember 9, 2022Simon BrugmanPopulation Shift Analysis: Monitoring Data Quality with PopmonMay 21, 2021Vito GentilePopmon Open Source Package — Population Shift Monitoring Made EasyMay 20, 2020Nicole MpozikaSoftwareKedro-popmonis a plugin to integrate popmon reporting with kedro. This plugin allows you to automate the process of popmon feature and output stability monitoring. Package created byMarian DabrowskiandStephane Collot.Project contributorsThis package was authored by ING Analytics Wholesale Banking (INGA WB).
Special thanks to the following people who have contributed to the development of this package:Ahmet Erdem,Fabian Jansen,Nanne Aben, Mathieu Grimal.Citing popmonIfpopmonhas been relevant in your work, and you would like to acknowledge the project in your publication, we suggest citing the following paper:Brugman, S., Sostak, T., Patil, P., Baak, M.popmon: Analysis Package for Dataset Shift Detection. Proceedings of the 21st Python in Science Conference. 161-168 (2022). (link)In BibTeX format:@InProceedings{popmon-proc-scipy-2022,author={ {S}imon {B}rugman and {T}omas {S}ostak and {P}radyot {P}atil and {M}ax {B}aak },title={ popmon: {A}nalysis {P}ackage for {D}ataset {S}hift {D}etection },booktitle={ {P}roceedings of the 21st {P}ython in {S}cience {C}onference },pages={ 161 - 168 },year={ 2022 },editor={ {M}eghann {A}garwal and {C}hris {C}alloway and {D}illon {N}iederhut and {D}avid {S}hupe },}Contact and supportIssues & Ideas & Support:https://github.com/ing-bank/popmon/issuesPlease note that INGA WB provides support only on a best-effort basis.LicenseCopyright INGA WB.popmonis completely free, open-source and licensed under theMIT license.