Package: nannyml
Website•Docs•Community Slack💡 What is NannyML?NannyML is an open-source python library that allows you toestimate post-deployment model performance(without access to targets), detect data drift, and intelligently link data drift alerts back to changes in model performance. Built for data scientists, NannyML has an easy-to-use interface, interactive visualizations, is completely model-agnostic and currently supports all tabular use cases, classification andregression.The core contributors of NannyML have researched and developed multiple novel algorithms for estimating model performance:confidence-based performance estimation (CBPE)anddirect loss estimation (DLE).
The nansters also invented a new approach to detectmultivariate data driftusing PCA-based data reconstruction.If you like what we are working on, be sure to become a Nanster yourself, join ourcommunity slackand support us with a GitHubstar ⭐.☔ Why use NannyML?NannyML closes the loop with performance monitoring and post deployment data science, empowering data scientist to quickly understand andautomatically detect silent model failure. By using NannyML, data scientists can finally maintain complete visibility and trust in their deployed machine learning models.
Allowing you to have the following benefits:End sleepless nights caused by not knowing your model performance 😴Analyse data drift and model performanceover timeDiscover theroot causeto why your models are not performing as expectedNo alert fatigue!React only when necessary if model performance is impactedPainlesssetup in any environment🧠 GO DEEPNannyML ResourcesDescription☎️NannyML 101New to NannyML? Start here!🔮Performance estimationHow the magic works.🌍Real world exampleTake a look at a real-world example of NannyML.🔑Key conceptsGlossary of key concepts we use.🔬Technical referenceMonitor the performance of your ML models.🔎BlogThoughts on post-deployment data science from the NannyML team.📬NewsletterAll things post-deployment data science. Subscribe to see the latest papers and blogs.💎New in v0.10.3New features, bug fixes.🧑‍💻ContributeHow to contribute to the NannyML project and codebase.Join slackNeed help with your specific use case? Say hi on slack!🔱 Features1. Performance estimation and monitoringWhen the actual outcome of your deployed prediction models is delayed, or even when post-deployment target labels are completely absent, you can use NannyML'sCBPE-algorithmtoestimate model performancefor classification or NannyML'sDLE-algorithmfor regression. These algorithms provide you with any estimated metric you would like, i.e. ROC AUC or RSME. Rather than estimating the performance of future model predictions, CBPE and DLE estimate the expected model performance of the predictions made at inference time.NannyML can alsotrack the realised performanceof your machine learning model once targets are available.2. Data drift detectionTo detectmultivariate feature driftNannyML usesPCA-based data reconstruction. Changes in the resulting reconstruction error are monitored over time and data drift alerts are logged when the reconstruction error in a certain period exceeds a threshold. This threshold is calculated based on the reconstruction error observed in the reference period.NannyML utilises statistical tests to detectunivariate feature drift. We have just added a bunch of new univariate tests including Jensen-Shannon Distance and L-Infinity Distance, check out thecomprehensive list. The results of these tests are tracked over time, properly corrected to counteract multiplicity and overlayed on the temporal feature distributions. (It is also possible to visualise the test-statistics over time, to get a notion of the drift magnitude.)NannyML uses the same statistical tests to detectedmodel output drift.Target distribution driftcan also be monitored using the same statistical tests. Bear in mind that this operation requires the presence of actuals.3. Intelligent alertingBecause NannyML can estimate performance, it is possible to weed out data drift alerts that do not impact expected performance, combatting alert fatigue. Besides linking data drift issues to drops in performance it is also possible to prioritise alerts according to other criteria using NannyML's Ranker.🚀 Getting startedInstall NannyMLNannyML depends onLightGBM. This might require you to set install additional
OS-specific binaries. You can follow theofficial installation guide.From PyPI:pipinstallnannymlFrom Conda:condainstall-cconda-forgenannymlRunning viaDocker:docker-v/local/config/dir/:/config/runnannyml/nannymlnmlrunHere be dragons!Use the latest development version of NannyML at your own risk:python-mpipinstallgit+https://github.com/NannyML/nannymlQuick StartThe following snippet is based on ourlatest release.importnannymlasnmlimportpandasaspdfromIPython.displayimportdisplay# Load real-world data:reference_df,analysis_df,_=nml.load_us_census_ma_employment_data()display(reference_df.head())display(analysis_df.head())# Choose a chunker or set a chunk size:chunk_size=5000# initialize, specify required data columns, fit estimator and estimate:estimator=nml.CBPE(problem_type='classification_binary',y_pred_proba='predicted_probability',y_pred='prediction',y_true='employed',metrics=['roc_auc'],chunk_size=chunk_size,)estimator=estimator.fit(reference_df)estimated_performance=estimator.estimate(analysis_df)# Show results:figure=estimated_performance.plot()figure.show()# Define feature columns:features=['AGEP','SCHL','MAR','RELP','DIS','ESP','CIT','MIG','MIL','ANC','NATIVITY','DEAR','DEYE','DREM','SEX','RAC1P']# Initialize the object that will perform the Univariate Drift calculations:univariate_calculator=nml.UnivariateDriftCalculator(column_names=features,chunk_size=chunk_size)univariate_calculator.fit(reference_df)univariate_drift=univariate_calculator.calculate(analysis_df)# Get features that drift the most with count-based ranker:alert_count_ranker=nml.AlertCountRanker()alert_count_ranked_features=alert_count_ranker.rank(univariate_drift)display(alert_count_ranked_features.head())# Plot drift results for top 3 features:figure=univariate_drift.filter(column_names=['RELP','AGEP','SCHL']).plot()figure.show()# Compare drift of a selected feature with estimated performanceuni_drift_AGEP_analysis=univariate_drift.filter(column_names=['AGEP'],period='analysis')figure=estimated_performance.compare(uni_drift_AGEP_analysis).plot()figure.show()# Plot distribution changes of the selected features:figure=univariate_drift.filter(period='analysis',column_names=['RELP','AGEP','SCHL']).plot(kind='distribution')figure.show()# Get target data, calculate, plot and compare realized performance with estimated performance:_,_,analysis_targets_df=nml.load_us_census_ma_employment_data()analysis_with_targets_df=pd.concat([analysis_df,analysis_targets_df],axis=1)display(analysis_with_targets_df.head())performance_calculator=nml.PerformanceCalculator(problem_type='classification_binary',y_pred_proba='predicted_probability',y_pred='prediction',y_true='employed',metrics=['roc_auc'],chunk_size=chunk_size)performance_calculator.fit(reference_df)calculated_performance=performance_calculator.calculate(analysis_with_targets_df)figure=estimated_performance.filter(period='analysis').compare(calculated_performance).plot()figure.show()📖 DocumentationPerformance monitoringEstimated performanceRealized performanceDrift detectionMultivariate feature driftUnivariate feature drift🦸 Contributing and CommunityWe want to build NannyML together with the community! The easiest to contribute at the moment is to propose new features or log bugs underissues. For more information, have a look athow to contribute.🙋 Get helpThe best place to ask for help is in thecommunity slack. Feel free to join and ask questions or raise issues. Someone will definitely respond to you.🥷 Stay updatedIf you want to stay up to date with recent changes to the NannyML library, you can subscribe to ourrelease notes. For thoughts on post-deployment data science from the NannyML team, feel free to visit ourblog. You can also sing up for ournewsletter, which brings together the best papers, articles, news, and open-source libraries highlighting the ML challenges after deployment.📍 RoadmapCurious what we are working on next? Have a look at ourroadmap. If you have any questions or if you would like to see things prioritised in a different way, let us know!📝 Citing NannyMLTo cite NannyML in academic papers, please use the following BibTeX entry.Version 0.10.3@misc{nannyml,
        title = {{N}anny{ML} (release 0.10.3)},
        howpublished = {\url{https://github.com/NannyML/nannyml}},
        month = mar,
        year = 2023,
        note = {NannyML, Belgium, OHL.},
        key = {NannyML}
    }📄 LicenseNannyML is distributed under an Apache License Version 2.0. A complete version can be foundhere. All contributions will be distributed under this license.