Package: evidently
EvidentlyAn open-source framework to evaluate, test and monitor ML models in production.Docs|Discord|User Newsletter|Blog|Twitter|Evidently Cloud:new: New releaseEvidently 0.4.0. Self-host an ML Monitoring interface ->QuickStart:bar_chart: What is Evidently?Evidently is an open-source Python library for data scientists and ML engineers. It helps evaluate, test, and monitor ML models from validation to production. It works with tabular, text data and embeddings.Evidently has a modular approach with 3 components on top of the sharedmetricsfunctionality.1. Tests: batch model checksTests perform structured data and ML model checks. They verify a condition and return an explicitpassorfail.You can create a custom Test Suite from 50+ tests or run a preset (for example,Data DriftorRegression Performance). You can get results as aJSON, Python dictionary, exportable HTML,visual reportinside Jupyter notebook, or as Evidently JSONsnapshot.Tests are best for automated checks. You can integrate them as a pipeline step using tools like Airflow.2. Reports: interactive visualizationsOld dashboards API was deprecated in v0.1.59. Here is themigration guide.Reports calculate various data and MLmetricsand render richvisualizations. You can create a custom Report or run a preset to evaluate a specific aspect of the model or data performance. For example, aData QualityorClassification Performancereport.You can get anHTML report(best for exploratory analysis and debugging),JSONor Python dictionary output (best for logging, documentation or to integrate with BI tools), or as Evidently JSONsnapshot.3. ML monitoring dashboardThis functionality is available from v0.4.0.You can self-host an ML monitoring dashboard to visualize metrics and test results over time. This functionality sits on top of Reports and Test Suites. You must store their outputs as Evidently JSONsnapshotsthat serve as a data source for the Evidently Monitoring UI.You can track 100+ metrics available in Evidently, from the number of nulls to text sentiment and embedding drift.:woman_technologist: Install EvidentlyMAC OS and LinuxEvidently is available as a PyPI package. To install it using pip package manager, run:pipinstallevidentlyEvidently is also available on Anaconda distribution platform. To install Evidently using conda installer, run:condainstall-cconda-forgeevidentlyIf you want visualize the Reports or Test Suites inside Jupyter notebook, you needjupyter nbextension. After installingevidently, run the two following commands in the terminal from the evidently directory. This is not required if you want to use Evidently Monitoring UI.To installjupyter nbextension, run:jupyternbextensioninstall--sys-prefix--symlink--overwrite--pyevidentlyTo enable it, run:jupyternbextensionenableevidently--py--sys-prefixThat's it! A single run after the installation is enough.WindowsEvidently is available as a PyPI package. To install it using pip package manager, run:pipinstallevidentlyTo install Evidently using conda installer, run:condainstall-cconda-forgeevidentlyNote: If you run Jupyter notebook on Windows, you will use a different method to display Reports and Test Suites. You must add the argumentinlinewhen calling the Report:report.show(mode='inline').  Read more about different environments in thedocs.:arrow_forward: Getting startedOption 1: Test SuitesThis is a simple Hello World example. Head to docs for a completeQuickstart for Reports and Test Suites.Prepare your data as two pandasDataFrames. The first is your reference data, and the second is current production data. The structure of both datasets should be identical. You need input features only to run some evaluations (e.g., Data Drift). In other cases (e.g., Target Drift, Classification Performance), you need Target and/or Prediction.After installing the tool, import the EvidentlyTest Suiteand required presets. We'll use a simple toy dataset:importpandasaspdfromsklearnimportdatasetsfromevidently.test_suiteimportTestSuitefromevidently.test_presetimportDataStabilityTestPresetfromevidently.test_presetimportDataQualityTestPresetiris_data=datasets.load_iris(as_frame='auto')iris_frame=iris_data.frameTo run theData StabilityTest Suite and display the output in the notebook:data_stability=TestSuite(tests=[DataStabilityTestPreset(),])data_stability.run(current_data=iris_frame.iloc[:60],reference_data=iris_frame.iloc[60:],column_mapping=None)data_stabilityYou can also save an HTML file. You'll need to open it from the destination folder.data_stability.save_html("file.html")To get the output as JSON:data_stability.json()Option 2: ReportsAfter installing the tool, import the EvidentlyReportand required presets:importpandasaspdfromsklearnimportdatasetsfromevidently.reportimportReportfromevidently.metric_presetimportDataDriftPresetiris_data=datasets.load_iris(as_frame='auto')iris_frame=iris_data.frameTo generate theData Driftreport, run:data_drift_report=Report(metrics=[DataDriftPreset(),])data_drift_report.run(current_data=iris_frame.iloc[:60],reference_data=iris_frame.iloc[60:],column_mapping=None)data_drift_reportSave the report as HTML. You'll later need to open it from the destination folder.data_drift_report.save_html("file.html")To get the output as JSON:data_drift_report.json()Option 3: ML monitoring dashboardThis will launch a demo project in the Evidently UI. Head to docs for a completeML Monitoring Quickstart.Recommended step: create a virtual environment and activate it.pip install virtualenv
virtualenv venv
source venv/bin/activateAfter installing Evidently (pip install evidently), run the Evidently UI with the demo projects:evidently ui --demo-projects allAccess Evidently UI service in your browser. Go to thelocalhost:8000.:computer: ContributionsWe welcome contributions! Read theGuideto learn more.:books: DocumentationFor more information, refer to a completeDocumentation. You can start with the tutorials:Get Started with Reports and Test SuitesGet Started with ML monitoring:card_index_dividers: ExamplesSimple examples on toy datasets to quickly explore what Evidently can do out of the box.TitleCode exampleTutorialContentsQuickStart Tutorial: ML MonitoringExampleTutorialPre-built ML monitoring dashboard.QuickStart Tutorial: Tests and ReportsJupyter notebookColabTutorialData Stability and custom Test Suites, Data Drift and Target Drift ReportsEvidently Metric PresetsJupyter notebookColab-Data Drift, Target Drift, Data Quality, Regression, Classification ReportsEvidently MetricsJupyter notebookColab-All individual MetricsEvidently Test PresetsJupyter notebookColab-NoTargetPerformance, Data Stability, Data Quality, Data Drift Regression, Multi-class Classification, Binary Classification, Binary Classification top-K test suitesEvidently TestsJupyter notebookColab-All individual TestsThere are more example in theCommunity Examplesrepository.IntegrationsExploreIntegrationsto see how to integrate Evidently in the prediction pipelines and with other tools.How-to guidesExplore theHow-to guidesto understand specific features in Evidently, such as working with text data.:phone: User NewsletterTo get updates on new features, integrations and code tutorials, sign up for theEvidently User Newsletter.:white_check_mark: Discord CommunityIf you want to chat and connect, join ourDiscord community!