Package: verifyml
VerifyMLVerifyML is an opinionated, open-source toolkit and workflow to help companies implement human-centric AI practices. It is built on 3 principles:A git and code first approach to model development and maintenance.Automatic generation of model cards - machine learning documents that provide context and transparency into a model's development and performance.Model tests for validating performance of models across protected groups of interest, during development and in production.ComponentsAt the core of the VerifyML workflow is a model card that captures 6 aspects of a model:Model detailsConsiderationsModel / data parametersQuantitative analysisExplainability analysisFairness analysisIt is adapted from Google's Model Card Toolkit and expanded to include broader considerations such as fairness and explainability.Amodel card editor, provides a web-based interface to gather input and align stakeholders across product, data science, compliance.Our Python toolkit supports data science workflows, and allows a custom model to be built and logged within the model card framework. The package also contains perfomance and fairness tests for model diagnostics, fairness and reliability checks.Being a standard protobuf format, the model card can be translated to various outputs including a model report, trade-off comparison and even tests results summary.InstallationThe Model Card Toolkit is hosted onPyPI, and can be installed withpip install verifyml.Getting StartedGenerate a model cardTheVerifyML card creatorprovides an easy way for teams to create and edit model cards in a WYSIWYG editor. Use it to bootstrap your model card or edit text records through a web browser. It is a client side application and no data gets stored on a server.Alternatively, generate a model card with the python toolkit:importverifyml.model_card_toolkitasmctlib# Initialize the Model Card Toolkit with a path to store generate assetsmct=mctlib.ModelCardToolkit(output_dir="model_card_output",file_name="breast_cancer_diagnostic_model_card")model_card=mct.scaffold_assets()Populate the model card with details# You can add free text fieldsmodel_card.model_details.name='Breast Cancer Wisconsin (Diagnostic) Dataset'# Or use helper classesmodel_card.model_parameters.data.append(mctlib.Dataset())model_card.model_parameters.data[0].graphics.description=(f'{len(X_train)}rows with{len(X_train.columns)}features')model_card.model_parameters.data[0].graphics.collection=[mctlib.Graphic(image=mean_radius_train),mctlib.Graphic(image=mean_texture_train)]Save and export to htmlhtml=mct.export_format(output_file="example.html")display.display(display.HTML(html))Model TestsModel tests provides an out of the box way to conduct checks and analysis on performance, explainability and fairness. The tests included in VerifyML are atomic functions that can be imported and run without a model card. However, by using it with a model card, it provides a way to standardize objectives and check for intended or unintended model biases. It also automates documentation and renders the insights to a business friendly report.Currently, VerifyML provides 5 classes of tests:Subgroup Disparity Test- For a given metric, assert that the difference between the best and worst performing group is less than a specified thresholdMin/Max Metric Threshold Test- For a given metric, assert that all groups should be below / above a specified thresholdPerturbation Test- Assert that a given metric does not change significantly after perturbing on a specified input variableFeature Importance Test- Assert that certain specified variables are not included as the top n most important featuresData Shift Test- Assert that the distributions of specified attributes are similar across two given datasets of interestThe detailedmodel tests readmecontains more information on the tests.You can also easily create your own model tests by inheriting from the base model test class. SeeDEVELOPMENTfor more details.Example usagefromverifyml.model_tests.FEATimportSubgroupDisparity# Ratio of false positive rates between age subgroups should not be more than 1.5sgd_test=SubgroupDisparity(metric='fpr',method='ratio',threshold=1.5)sgd_test.run(output)# test data with prediction resultssgd_test.plot(alpha=0.05)Adding the test to the model cardimportverifyml.model_card_toolkitasmctlibmc_sgd_test=mctlib.Test()mc_sgd_test.read_model_test(sgd_test)model_card.fairness_analysis.fairness_reports[0].tests=[mc_smt_test]SchemaModel cards are stored as a protobuf format. The reference model card protobuf schema can be found in theproto directory. A translated copy in json schema format is also made available for convenience in theschema folderTemplatesModel cards can be rendered into various reports through the use of templates. The template folder contains two html templates - a default model report and a compare template, and a default markdown model report.Contributions and DevelopmentContributions are always welcome - check outCONTRIBUTINGThe package and it's functionalities can be easily extended to meet the needs of a team. Check outDEVELOPMENTfor more info.Prior ArtThe model card in VerifyML is adapted from Google'sModel Card Toolkit. It is backward compatible with v0.0.2 and expands on it by adding sections on explainability and fairness. You can specify the desired rendering template by specifying thetemplate_pathargument when calling themct.export_formatfunction. For example:mct.export_format(output_file="example.md",template_path="path_to_my_template")View thetemplates' READMEfor more information on creating your own jinja templates.References[1]https://arxiv.org/abs/1810.03993LicenseVerifyML is licensed under the Apache License, Version 2.0. SeeLICENSEfor the full license text.Generating DocsDocs are generated usingpydoc-markdown, and our configuration is specified inpydoc-markdown.yml. The package reads the yml file, then converts the referenced READMEs and code files into correspondingmkdocsmarkdown files, together with amkdocs.ymlconfig file. These converted files can be found in abuild/docsdirectory, which will appear after the commands below are run.PreviewTo preview the docs locally, run./docs.shserveThis creates doc files inbuild/docs/, then serves them atlocalhost:8000.BuildTo build the HTML files, run./docs.shbuildThis creates doc files inbuild/docs/, then creates their HTML equivalents inbuild/html/.DetailsTo render Jupyter Notebooks in the docs, we use themkdocs-jupyterplugin, and reference the notebooks inpydoc-markdown.yml(e.g.source: example.ipynbin one of the entries).However, becausepydoc-markdownconverts everything to Markdown files by default, only the notebook text would show up by default. Thus, some intermediate steps (/ hacks) are required for the notebook to render correctly:Build the docs, converting the notebook text into a Markdown file (e.g.build/docs/example.md)Rename the built file's extension from Markdown back into a notebook format (e.g.mv example.md example.ipynbin bash)Edit the builtmkdocs.ymlfile such that the notebook's entry points to the renamed file in step 2 (this is done byconvert_md_to_ipynb.py)./docs.shhandles these steps.