Package: veritastool
Veritas ToolkitThe purpose of this toolkit is to facilitate the adoption of Veritas Methodology on Fairness & Transparency Assessment and spur industry development. It will also
benefit customers by improving the fairness and transparency of financial services delivered by AIDA systems.InstallationThe easiest way to install veritastool is to download it fromPyPI. It's going to install the library itself and its prerequisites as well. It is suggested to create virtual environment with requirements.txt file first.pipinstallveritastoolThen, you will be able to import the library and use its functionalities. Before we do that, we can run a test function on our sample datasets to see if our codes are performing as expected.fromveritastool.util.utilityimporttest_function_cstest_function_cs()Output:InitializationYou can now import the custom library that you would to use for diagnosis. In this example we will use the Credit Scoring custom library.fromveritastool.model.modelwrapperimportModelWrapperfromveritastool.model.model_containerimportModelContainerfromveritastool.usecases.credit_scoringimportCreditScoringOnce the relevant use case object (CreditScoring) and model container (ModelContainer) has been imported, you can upload your contents into the container and initialize the object for diagnosis.importpickleimportnumpyasnp#Load Credit Scoring Test Data# NOTE: Assume current working directory is the root folder of the cloned veritastool repositoryfile="./veritastool/examples/data/credit_score_dict.pickle"input_file=open(file,"rb")cs=pickle.load(input_file)#Model Contariner Parametersy_true=np.array(cs["y_test"])y_pred=np.array(cs["y_pred"])y_train=np.array(cs["y_train"])p_grp={'SEX':[1],'MARRIAGE':[1]}up_grp={'SEX':[2],'MARRIAGE':[2]}x_train=cs["X_train"]x_test=cs["X_test"]model_name="credit_scoring"model_type="classification"y_prob=cs["y_prob"]model_obj=LogisticRegression(C=0.1)model_obj.fit(x_train,y_train)#fit the model as required for transparency analysis#Create Model Containercontainer=ModelContainer(y_true,p_grp,model_type,model_name,y_pred,y_prob,y_train,x_train=x_train,\x_test=x_test,model_object=model_obj,up_grp=up_grp)#Create Use Case Objectcre_sco_obj=CreditScoring(model_params=[container],fair_threshold=80,fair_concern="eligible",\fair_priority="benefit",fair_impact="normal",perf_metric_name="accuracy",\tran_row_num=[20,40],tran_max_sample=1000,tran_pdp_feature=['LIMIT_BAL'],tran_max_display=10)API functionsBelow are the API functions that the user can execute to obtain the fairness and transparency diagnosis of their use cases.EvaluateThe evaluate API function computes all performance and fairness metrics and renders it in a table format (default). It
also highlights the primary performance and fairness metrics (automatic if not specified by user).cre_sco_obj.evaluate()Output:You can also toggle the widget to view your results in a interactive visualization format.cre_sco_obj.evaluate(visualize=True)Output:TradeoffComputes trade-off between performance and fairness.cre_sco_obj.tradeoff()Output:** Note: Replace {Balanced Accuracy} with the respective given metrics.Feature ImportanceComputes feature importance of protected features using leave one out analysis.cre_sco_obj.feature_importance()Output:Root CauseComputes the importance of variables contributing to the bias.cre_sco_obj.root_cause()Output:MitigateUser can choose methods to mitigate the bias.mitigated=cre_sco_obj.mitigate(p_var=[],method=['reweigh','correlation','threshold'])Output:ExplainRuns the transparency analysis - global & local interpretability, partial dependence analysis and permutation importance#run the entire transparency analysiscre_sco_obj.explain()Output:#get the local interpretability plot for specific row index and modelcre_sco_obj.explain(local_row_num=20)Output:CompileGenerates model artifact file in JSON format. This function also runs all the API functions if it hasn't already been run.cre_sco_obj.compile()Output:Model ArtifactA JSON file that stores all the results from all the APIs.Output:ExamplesYou may refer to our example notebooks below to see how the toolkit can be applied:FilenameDescriptionCS_Demo.ipynbTutorial notebook to diagnose a credit scoring model for predicting customers' loan repayment.CM_Demo.ipynbTutorial notebook to diagnose a customer marketing uplift model for selecting existing customers for a marketing call to increase the sales of loan product.BaseClassification_demo.ipynbTutorial notebook for a multi-class propensity modelBaseRegression_demo.ipynbTutorial notebook for a prediciton of a continuous target variablePUW_demo.ipynbTutorial notebook for a binary classification model to predict whether to award insurance policy by assessing riskNewUseCaseCreation_demo.ipynbTutorial notebook to create a new use case note-book and add custom metricsnonPythonModel_customMetric_demo.ipynbTutorial notebook to diagnose a credit scoring model by LibSVM (non-Python) with custom metric.LicenseVeritas Toolkit is licensed under the Apache License, Version 2.0 - seeLICENSEfor more details.