Package: mlmonitor
ML Model Lifecycle with mlmonitormlmonitor python module is designed to help you:Operatea curated collection of model use cases, running on AWS Sagemaker,Azure and Watson Machine Learning, reporting metadata to AI Factsheets and monitored in IBM Watson OpenScale.Experiencean end-to-end model metadata capture from development to model operationalization states.Acceleratethe setup time of monitors such as quality, input and output drift, fairness, explainability and custom metrics.Rundrift and fairness demo scenarios to better understand algorithms supported by Watson OpenScale (e.g.KS tests,bias mitigation algorithms,local post-hoc explanation).0. Prerequisites0.1 install mlmonitor package$pipinstallmlmonitorThis command will installmlmonitorwith its dependencies such asibm-aigov-facts-clientwhich is the AI factsheets client library built from thisrepo.Documentation is availablehereNote :latest releases of this client can be installed from test PiPy serverhttps://test.pypi.org/simple/gitstat0.2 SetMONITOR_CONFIG_FILEenvironment variablepopulatecredentials.cfgfile , see section 2 for more details .Once this file is populated setMONITOR_CONFIG_FILEenvironment variable as follow :$exportMONITOR_CONFIG_FILE=<GitHub-directory>/credentials.cfgyou are now setup to usemlmonitor1. List of model use cases supportedTaskdirectoryCommenttrain scriptinference scriptCustomer churn prediction (xgboost binary classification)use_case_churncompleted e2etrain_cc_xg_boost.pyinference_cc_xg_boost.pyGerman Credit Risk (scikit binary classification)use_case_gcrcompleted e2etrain_gcr.pyinference_aws_gcr.pyHandwritten digit detection Kerasuse_case_mnist_tfcompleted e2etf_cnn_train.pytf_cnn_inference.pHandwritten digit detection Pytorchuse_case_mnist_pttrain onlypytorch_train.pypytorch_inference.pyHandwritten digit detection Pytorch Lightninguse_case_mnist_ptlttrain onlyptlt_train.pyptlt_inference.py2. configuration ofmlmonitorlib details2.1 Option 1 with configuration fileUpdatecredentials.cfgfile with your IBM API keys 🔑 , AWS credentials , COS details and OpenScale instance id...saassection is required For IBM Cloud environmentpremsection is required For Cloud Pak for Data running on OCP environment{"saas":{"apikey":"xxxxx","wml_url":"https://<cloud region>.ml.cloud.ibm.com","wos_instance_id":"xxxxxxxxxxxxxxxxx","default_space":"xxxxx","cos_resource_crn":"xxxxx","cos_endpoint":"https://s3.<cloud region>.cloud-object-storage.appdomain.cloud","bucket_name":"xxxxx"},"prem":{"version":"4.7","username":"","apikey":"xxxxx","wos_instance_id":"00000000-0000-0000-0000-000000000000","wml_instance_id":"openshift","default_space":"xxxxx","ibm_auth_endpoint":"xxxxxxxxxxxxxxxxx"},"aws_credentials":{"access_key":"XXXXXXXXXXXXXXXXXXXXX","secret_key":"XXXXXXXXXXXXXXXXXXXXX","region_name":"XXXXXXXX"},"azure":{"client_id":"xxxxx","client_secret":"xxxxx","subscription_id":"xxxxx","tenant_id":"xxxxx","resource_group":"xxxxx","workspace_name":"xxxxx"}}SetENVenvironment variable to 'saas' or 'prem'SetMONITOR_CONFIG_FILEenvironment variable with the complete path for this configuration file2.1.1 SaaS environment ('saas' section)saassection must be filled if services are running  on IBM Cloud with mandatory fields :apikey: IBM Cloud API key to instantiate service instances for Watson Machine Learning and Watson OpenScaledefault_space: deployment space to be used for WML models and custom metrics providers for custom monitorswml_url: URL of Watson Machine Learning service (region specific)cos_resource_crn: Cloud Resource Name for Cloud object storage to be used for storing  WML and WOS model traning data references.cos_endpoint: Cloud object storage url (region specific)https://s3.us-east.cloud-object-storage.appdomain.cloudbucket_name: Bucket name to be used for traning data reference uploads.wos_instance_idis optional but recommended to identify the SaaS service instance to be used.wos_urlis optional and set by default tohttps://api.aiopenscale.cloud.ibm.com2.1.2 On prem environment ('prem' section)premsection must be filled if services are running  on Cloud Pak for Data on OCP (CP4D >= 4.7 is required):apikey: Cloud Pak for Data API key to instantiate service instances for Watson Machine Learning and Watson OpenScaledefault_space: deployment space to be used for WML models and custom metrics providers for custom monitorsversion: Cloud Pak for Data versionusername: Cloud Pak for Data username to be used to clientsibm_auth_endpoint: Cloud Pak for Data urlwos_instance_idis optional but recommended to identify the Watson OpenScale instance defaut value is set to "00000000-0000-0000-0000-000000000000"wml_instance_idis optional default value is set to "openScale"2.1.3 Sagemaker credentialsawssection must be filled if models should be trained or deployed in Sagemakeraccess_key: AWS access key with Sagemaker accesssecret_key: AWS secret key with Sagemaker accessregion_name: AWS regionrole: Sagemaker execution role2.2 Option 2 with environment variablesAlternatively you need to setup environment variables listed here.API_KEYAUTH_ENDPOINTAWS_ACCESS_KEY_IDAWS_SECRET_ACCESS_KEYAWS_DEFAULT_REGIONAWS_ROLECOS_RESOURCE_CRNCOS_ENDPOINTBUCKET_NAMEWOS_URLWOS_INSTANCE_IDUSERNAMEVERSIONWML_INSTANCE_IDWML_SPACE_ID3. How to use itfor more details , refer to Readme of each model use case e.guse_case_churn(venv-mlmonitor)$python
>>>frommlmonitorimportSageMakerModelUseCase
>>>source_dir='use_case_churn'>>>catalog_id=<catalogid>
>>>model_entry_id=<Modelusecaseidforchurnmodels>
>>>model_use_case=SageMakerModelUseCase(source_dir,catalog_id,model_entry_id)>>>model_use_case.train()>>>model_use_case.display_states()-trained:[True]-deployed:[False]-governed:[False]-monitored:[False]4. Onboard a new ML model using mlmonitorEach model use case should be placed under a folder following this naming conventionuse_case_<use case name>.Please refer to documentation of each model use case to deploy and monitor a model usingmlmonitorGerman Credit RiskReadmehas detailed instructions.4.1 AWS code repositoriesThey should contain :├── use_case_<use case name>               <- ALL THIS CODE DIRECTORY is SHIPPED AND EXECUTED IN AWS SAGEMAKER
│   ├── __init__.py    <- Makes use_case a Python module
│   ├── train.py       <- Train script to be executed in SM
│   ├── inference.py   <-  Inference script to be executed in SM
│   ├── test_inference.py <- test inference scripts on locally trained model before deploying to AWS
│   ├── test_train.py <- test training scripts before sending training job to AWS Sagemaker
│   └── model_signature.json <- all the model details to onboard this model in WOS and AWS SM
│   └── requirements.txt <- all these dependencies (included Factsheets client) to be installed on AWS containerThese directories are shipped to AWS at training or deployment time and should contain custom training script(s) , custom inference script(s) and dependencies.4.1 model signature filesEach model use case should be self-contained and includemodel_signature.jsonfile following the structure{"signature":{"feature_columns":[],"class_label":"","prediction_field":"","probability_fields":[],"categorical_columns":[],"problem_type":"","data_type":"structured","description":"description"},"datasets":{"training_data":"","validation_data":"","test_data":""},"aws_runtime":{"train_script":"train_cc_xg_boost.py","inference_script":"inference_cc_sk.py","train_framework":"xgboost","train_framework_version":"1.5-1","train_py_version":"py3","inference_framework":"sklearn","inference_framework_version":"1.0-1","train_instance":"ml.m4.xlarge","inference_instance":"ml.m4.xlarge","inference_py_version":"py3","prefix":"sagemaker/DEMO-xgboost-churn","job_name":"sm-cc-xgboost","serializer":"json","deserializer":"json"},"wml_runtime":{"train_module":"train_cc_xg_boost","train_method":"train_wml","inference_instance":"runtime-22.2-py3.10","inference_framework":"scikit-learn","inference_framework_version":"1.1","inference_script":"inference_cc_sk.py"},"azure_runtime":{"train_script":"train_cc_xg_boost.py","train_py_version":"3.8","inference_script":"inference_cc_sk.py","inference_compute":"aci","aks_cluster_name":"aks-cluster","cpu_cores":1,"memory_gb":1,"auth_enabled":false,"description":"Customer Churn prediction - monitored in WOS","tags":{"data":"customer churn","method":"xgboost"},"conda_packages":["pandas==1.5.2","boto3","seaborn","matplotlib"],"pip_packages":["ibm-aigov-facts-client==1.0.67","xgboost==1.6.1","scikit-learn==1.0.1","ibm_watson_openscale==3.0.27","pygit2"],"inference_py_version":"3.9","train_module":"train_cc_xg_boost","train_method":"train_wml"},"hyperparameters":{"max_depth":5,"eta":0.2,"gamma":4,"min_child_weight":6,"subsample":0.8,"objective":"binary:logistic","num_round":200,"verbosity":0},"quality_monitor":{"enabled":true,"parameters":{"min_feedback_data_size":10},"thresholds":[{"metric_id":"area_under_roc","type":"lower_limit","value":0.80}]},"fairness_monitor":{"enabled":true,"parameters":{"features":[{"feature":"Day Mins","majority":[[2.501,5.330],[5.331,7.936],[7.937,20]],"minority":[[0.000,2.500]],"threshold":0.95}],"favourable_class":[0],"unfavourable_class":[1],"min_records":100}},"drift_monitor":{"enabled":true,"parameters":{"min_samples":100,"drift_threshold":0.1,"train_drift_model":false,"enable_model_drift":true,"enable_data_drift":true},"learn_constraints":{"two_column_learner_limit":200,"categorical_unique_threshold":0.8,"user_overrides":[]}},"explain_monitor":{"enabled":true}"mrm_monitor":{"enabled":true}"custom_monitor":{"enabled":true,"names":["tp","fp","tn","fn","cost","total"],"thresholds":[200,10,200,10,6000,200],"provider_name":"Custom_Metrics_Provider_churn","custom_monitor_name":"Custom_Metrics_Provider_churn","wml_function_provider":"Custom_Metrics_Provider_Deployment_churn-deploy"}}4.2 model perturbation filesSimilarly tomodel_signature.json, each model use case should includemodel_perturbation.jsonfile following the structure{"drift":{"single_column_1":{"total_records":100,"ratios":[0.01,0.05,0.1,0.2,0.3],"target_column":"LoanAmount","perturbation_fn":"x + 15000"},"single_column_2":{"total_records":100,"ratios":[0.1,0.2,0.4,0.8,1.0],"target_column":"LoanAmount","perturbation_fn":"x + 15000"},"double_column_1":{"total_records":100,"ratios":[0.1,0.2,0.3,0.6],"source_column":"LoanPurpose","source_cond":"car_used","target_column":"LoanAmount","perturbation_fn":"x + np.mean(x)*100"}}}The JSON file uses the following nomenclature:{
    <monitor type>: {
        <scenario ID>: {
            <scenario parameters>: <parameters values>
        }
    }
}In eachscenario_id, the following parameters can be used:total_records: number of records sent to Watson OpenScale in each iterationratios: a list of percentages used to iterate over. In each iteration, the percentage defines the ratio of records to perturbtarget_column: column where to apply the perturbationperturbation_fn: perturbation function applied on the target columnsource_column: for two-column constraints, column used to filter the datasource_condition: for two-column constraints, filter condition for the source columnCurrently, there are two ways to apply drift on the payload data: single column perturbation or two-column perturbation. Ifsource_columnandsource_condare defined in the scenario, two-column perturbation will be applied.These parameters are then used by theModelPerturbatorobject to perturb the payload data sent to Watson OpenScale.