Package: accelo
Acceldata ML Observability SDKThe SDK helps Data organizations track their ML models, data that deliver business value.Pre-requisitesRegistering yourself with Acceldata Data Observability Cloud platformDriven through Acceldata Cloud PlatformEnabling ML Observability toolkitDriven through Acceldata Cloud PlatformGenerating API keysDriven through Acceldata ML Observability UISetting up env varsexportCLOUD_ACCESS_KEY=XXXX0000exportCLOUD_SECRET_KEY=XXXX0000exportACCELO_API_ACCESS_KEY=XXXX0000exportACCELO_API_SECRET_KEY=XXXX0000exportACCELO_API_ENDPOINT=https://some_acceldata_endpointInstall the SDKpipinstallacceloSet Go!Sample Usage PatternsBefore we delve into code, let's just see an example of a pattern in which you can use the SDK.Project CreationModesUI - Users will be able to create projects via the Catalog UI where they can either have a model view or a project viewAPI - Users can create a project in their training pipeline. If a project already exists, API throws a custom error that can be used to avoid any failures in the training pipelineModel Registration and Baseline logging (training pipeline)User registers a model against a projectModel registration API expects the project id, model name and bunch of other metadata that can be used to track models on the catalog UIPrediction logging (serving pipeline)The serving pipeline can be used to log the predictions to Acceldata datastoreThe API expects model id, model version, and predictions along with their id columns as mandatory params.Actual logging (actuals pipeline)The actuals for any features may arrive at a later point and the API provides 2 ways to log the actuals.UUIDs: generated by the API during the serving pipeline stage; but the users are expected to keep track of them and map them to the appropriate actualsID COLUMNS: If users specify certain columns to considered as the ID’s, the API will be able to automatically log the actuals against the API’s and the backend services will be able to compare the actuals to predictions based on these ID COLUMNSNote: Please refer to the API documentation for more information.Basic APIsFinally, let's see how you can annotate the SDK into your production code pipelines. Below are some examples of how a Data Scientist or ML Engineer can annotate the SDK into the
existing ML code and observe them using Acceldata ML Observability platform.Import the libraryfromaccelo_mlopsimportAcceloClientCreating a client with a workspaceThe workspace is the top level name that you would want to associate your organization with.
This can also be thought of like a tenant name.client=AcceloClient(workspace='your_organization_name')Creating a ProjectNow, when it comes to code, the atomic unit is aProject. The project name can be a team name, domain name within
a company or any other logical separation Data Science groups.client.create_project(name='marketing-team',description='All models related to the marketing team reside here. ')Register a ModelNow, assuming that you have developed a model that you want to observe using the Acceldata ML Observability platform.
The model object is calledclassifier.model_metadata={'frequency':'DAILY','model_type':'binary_classification','performance_metric':'f1_score','model_obj':classifier}additional_params={'owner':'research@preview.com','last_trained':'2021-08-01','training_job_name':'click_prediction_ml_pipeline','label':'flower_type','total_consumers':2}client.register_model(project_id=12,model_name='click_prediction_model',model_version='v1',model_metadata=model_metadata,**additional_params)Let's see what above variables mean.classifier: this is the model objectmodel_meatadata: this is a mandatory dictionary users have to pass to the register model call to make most use of the ML observability platform.additional_params: this is a optional dictionary users can use to log any additional details about the model which might be useful when viewed in the ML Catalog.Now, it's time to log the data that was used in model.Log baseline dataclient.log_baseline(model_id=client.model_id,model_version='v1',baseline_data=X_train,labels=y_train,label_name='click',id_cols=['campaign_id'],publish_date='2021-08-02')This API call logs your baseline data to Acceldata data store and will be further used for analysis that you sign up for.Log predictionsids=client.log_predictions(model_id=client.model_id,model_version='v1',feature_data=feature_data,predictions=preds,publish_date='2021-06-02')Note: As of now, we support batch predictions only but soon enough, will be able to support logging online
predictions.Log actualsAt a later time, when actuals arrive, you'd be able to log them using below API.client.log_actuals(model_id=client.model_id,model_version='v1',id_cols_df=id_columns_frane,actuals=y_test,publish_date='2021-06-03')You are now done logging both metadata and the data itself.Detailed activity logs can be viewed in thead-mlops.logfile in the directory where your code file exists, however, location of the log file is configurable.What happens after you create a project and register a model?MetadataThe model and the other metadata are now part of Acceldata ML Catalog and can be viewed on the UI.DataThebaseline, prediction, actualdata are logged into the Acceldata Store. This data will be used for further analysis.DashboardYou will be able to track model performance, data drifts, etc by visiting this dashboard.AlertsYou can set alerts on charts, generate reports, etc using the dashboard or the catalog.Contact UsPlease get in touch with us atcontact@acceldata.iofor access to Acceldata catalog, dashboard, and assistance with bringing ML Observability into your organization.