Package: akerbp-mlops
MLOps FrameworkThis is a framework for MLOps that deploys models as functions in Cognite Data
FusionUser GuideReference guideThis assumes you are already familiar with the framework, and acts as a quick reference guide for deploying models using the prediction service, i.e. when model training is performed outside of the MLOps framework.Train model to generate model artifactsManually upload artifacts to your test environmentThis includes model artifacts generated during training, mapping- and settings-file for the model, scaler object etc. Basically everything that is needed to preprocess the data and make predictions using the trained model.Deploy prediction service to testThis is handled by the CI/CD pipeline on GitHubManually promote model artifacts from test to productionManually trigger deployment of model to productionTrigger in the CI/CD pipelineCall deployed modelSee section  "Calling a deployed model prediction service hosted in CDF" belowGetting Started:Follow these steps (in the context of your virtual environment):Install package:pip install akerbp-mlops[cdf](On some OSes you may need to escape the brackets by doing sopip install "akerbp-mlops[cdf]")Set up pipeline files.github/workflows/main.ymland config filemlops_settings.yamlby running this command from your repo's root folder:python-makerbp.mlops.deployment.setupFill in user settings and then validate them by running this (from repo root):python-c"from akerbp.mlops.core.config import validate_user_settings; validate_user_settings()"alternatively, run the setup again:python-makerbp.mlops.deployment.setupCommit the pipeline and settings files to your repoBecome familiar with the model template (see foldermodel_code) and make
sure your model follows the same interface and file structure (seeFiles and Folders Structure)A this point every git push in master branch will trigger a deployment in the
test environment. More information about the deployments pipelines is provided
later.Updating MLOpsFollow these steps:Install a new version using pip, e.g.pip install akerbp-mlops[cdf]==x, or upgrade your existing version to the latest release by runningpip install --upgrade akerbp-mlops[cdf]Run this command from your repo's root folder:python-makerbp.mlops.deployment.setupThis will update the GitHub pipeline with the newest release of akerbp.mlops and validate your settings. Once the settings are validated, commit changes and
you're ready to go!General GuidelinesUsers should consider the following general guidelines:Model artifacts shouldnotbe committed to the repo. Foldermodel_artifactdoes store model artifacts for the model defined inmodel_code, but it is
just to help users understand the framework (see this sectionon how to handle model artifacts)Follow the recommended file and folder structure (see this section)There can be several models in your repo: they need to be registered in the
settings, and then they need to have their own model and test filesFollow the import guidelines (see this section)Make sure the prediction service gets access to model artifacts (see this section)ConfigurationMLOps configuration is stored inmlops_settings.yaml. Example for a project
with a single model:model_name:model1human_friendly_model_name:'MyFirstModel'model_file:model_code/model1.pyreq_file:model_code/requirements.modelartifact_folder:model_artifactartifact_version:1# Optionaltest_file:model_code/test_model1.pyplatform:cdfdataset:mlopspython_version:py39helper_models:-my_helper_modelinfo:prediction:&descdescription:'Descriptionpredictionservice,model1'metadata:required_input:-ACS-RDEP-DENtraining_wells:-3/14-2/7-18input_types:-int-float-stringunits:-s/ft-1-kg/m3output_curves:-ACoutput_units:-s/ftpetrel_exposure:Falseimputed:Truenum_filler:-999.15cat_filler:UNKNOWNowner:data@science.comtraining:<<:*descdescription:'Descriptiontrainingservice,model1'metadata:required_input:-ACS-RDEP-DENoutput_curves:-AChyperparameters:learning_rate:1e-3batch_size:100epochs:10FieldDescriptionmodel_namea suitable name for your model. No spaces or dashes are allowedhuman_friendly_model_nameName of function (in CDF)model_filemodel file path relative to the repo's root folder. All required model code should be under the top folder in that path (model_codein the example above).req_filemodel requirement file. Do not use.txtextension!artifact_foldermodel artifact folder. It can be the name of an existing local folder (note that it should not be committed to the repo). In that case it will be used in local deployment. It still needs to be uploaded/promoted with the model manager so that it can be used in Test or Prod. If the folder does not exist locally, the framework will try to create that folder and download the artifacts there. Set tonullif there is no model artifact.artifact_version (optional)artifact version number to use during deployment. Defaults to the latest version if not specifiedtest_filetest file to use. Set tonullfor no testing before deployment (not recommended).platformdeployment platforms, eithercdf(Cognite) orlocalfor local testing.python_versionIfplatformis set tocdf, thepython_versionrequired by the model to be deployed needs to be specified. Available versions can be foundherehelper_modelsArray of helper models using for feature engineering during preprocessing. During deployment, iterate through this list and check that helper model requirements are the same as the main model. For now we only check for akerbp-mlpetdatasetCDF Dataset to use to read/write model artifacts (seeModel Manager). Set tonullis there is no dataset (not recommended).infodescription, metadata and owner information for the prediction and training services. Training field can be discarded if there's no such service.Note:
allpathsshould beunix style, regardless of the platform.Notes on metadata:
We need to specify the metadata under info as a dictionary with strings as keys and values, as CDF only allows strings for now. We are also limited to the followingKeys can contain at most 16 charactersValues can contain at most 512 charactersAt most 16 key-value pairsMaximum size of entire metadata field is 512 bytesIf there are multiple models, model configuration should be separated using---. Example:model_name:model1human_friendly_model_name:'MyFirstModel'model_file:model_code/model1.py(...)---# <- this separates model1 and model2 :)model_name:model2human_friendly_model:'MySecondModel'model_file:model_code/model2.py(...)Files and Folders StructureAll the model code and files should be under a single folder, e.g.model_code.Requiredfiles in this folder:model.py: implements the standard model interfacetest_model.py: tests to verify that the model code is correct and to verify
correct deploymentrequirements.model: libraries needed (with specificversion numbers),
can't be calledrequirements.txt. Add the MLOps framework like this:# requirements.model(...)# your other reqsakerbp-mlops==MLOPS_VERSIONDuring deployment,MLOPS_VERSIONwill be automatically replaced by the
specific versionthat you have installed locally. Make sure you have the latest release on your local machine prior to model deployment.For the prediction service we require the model interface to have the following class and functioninitialization(), with required argumentspath to artifact foldersecretsthese arguments can safely be set to None, and the framework will handle everything under the hood.only set path to artifact folder as None if not using any artifactspredict(), with required argumentsdatainit_object (output from initialization() function)secretsYou can safely put the secrets argument to None, and the framework will handle the secrets under the hood.ModelException class with inheritance from an Exception base classFor the training service we require the model interface to have the following class and functiontrain(), with required argumentsfolder_pathpath to store model artifacts to be consumed by the prediction serviceModelException class with inheritance from an Exception base classThe following structure is recommended for projects with multiple models:model_code/model1/model_code/model2/model_code/common_code/This is because when deploying a model, e.g.model1, the top folder in the
path (model_codein the example above) is copied and deployed, i.e.common_codefolder (assumed to be needed bymodel1) is included. Note thatmodel2folder would also be deployed (this is assumed to be unnecessary but
harmless).Import GuidelinesThe repo's root folder is the base folder when importing. For example, assume
you have these files in the folder with model code:model_code/model.pymodel_code/helper.pymodel_code/data.csvIfmodel.pyneeds to importhelper.py, use:import model_code.helper. Ifmodel.pyneeds to readdata.csv, the right path isos.path.join('model_code', 'data.csv').It's of course possible to import from the Mlops package, e.g. its logger:fromakerbp.mlops.coreimportloggerlogging=logger.get_logger("logger_name")logging.debug("This is a debug log")ServicesWe consider two types of services: prediction and training.Deployed services can be called withfromakerbp.mlops.xx.helpersimportcall_functionoutput=call_function(external_id,data)Wherexxis either'cdf'or'gc', andexternal_idfollows the
structuremodel-service-model_env:model: model name given by the user (settings file)service: eithertrainingorpredictionmodel_env: eitherdev,testorprod(depending on the deployment
environment)The output has a status field (okorerror). If they are 'ok', they have
also apredictionandprediction_fileortrainingfield (depending on the type of service). The
former is determined by thepredictmethod of the model, while the latter
combines artifact metadata and model metadata produced by thetrainfunction.
Prediction services have also amodel_idfield to keep track of which model
was used to predict.See below for more details on how to call prediction services hosted in CDF.Deployment PlatformModel services (described below) can be deployed to CDF, i.e. Cognite Data Fusion or Google Cloud Run. The deployment platform is specified in the settings file.CDF Functions include metadata when they are called. This information can be
used to redeploy a function (specifically, thefile_idfield). Example:importakerbp.mlops.cdf.helpersascdfhuman_readable_name="My model"external_id="my_model-prediction-test"cdf.set_up_cdf_client('deploy')cdf.redeploy_function(human_readable_nameexternal_id,file_id,'Description','your@email.com')Note that the external-id of a function needs to be unique, as this is used to distinguish functions between services and hosting environment.It's possible to query available functions (can be filtered by environment
and/or tags). Example:importakerbp.mlops.cdf.helpersascdfcdf.set_up_cdf_client('deploy')all_functions=cdf.list_functions()test_functions=cdf.list_functions(model_env="test")tag_functions=cdf.list_functions(tags=["well_interpretation"])Functions can be deleted. Example:importakerbp.mlops.cdf.helpersascdfcdf.set_up_cdf_client('deploy')cdf.delete_service("my_model-prediction-test")Functions can be called in parallel. Example:fromakerbp.mlops.cdf.helpersimportcall_function_parallelfunction_name='my_function-prediction-prod'data=[dict(data='data_call_1'),dict(data='data_call_2')]response1,response2=call_function_parallel(function_name,data)#TODO - Document common use cases for GCRModel ManagerModel Manager is the module dedicated to managing the model artifacts used by
prediction services (and generated by training services). This module uses CDF
Files as backend.Model artifacts are versioned and stored together with user-defined metadata.
Uploading a new model increases the version count by 1 for that model and
environment. When deploying a prediction service, the latest model version is
chosen. It would be possible to extend the framework to allow deploying specific
versions or filtering by metadata.Model artifacts are segregated by environment (e.g. only production artifacts
can be deployed to production). Model artifacts have to be uploaded manually to
test (or dev) environment before deployment. Code example:importakerbp.mlops.model_managerasmmmetadata=train(model_dir,secrets)# or define it directlymm.setup()folder_info=mm.upload_new_model_version(model_name,model_env,folder_path,metadata)If there are multiple models, you need to do this one at at time. Note thatmodel_namecorresponds to one of the elements inmodel_namesdefined inmlops_settings.py,model_envis the target environment (where the model should be
available),folder_pathis the local model artifact folder andmetadatais a
dictionary with artifact metadata, e.g. performance, git commit, etc.Model artifacts needs to be promoted to the production environment (i.e. after
they have been deployed successfully to test environment) so that a prediction
service can be deployed in production.# After a model's version has been successfully deployed to testimportakerbp.mlops.model_managerasmmmm.setup()mm.promote_model('model','version')VersioningEach model artifact upload/promotion increments a version number (environment dependent)
available in Model Manager. However, this doesn't modify the model
artifacts used in existing prediction services (i.e. nothing changes in CDF
Functions). To reflect the newly uploaded/promoted model artifacts in the existing services one need to deploy the services again. Note that we dont have to specify the artifact version explicitly if we want to deploy using the latest artifacts, as this is done by default.Recommended process to update a model artifact and prediction service:New model features implemented in a feature branchNew artifact generated and uploaded to test environmentFeature branch merged with masterTest deployment is triggered automatically: prediction service is deployed to
test environment with the latest artifact version (in test)Prediction service in test is verifiedArtifact version is promoted manually from command line whenever suitableProduction deployment is triggered manually from GitHub: prediction
service is deployed to production with the latest artifact version (in prod)It's possible to get an overview of the model artifacts managed by Model
Manager. Some examples (seeget_model_version_overviewdocumentation for other
possible queries):importakerbp.mlops.model_managerasmmmm.setup()# all artifactsfolder_info=mm.get_model_version_overview()# all artifacts for a given modelfolder_info=mm.get_model_version_overview(model_name='xx')If the overview shows model artifacts that are not needed, it is possible to
remove them. For example if artifact "my_model/dev/5" is not needed:model_to_remove="my_model/dev/5"mm.delete_model_version(model_to_remove)Model Manager will by default show information on the artifact to delete and ask
for user confirmation before proceeding. It's possible (but not recommended) to
disable this check. There's no identity check, so it's possible to delete any
model artifact (from other data scientist). Be careful!It's possible to download a model artifact (e.g. to verify its content). For
example:mm.download_model_version('model_name','test','artifact_folder',version=5)If no version is specified, the latest one is downloaded by default.By default, Model Manager assumes artifacts are stored in themlopsdataset.
If your project uses a different one, you need to specify during setup (seesetupfunction).Further information:Model Manager requires specific environmental variables (see next
section) or a suitable secrets to be passed to thesetupfunction.In projects with a training service, you can rely on it to upload a first
version of the model. The first prediction service deployment will fail, but
you can deploy again after the training service has produced a model.When you deploy from the development environment (covered later in this
document), the model artifacts in the settings file can point to existing
local folders. These will then be used for the deployment. Version is then
fixed tomodel_name/dev/1. Note that these artifacts are not uploaded to CDF
Files.Prediction services are deployed with model artifacts (i.e. the artifact is
copied to the project file used to create the CDF Function) so that they are
available at prediction time. Downloading artifacts at run time would require
waiting time, and files written during run time consume ram memory).Model versioningTo allow for model versioning and rolling back to previous model deployments, the external id of the functions (in CDF) includes a version number that is reflected by the latest artifact version number when deploying the function (see above).
Everytime we upload/promote new model artifacts and deploy our services, the version number of the external id of the functions representing the services are incremented (just as the version number for the artifacts).To distinguish the latest model from the remaining model versions, we redeploy the latest model version using a predictable external id that does not contain the version number. By doing so we relieve the clients need of dealing with version numbers, and they will call the latest model by default. For every new deployment, we will thus have two model deployments - one with the version number, and one without the version number in the external id.  However, the predictable external id is persisted across new model versions, so when deploying a new version the latest one, with the predictable external id, is simply overwritten.We are thus concerned with two structures for the external id<model_name>-<service>-<model_env>-<version>for rolling back to previous versions, and<model_name>-<service>-<model_env>for the latest deployed modelFor the latest model with a predictable external id, we tag the description of the model to specify that the model is in fact the latest version, and add the version number to the function metadata.We can now list out multiple models with the same model name and external id prefix, and choose to make predictions and do inference with a specific model version. An example is shown below.# List all prediction services (i.e. models) with name "My Model" hosted in the test environment, and model corresponding to the first element of the listfromakerbp.mlops.cdf.helpersimportget_clientclient=get_client(client_id=<client_id>,client_secret=<client_secret>)my_models=client.functions.list(name="My Model",external_id_prefix="mymodel-prediction-test")my_model_specific_version=my_models[0]Calling a deployed model prediction service hosted in CDFThis section describes how you can call deployed models and obtain predictions for doing inference.
We have two options for calling a function in CDF, either using the MLOps framework directly or by using the Cognite SDK. Independent of how you call your model, you have to pass the data as a dictionary with a key "data" containing a dictionary with your data, where the keys of the inner dictionary specifies the columns, and the values are list of samples for the corresponding columns.First, load your data and transform it to a dictionary as assumed by the framework. Note that the data dictionary you pass to the function might vary based on your model interface. Make sure to align with what you specified in yourmodel.pyinterface.importpandasaspddata=pd.read_csv("path_to_data")input_data=data.drop(columns=[target_variables])data_dict={"data":input_data.to_dict(orient=list),"to_file":True}The "to_file" key of the input data dictionary specifies how the predictions can be extracted downstream. More details are provided belowCalling deployed model using MLOps:Set up a cognite client with sufficient access rightsExtract the response directly by specifying the external id of the model and passing your data as a dictionaryNote that the external id is on the form"<model_name>-<service>-<model_env>-<version>", and"<model_name>-<service>-<model_env>"Use the latter external id if you want to call the latest model. The former external id can be used if you want to call a previous version of your model.fromakerbp.mlops.cdf.helpersimportset_up_cdf_client,call_functionset_up_cdf_client(context="deploy")#access CDF data, files and functions with deploy contextresponse=call_function(function_name="<model_name>-prediction-<model_env>",data=data_dict)Calling deployed model using the Cognite SDK:set up cognite client with sufficient access rightsRetreive model from CDF by specifying the external-id of the modelCall the functionExtract the function call response from the function callfromakerbp.mlops.cdf.helpersimportget_clientclient=get_client(client_id=<client_id>,client_secret=<client_secret>)client=CogniteClient(config=cnf)function=client.functions.retrieve(external_id="<model_name>-prediction-<model_env>")function_call=function.call(data=data_dict)response=function_call.get_response()Depending on how you specified the input dictionary, the predictions are available directly from the response or needs to be extracted from Cognite Files.
If the input data dictionary contains a key "to_file" with value True, the predictions are uploaded to cognite Files, and the 'prediction_file' field in the response will contain a reference to the file containing the predictions. If "to_file" is set to False, or if the input dictionary does not contain such a key-value pair, the predictions are directly available through the function call response.If "to_file" = True, we can extract the predictions using the following code-snippetfile_id=response["prediction_file"]bytes_data=client.files.download_bytes(external_id=file_id)predictions_df=pd.DataFrame.from_dict(json.loads(bytes_data))Otherwise, the predictions are directly accessible from the response as follows.predictions=response["predictions"]Extracting metadata from deployed model in CDFOnce a model is deployed, a user can extract potentially valuable metadata as follows.my_function=client.functions.retrieve(external_id="my_model-prediction-test")metadata=my_function.metadataWhere the metadata corresponds to whatever you specified in the mlops_settings.yaml file. For this example we get the following metadata{'cat_filler': 'UNKNOWN',
 'imputed': 'True',
 'input_types': '[int, float, string]',
 'num_filler': '-999.15',
 'output_curves': '[AC]',
 'output_unit': '[s/ft]',
 'petrel_exposure': 'False',
 'required_input': '[ACS, RDEP, DEN]',
 'training_wells': '[3/1-4]',
 'units': '[s/ft, 1, kg/m3]'}Local Testing and DeploymentIt's possible to tests the functions locally, which can help you debug errors
quickly. This is recommended before a deployment.Define the following environmental variables (e.g. in.bashrc):exportMODEL_ENV=devexportCOGNITE_OIDC_BASE_URL=https://api.cognitedata.comexportCOGNITE_TENANT_ID=<tenantid>exportCOGNITE_CLIENT_ID_WRITE=<writeaccessclientid>exportCOGNITE_CLIENT_SECRET_WRITE=<writeaccessclientsecret>exportCOGNITE_CLIENT_ID_READ=<readaccessclientid>exportCOGNITE_CLIENT_SECRET_READ=<readaccessclientsecret>From your repo's root folder:python -m pytest model_code(replacemodel_codeby your model code folder
name)deploy_prediction_servicedeploy_training_service(if there's a training service)The first one will run your model tests. The last two run model tests but also
the service tests implemented in the framework and simulate deployment.If you want to run tests only you need to setTESTING_ONLY=Truebefore calling the deployment script.Automated Deployments from BitbucketDeployments to the test environment are triggered by commits (you need to push
them). Deployments to the production environment are enabled manually from the
Bitbucket pipeline dashboard. Branches that match 'deploy/*' behave as master.
Branches that matchfeature/*run tests only (i.e. do not deploy).It is assumed that most projects won't include a training service. A branch that
matches 'mlops/*' deploys both prediction and training services. If a project
includes both services, the pipeline file could instead be edited so that master
deployed both services.It is possible to schedule the training service in CDF, and then it can make
sense to schedule the deployment pipeline of the model service (as often as new
models are trained)NOTE: Previous version of akerbp-mlops assumes that callingLOCAL_DEPLOYMENT=True deploy_prediction_servicewill not deploy models and run tests.
The package is now refactored to only trigger tests when the environment variableTESTING_ONLYis set toTrue.
Make sure to update the pipeline definition for branches with prefixfeature/to callTESTING_ONLY=True deploy_prediction_serviceinstead.GitHub SetupThe following environments need to be defined inrepository settings > deployments:dev: where two environment variables are definedMODEL_ENV=devSERVICE_NAME=predictiontest: where two environment variables are definedMODEL_ENV=testSERVICE_NAME=predictionprod: where two environment variables are definedMODEL_ENV=prodSERVICE_NAME=predictioneThe following secrets need to be defined inrepository settings > Secrets and variables > Actions > Repository secrets:COGNITE_CLIENT_ID_WRITECOGNITE_CLIENT_SECRET_WRITECOGNITE_CLIENT_ID_READCOGNITE_CLIENT_SECRET_READCOGNITE_OIDC_BASE_URLCOGNITE_TENANT_ID(these should be CDF client id and secrets for respective read and write access).GitHub Actions need to be enabled on the repo.Developer/Admin GuideThis package is managed usingpoetry. Please refer to thepoetry documentationfor more information on how to use poetry and install itInstallationTo install the package, run the following command from the root folder of the repopoetryinstall-Ecdf--with=dev,pre-commit,version,testPoetry usesgroupsto manage dependencies. The above command installs the package with all the defined groups in the toml file.Package versioningThe versioning of the package followsSemVer, using theMAJOR.MINOR.PATCHstructure. We are thus updating the package version using the following conventionIncrement MAJOR when making incompatible API changesIncrement MINOR when adding backwards compatible functionalityIncrement PATCH when making backwards compatible bug-fixesThe version is updated based on the latest commit to the repo, and we are currently using the following rules.The MAJOR version is incremented if the commit message includes the wordmajorThe MINOR version is incremented if the commit message includes the wordminorThe PATCH number is incremented if neithermajornorminorif found in the commit messageIf the commit message includes the phraseprerelease, the package version is extended witha, thus taking the formMAJOR.MINOR.PATCHa.Note that the above keywords arenotcase sensitive. Moreover,majortakes precedence overminor, so if both keywords are found in the commit message, the MAJOR version is incremented and the MINOR version is kept unchanged.In dev and test environment, we release the package using the pre-release tag, and the package takes the following version numberMAJOR.MINOR.PATCH-alpha.PRERELEASE.The version number is automatically generated by combiningpoetry-dynamic-versioningwith theincrement_package_version.pyscript and is based off git tagging and the incremental version numbering system mentioned above.MLOps Files and FoldersThese are the files and folders in the MLOps repo:srccontains the MLOps framework packagemlops_settings.yamlcontains the user settings for the dummy modelmodel_codeis a model template included to show the model interface. It is
not needed by the framework, but it is recommended to become familiar with it.model_artifactstores the artifacts for the model shown inmodel_code.
This is to help to test the model and learn the framework..github/*describes all the relevant configurations for the CI/CD pipeline run by GitHub Actionsbuild.shis the script to build and upload the packagepyproject.tomlis the project's configuration fileLICENSEis the package's licenseCDF DatasetsIn order to control access to the artifacts:Set up a CDF Dataset withwrite_protected=Trueand aexternal_id, which
by default is expected to bemlops.Create a group of owners (CDF Dashboard), i.e. those that should have write
accessLocal Testing (only implemented for the prediction service)To perform local testing of before pushing to GITHUB, you can run the following
commands:poetryrunpython-mpytest(assuming you have first runpoetry install -E cdf --with=dev,pre-commit,version,test"in the same environment)Build and Upload PackageCreate an account in pypi, then create a token and a$HOME/.pypircfile if you want to deploy from local. Editpyproject.tomlfile and note the following:Dependencies need to be registeredBash scripts will be installed in abinfolder in thePATH.The pipeline is setup to build the library, but it's possible to
build and upload the library from the development environment as well (as long as you have thePYPI_TOKENenvironment variable set). To do so, run:bashbuild.shIn order to authenticate to GitHub to deploy to pypi you need to setup a token. Copy its content and add that to the secured repository secretPYPI_TOKEN.Notes on the codeService testing happens in an independent process (subprocess library) to avoid
setup problems:When deploying multiple models the service had to be reloaded before testing
it, otherwise it would be the first model's service. Model initialization in
the prediction service is designed to load artifacts only once in the processIf the model and the MLOps framework rely on different versions of the same
library, the version would be changed during runtime, but the
upgraded/downgraded version would not be available for the current process