Package: cortex-serving-client
Cortex Serving ClientWarning: Cortex Labs joined Databricks and maintenance may end. Consider migrating to other tools.Cortex Serving Client makes Python serving automation simple.
It is a Python wrapper aroundCortex's command-line clientthat provides garbage API collection.
Cortex hasofficial Python client now(source), but this project offers advanced features (GC, temporary deployment, timeouts) not present in the vanilla.Main feature of this package is that you can use it on top of your codebase created for Cortex Version <= 0.34, meaning that:deployment directory is automatically zipped and uploaded to S3 bucketwe prepared a base docker image that downloads this zipped code, unzips it and runs it in an Uvicorn worker=> you can simply deploy yourPythonPredictorusing Cortex 0.42 without having to wrap it inside your own docker imageAdditional features:Automate your Cortex AWS cluster from Python.Prevent accidental charges by auto-removing deployments that exceeded a timeout.Execute operations: deploy, delete, get, get all.Stream remote logs into the local log with thread name set to the API name.Supported Cortex Version: 0.40.0 (See requirements.txt)Here isa video about the package (version for Cortex 0.33 = before the big changes).How Does It Work?After implementing your predictor module in a folder (seeexample/dummy_dir),
you can deploy it to your Cortex cluster,
and execute a prediction via a POST request.Here isa video of the demo below.Working ExampleBelow is a snippet fromexample.py:The deployment dict has these additional fields compared to Cortex docs:"project_name":<string>in deployment rootname of the project, zipped source code is going to be uploaded to S3 path:<project_name>/<api_name>.zippredictor_path: Module containing your predictor:cls.__module__= e.g.predictors.my_predictorOptionalpredictor_class_name:cls.__name__of your predictor class, default isPythonPredictor"config":<dict>incontainerspecificationconfig dict that will be saved topredictor_config.jsonin the root of deployment dirthis file can then be loaded inmain.pyand passed to the PythonPredictor constructor = can be seen inresources/main.pydeployment={"name":"dummy-a","project_name":"test","kind":"RealtimeAPI","predictor_path":"dummy_predictor","pod":{"containers":[{"config":{"geo":"cz","model_name":"CoolModel","version":"000000-000000"},"env":{"SECRET_ENV_VAR":"secret",},"compute":{"cpu":'200m',"mem":f"{0.1}Gi"},}],},}# Deploywithcortex.deploy_temporarily(deployment,deploy_dir="dummy_dir",api_timeout_sec=30*60,verbose=True,)asget_result:# Predictresponse=post(get_result.endpoint,json={}).json()Required changes for projects using Cortex version <= 0.34optionally addmain.pyto the root of your cortex deployment folderif there is nomain.pyin the root of the deployment folder, the default one fromresources/main.pywill be usedrestructure your deployment dict to look like the one inexample.pyGarbage API CollectionGarbage API collection auto-removes forgotten APIs to reduce costs.Each deployed API has a timeout period configured during deployment after which it definitely should not exist in the cluster anymore.
This timeout is stored in a Postgres database table.
Cortex client periodically checks currently deployed APIs and removes expired APIs from the cluster.Can You Rollback?How do you deal with new model failure in production?
Do you have the ability to return to your model's previous working version?
There is no generic solution for everybody.
But you can implement the best suiting your needs using the Python API for Cortex.
Having a plan B is a good idea.Our Use CaseWe use this project to automate deployment to auto-scalable AWS instances.
The deployment management is part of application-specific Flask applications,
which call to Python-Cortex-Serving-Client to command environment-dedicated Cortex cluster.In cases where multiple environments share a single cluster, a shared Cortex database Postgres instance is required.Read more about our use case inCortex Client release blog post.
Or you canwatch a video about our use case.Get StartedThis tutorial will help you to getthe basic examplerunning under 15 minutes.Pre-requisitesLinux OSDockerPostgresSetup DatabaseFollow instructions below to configure local database,
or configure cluster database,
and re-configure db inthe example script.sudosupostgres;psqlpostgrespostgres;createdatabasecortex_test;createrolecortex_testloginpassword'cortex_test';grantallprivilegesondatabasecortex_testtocortex_test;You may need to configure alsovi/etc/postgresql/11/main/pg_hba.conf# change a matching line into following to allow localhost network access# host    all             all             127.0.0.1/32            trustsudosystemctlrestartpostgresql;Install CortexSupportedCortex.devversion is a Python dependency version installed throughrequirements.txt.Cortex requires havingDockerinstalled on your machine.Deploy Your First ModelThe deployment and prediction example resides inthe example script.
Make sure you have created a virtual environment, and installed requirements inrequirements.txtandrequirements-dev.txt,
before you execute it.Contact UsSubmit an issueor a pull request if you have any problems or need an extra feature.