Package: mlserver
MLServerAn open source inference server for your machine learning models.OverviewMLServer aims to provide an easy way to start serving your machine learning
models through a REST and gRPC interface, fully compliant withKFServing's V2
Dataplanespec. Watch a quick video introducing the projecthere.Multi-model serving, letting users run multiple models within the same
process.Ability to runinference in parallel for vertical
scalingacross multiple models through a pool of inference workers.Support foradaptive
batching,
to group inference requests together on the fly.Scalability with deployment in Kubernetes native frameworks, includingSeldon CoreandKServe (formerly known as KFServing), where
MLServer is the core Python inference server used to serve machine learning
models.Support for the standardV2 Inference Protocolon
both the gRPC and REST flavours, which has been standardised and adopted by
various model serving frameworks.You can read more about the goals of this project on theinitial design
document.UsageYou can install themlserverpackage running:pipinstallmlserverNote that to use any of the optionalinference runtimes,
you'll need to install the relevant package.
For example, to serve ascikit-learnmodel, you would need to install themlserver-sklearnpackage:pipinstallmlserver-sklearnFor further information on how to use MLServer, you can check any of theavailable examples.Inference RuntimesInference runtimes allow you to define how your model should be used within
MLServer.
You can think of them as thebackend gluebetween MLServer and your machine
learning framework of choice.
You can read more aboutinference runtimes in their documentation
page.Out of the box, MLServer comes with a set of pre-packaged runtimes which let
you interact with a subset of common frameworks.
This allows you to start serving models saved in these frameworks straight
away.
However, it's also possible towrite custom
runtimes.Out of the box, MLServer provides support for:FrameworkSupportedDocumentationScikit-Learn✅MLServer SKLearnXGBoost✅MLServer XGBoostSpark MLlib✅MLServer MLlibLightGBM✅MLServer LightGBMCatBoost✅MLServer CatBoostTempo✅github.com/SeldonIO/tempoMLflow✅MLServer MLflowAlibi-Detect✅MLServer Alibi DetectAlibi-Explain✅MLServer Alibi ExplainHuggingFace✅MLServer HuggingFaceExamplesTo see MLServer in action, check outour full list of
examples.
You can find below a few selected examples showcasing how you can leverage
MLServer to start serving your machine learning models.Serving ascikit-learnmodelServing axgboostmodelServing alightgbmmodelServing acatboostmodelServing atempopipelineServing a custom modelServing analibi-detectmodelServing aHuggingFacemodelMulti-Model Serving with multiple frameworksLoading / unloading models from a model repositoryDeveloper GuideVersioningBoth the mainmlserverpackage and theinference runtimes
packagestry to follow the same versioning schema.
To bump the version across all of them, you can use the./hack/update-version.shscript.For example:./hack/update-version.sh0.2.0.dev1TestingTo run all of the tests for MLServer and the runtimes, use:maketestTo run run tests for a single file, use something like:tox-epy3--tests/batch_processing/test_rest.py