Package: chantilly
chantillyis a deployment tool foronline machine learningmodels. It is designed to work hand in hand withriver.Table of contentsTable of contentsIntroductionInstallationUser guideRunning the serverPicking a flavorUploading a modelMaking a predictionUpdating the modelMonitoring metricsMonitoring eventsVisual monitoringUsage statisticsUsing multiple modelsConfiguration handlingUsing a different storage backendRedisImporting librariesDeploymentExamplesDevelopmentRoadmapTechnical stackSimilar alternativesLicenseIntroductionThere are many tools for deploying machine learning models. However, none of them supportonline models that can learn on the fly, butchantillydoes.Here are some advantages:Simple:chantillyis essentially just a Flask app.Straightforward: you provide a model,chantillyprovides a few API routes to do the rest.Convenient:chantillyrids you of the burden of storing features between prediction and learning steps.Flexible: use any model you wish, as long as it's written in Python and implements a couple of required methods.Note thatchantillyis very young, and is therefore subject to evolve. We're also eager for feedback and are happy to work hand in hand with you if you have specific needs.Installationchantillyis intended to work withPython 3.7 or above. You can install it from PyPI:>pipinstallchantillyYou can also install the latest development version as so:>pipinstallgit+https://github.com/online-ml/chantillyUser guideRunning the serverOnce you've followed the installation step, you'll get access to thechantillyCLI. You can see the available commands by runningchantilly --help. You can start a server with theruncommand:>chantillyrunThis will start aFlaskserver with all the necessary routes for uploading a model, training it, making predictions with it, and monitoring it. By default, the server will be accessible atlocalhost:5000, which is what we will be using in the rest of the examples in the user guide. You can runchantilly routesin order to see all the available routes.Picking a flavorThe first thing you need to do is pick a flavor. Currently, the available flavors are:regressionfor regression tasks.binaryfor binary classification tasks.multiclassfor multi-class classification tasks.You can set the flavor by sending a POST request to@/api/init, as so:importrequestsconfig={'flavor':'regression'}requests.post('http://localhost:5000/api/init',json=config)You can also set the flavor via the CLI:>chantillyinitregression:warning: Setting the flavor will erase everything and thus provide a clean slate.You can view the current flavor by sending a GET request to@/api/init:r=requests.get('http://localhost:5000/api/init')print(r.json()['flavor'])Uploading a modelYou can upload a model by sending a POST request to the@/api/modelroute. You need to provide a model which has been serialized withpickleordill(we recommend the latter). For example:fromriverimportcomposefromriverimportlinear_modelfromriverimportpreprocessingimportdillimportrequestsmodel=compose.Pipeline(preprocessing.StandardScaler(),linear_model.LinearRegression())requests.post('http://localhost:5000/api/model',data=dill.dumps(model))Likewise, the model can be retrieved by sending a GET request to@/api/model:r=requests.get('http://localhost:5000/api/model')model=pickle.loads(r.content)Note thatchantillywill validate the model you provide to make sure it works with the flavor you picked. For instance, if you picked theregressionflavor, then the model has to implementfit_oneandpredict_one.You can also add a upload by using the CLI. First, you need to serialize a model and dump it to a file:withopen('model.pkl','wb')asfile:dill.dump(model,file)Then, call theadd-modelsub-command:>chantillyadd-modelmodel.pklMaking a predictionPredictions can be obtained by sending a POST request to@/api/predict. The payload you send has to contain a field namedfeatures. The value of this field will be passed to thepredict_onemethod of the model you uploaded earlier on. If the model you providedpredict_proba_onethen that will be used instead. Here is an example:importrequestsr=requests.post('http://localhost:5000/api/predict',json={'id':42,'features':{'shop':'Ikea','item':'Dombäs','date':'2020-03-22T10:42:29Z'}})print(r.json()['prediction'])Note that in the previous snippet we've also provided anidfield. This field is optional. If is is provided, then the features will be stored by thechantillyserver, along with the prediction. This allows not having to provide the features again when you want to update the model later on.Updating the modelThe model can be updated by sending a POST request to@/api/learn. If you've provided an ID in an earlier call to@/api/predict, then you only have to provide said ID along with the ground truth:importrequestsrequests.post('http://localhost:5000/api/learn',json={'id':42,'ground_truth':10.21})However, if you haven't passed an ID earlier on, then you have to provide the features yourself:requests.post('http://localhost:5000/api/learn',json={'features':{'shop':'Ikea','item':'Dombäs','date':'2020-03-22T10:42:29Z'},'ground_truth':10.21})Note that theidfield will have precedence in case both ofidandfeaturesare provided. We highly recommend you to use theidfield. First of all it means that you don't have to take care of storing the features between calls to@/api/predictand@/api/learn. Secondly it makes the metrics more reliable because they will be using the predictions that were made at the time@/api/predictwas called.Monitoring metricsYou can access the current metrics via a GET request to the@/api/metricsroute.Additionally, you can access a stream of metric updates by using the@/api/stream/metrics. This is a streaming route which implementsserver-sent events (SSE). As such it will notify listeners every time the metrics are updates. For instance, you can use thesseclient, which is a thin layer on top ofrequests:importjsonimportsseclientmessages=sseclient.SSEClient('http://localhost:5000/api/stream/metrics')formsginmessages:metrics=json.loads(msg.data)print(metrics)You can use the following piece of JavaScript to do the same thing in a browser:vares=newEventSource('http://localhost:5000/api/stream/metrics');es.onmessage=e=>{varmetrics=JSON.parse(e.data);console.log(metrics)};Monitoring eventsYou can also listen to all the prediction and learning events via the@/api/stream/eventsroute. This will yield SSE events with an event name attached, which is either 'predict' or 'learn'. From a Python interpreter, you can do the following:importjsonimportsseclientmessages=sseclient.SSEClient('http://localhost:5000/api/stream/events')formsginmessages:data=json.loads(msg.data)ifmsg.event=='learn':print(data['model'],data['features'],data['prediction'],data['ground_truth'])else:print(data['model'],data['features'],data['prediction'])In JavaScript, you can you use theaddEventListenermethod:vares=newEventSource('http://localhost:5000/api/stream/events');es.addEventListener('learn',e=>{vardata=JSON.parse(e.data);console.log(data.model,data.features,data.prediction,data.ground_truth)};es.addEventListener('predict',e=>{vardata=JSON.parse(e.data);console.log(data.model,data.features,data.prediction)};Visual monitoringA live dashboard is accessible if you navigate tolocalhost:5000in your browser.Under the hood the dashboard is simply listening to the API's streaming routes.Usage statisticsYou can obtain some essential statistics, by querying the@/api/statsroutes:importrequestsr=requests.get('http://localhost:5000/api/stats')print(r.json())Here is an output example:{"learn":{"ewm_duration":3408682,"ewm_duration_human":"3ms408μs682ns","mean_duration":6541916,"mean_duration_human":"6ms541μs916ns","n_calls":98},"predict":{"ewm_duration":3190724,"ewm_duration_human":"3ms190μs724ns","mean_duration":5248635,"mean_duration_human":"5ms248μs635ns","n_calls":213}}Themean_durationfields contain the average duration of each endpoint. Theewm_durationfields contain anexponential moving averageof said duration, and therefore gives you an idea of the recent performance, which can allow you to detect arising performance issues. Note that these durations do not include the time it takes to transmit the response over the network. These durations only pertain to the processing time onchantilly's side, including but not limited to calls to the model.These statistic are voluntarily very plain. Their only purpose is to provide a quick healthcheck. The proper way to monitor a web application's performance, including a Flask app, is to use purpose-built tools. For instance you could useLokito monitor the application logs andGrafanato visualize and analyze them.Using multiple modelsYou can use different models by giving them names. You can provide a name to a model by adding a suffix to@/api/model:fromriverimporttreeimportdillimportrequestsmodel=tree.DecisionTreeClassifier()requests.post('http://localhost:5000/api/model/barney-stinson',data=dill.dumps(model))You can also pick a name when you add the model through the CLI:>chantillyadd-modelmodel.pkl--namebarney-stinsonYou can then choose which model to use when you make a prediction:r=requests.post('http://localhost:5000/api/predict',json={'id':42,'features':{'shop':'Ikea','item':'Dombäs','date':'2020-03-22T10:42:29Z'},'model':'barney-stinson'})The model which was provided last will be used by default if themodelfield is not specified. If you provide anid, then the model which was used for making the prediction will be the one that is updated once the ground truth is made available. You can also specify which model to update directly as so:requests.post('http://localhost:5000/api/learn',json={'id':42,'ground_truth':10.21,'model':'barney-stinson'})Note that the data associated with the givenidis deleted once the model has been updated. In other words you can't call the@/api/modelwith the sameidtwice.You can view the available models as well as the default model by sending a GET request to@/api/models:r=requests.get('http://localhost:5000/api/models')print(r.json())You can delete a model by sending a DELETE request to@/api/model:requests.delete('http://localhost:5000/api/model/barney-stinson')Configuration handlingchantillyfollows Flask'sinstance folderpattern. This means that you can configurechantillyvia a file namedinstance/config.py. Note that the location is relative to where you are launching thechantilly runcommand from (more information can be foundhere). You can also configurechantillyby setting environment variables.You can set all thebuiltin variablesthat Flask provides. You can also set the following variables which are specific tochantilly:STORAGE_BACKEND: determines whichstorage backendto use.SHELVE_PATH: location of theshelvedatabase file. Only applies ifSTORAGE_BACKENDis set toshelve.REDIS_HOST: required ifSTORAGE_BACKENDis set toredis.REDIS_PORT: required ifSTORAGE_BACKENDis set toredis.REDIS_DB: required ifSTORAGE_BACKENDis set toredis.Theinstance/config.pyis a Python file that gets executed before the app starts, therefore this is also where you canconfigure logging. Here is an exampleinstance/config.pyfile:fromlogging.configimportdictConfigdictConfig({'version':1,'formatters':{'default':{'format':'[%(asctime)s]%(levelname)sin%(module)s:%(message)s',}},'handlers':{'wsgi':{'class':'logging.FileHandler','filename':'/var/log/chantilly/error.log','formatter':'default'}},'root':{'level':'INFO','handlers':['wsgi']}})SECRET_KEY='keep_it_secret_keep_it_safe'SHELVE_PATH='/usr/local/chantilly'Using a different storage backendCurrently, the default storage backend is based on theshelvemodule. It's possible to use a different backend by setting theSTORAGE_BACKENDenvironment variable.RedisAdd the following to yourinstance/config.pyfile:STORAGE_BACKEND='redis'REDIS_HOST='localhost'REDIS_PORT=6379REDIS_DB=0Naturally, the values have to be chosen according to your Redis setup.Importing librariesIt's highly likely that your model will be using external dependencies. A prime example is thedatetimemodule, which you'll probably want to use to parse datetime strings. Instead of specifying which libraries you wantchantillyto import, the current practice is to import your requirementswithinyour model. For instance, here is an excerpt taken from theNew-York city taxi trips example:fromriverimportcomposefromriverimportlinear_modelfromriverimportpreprocessingdefparse(trip):importdatetimeasdttrip['pickup_datetime']=dt.datetime.fromisoformat(trip['pickup_datetime'])returntripdefdatetime_info(trip):importcalendarday_no=trip['pickup_datetime'].weekday()return{'hour':trip['pickup_datetime'].hour,**{day:i==day_nofori,dayinenumerate(calendar.day_name)}}model=compose.FuncTransformer(parse)model|=compose.FuncTransformer(datetime_info)model|=preprocessing.StandardScaler()model|=linear_model.LinearRegression()Note that you need to make sure that the Python interpreter you're runningchantillywith has access to the libraries you want to use.DeploymentEssentially,chantillyis just a Flask application. Therefore, it allows the samedeployment optionsas any other Flask application.ExamplesNew-York city taxi trips 🚕Deployment with Docker Compose 🐋Developmentgitclonehttps://github.com/online-ml/chantillycdchantilly
python3-mvenvenvsourceenv/bin/activate
pipinstall-e".[dev]"pythonsetup.pydevelopThere are some extra dependencies that can be installed if necessary.pipinstall-e".[dev,redis]"You can then run tests.pytestThe default testing environment uses theshelvemodule; you can also use redis:pytest--redisYou may also run the app in development mode.exportFLASK_ENV=development
chantillyrunTo deploy to PyPI:Updatechantilly/__version__.pypip install twinepython setup.py sdist bdist_wheelRoadmapHTTP long polling: Currently, clients can interact withriverover a straightforward HTTP protocol. Therefore the speed bottleneck comes from the web requests, not from the machine learning. We would like to provide a way to interact withchantillyvia long-polling. This means that a single connection can be used to process multiple predictions and model updates, which reduces the overall latency.Scaling: At the momentchantillyis designed to be run as a single server. Ideally we want to allowchantillyin a multi-server environment. Predictions are simple to scale because the model can be used concurrently. However, updating the model concurrently leads toreader-write problems. We have some ideas in the pipe, but this is going to need some careful thinking.Grafana dashboard: The current dashboard is a quick-and-dirty proof of concept. In the long term, we would like to provide a straighforward way to connect with aGrafanainstance without having to get your hands dirty. Ideally, we would like to use SQLite as a data source for simplicity reasons. However, The Grafana teamhas not plannedto add support for SQLite. Instead, they encourage users to useplugins. We might also look intoPrometheusandInfluxDB.Support more paradigms: For the moment we cater to regression and classification models. In the future we also want to support other paradigms, such as time series forecasting and recommender systems.Technical stackFlaskfor the web server.dillfor model serialization.marshmallowfor the API input validation.Vue.js,Chart.js, andMoment.jsfor the web interface.Similar alternativesMost machine learning deployment tools only support making predictions with a trained model. They don't cater to online models which can be updated on the fly. Nonetheless, some of them are quite interesting and are very much worth looking into!CortexClipperLicenseriveris free and open-source software licensed under the3-clause BSD license.