Package: model-serving
model-servingFlask based python wrapper for deploying models as a REST based service based on💡 Flask💡 Gunicorn💡 Protobuf3 (optional for schema validation)🥳 flask-caching🥳 prometheus metricsThe repo also contains examples of registering end points and a Makefile to run the serviceInstallationpip install model-servingProject Structureaylien_model_serving
│
|--- requirements.txt
|--- Makefile
│   
│
└───app
│    |-- app_factory.py
│    |-- cached_app_factory.py       
│      
│   
└───examples
    │-----example_schema.proto
    |-----example_schema_pb2.py(autogenerated by protoc)
    │-----example_serving_handler.py
    |-----example_serving_handler_cached.pyHow it worksIt runs a web service on the given port (defaults to8000).Any incoming request JSON will be passed to yourServingHandler.process_requestYourServingHandler.process_requestis expected to return ajsonThe request and response will be validated with a protobuf schema (optional)This library wraps common service code, monitoring, exception handling, etc.UsageInstall this library as a dependency for whatever model you want to serve.Create aServingHandler(see below for interface details).Run the make targetmake COMMAND_UNCACHED='ServingHandler.run_app()' example-serviceInterfacesThe main interface to flask apps defined inapp_factoryis theprocess_jsonfunction.
This function expects to receive json input, optionally perform schema
validation, then call thecallable_handlerfunction using each of the fields
in the json object as a keyword argument to the function. The function is expected to
return an object that can be parsed to json and sent as the response.This design allows for a very simple but powerful interface that can easily make an endpoint
out of just about any Python function.Example Serving HandlerThe example serving handler definedheredoes the followingDefines a method predict_lang. For the purposes of this example, this returns a static prediction. Ideally would be the prediction
or classification from your model.Imports a protobuf3 generated .py schema file(only if you require the json message to be schema validated)Defines a function process_request that calls the wrapper function process_json with the callable from 1 and schema from 2Registers process_request and its route mappingRepeat 1-4 for a (route, callable) pair if you have more than one service end point.importexamples.example_schema_pb2asschemafromaylien_model_serving.app_factoryimportFlaskAppWrapper,InvalidRequestdefpredict_lang(text):return"en",0.71defpredict(title=None,body=None,enrichments=None):ifbodyisNone:body=enrichments["extracted"]["value"]["body"]iftitleisNoneandbodyisNone:raiseInvalidRequest("Missing text")article_text=f"{title}{body}"detected_lang,confidence=predict_lang(article_text)return{'language':detected_lang,'confidence':confidence,'error':'Not an error','version':'0.0.1'}defprocess_request():returnFlaskAppWrapper.process_json(predict,schema=schema)defrun_app():routes=[{"endpoint":"/","callable":process_request,"methods":["POST"]}]returnFlaskAppWrapper.create_app(routes)Note that the FlaskAppWrapper accepts a callable in the process_json , and if you'd like to load a classifier or model.bin in your memory you could modify it like below 👇classClassifyHandler:def__init__(self):self.classifier=Classifier()#this is the classifier to load , or a binary in local file storagedef__call__(self,text):returnself.classifier.predict(text)defrun_app():classify_handler=ClassifyHandler()routes=[{"endpoint":"/classify","callable":classify_handler,"methods":["POST"]}]returnFlaskAppWrapper.create_app(routes)