Package: tfserve
TFServeTFServe is a framework designed to serve tensorflow models in a simple and easy way as an HTTP API server. It's built on top ofWerkzeug.How to install$pipinstalltfserveAfter installingtfserve, install eithertensorflowoftensorflow-gpu(the latter if you have GPU available).$pipinstalltensorflowor$pipinstalltensorflow-gpuHow to usePython APIYou will need 5 parts:Model: it can be a.pbfile or a model directory containing ckpt files.Input tensor names: name of the input tensors of the graph.Output tensor names: name of the output tensors of the graph.encode: python function that receives the request body data and outputs adictmapping input tensor names to input numpy values.decode: python function that receives adictmapping output tensor names to output numpy values and returns the HTTP response.Follow the example to learn how to combine these parts...ExampleDeploy image classification service that receives a binary jpg image and returns the class of the object found in the image alongside it's probability.# 1. Model: trained mobilenet on ImageNet that can be downloaded from#           https://storage.googleapis.com/mobilenet_v2/checkpoints/mobilenet_v2_1.4_224.tgzMODEL_PATH="mobilenet_v2_1.4_224/mobilenet_v2_1.4_224_frozen.pb"# 2. Input tensor names:INPUT_TENSORS=["import/input:0"]# 3. Output tensor names:OUTPUT_TENSORS=["import/MobilenetV2/Predictions/Softmax:0"]# 4. encode function: Receives raw jpg image as request data. Returns dict#                     mappint import/input:0 to numpy value.defencode(request_data):withtempfile.NamedTemporaryFile(mode="wb",suffix=".jpg")asf:f.write(request_data)# Model receives 224x224 normalized RGB image.img=Image.open(f.name).resize((224,224))img=np.asarray(img)/255.return{INPUT_TENSORS[0]:img}# 5. decode function: Receives `dict` mapping import/MobilenetV2/Predictions/Softmax:0 to#                     numpy value and builds dict with for json response.defdecode(outputs):p=outputs[OUTPUT_TENSORS[0]]# 1001 vector with probabilities for each class.index=np.argmax(p)return{"class":index_to_class_map[index-1],"prob":float(p[index])}That's it! Now create TFServeApp object and run it!app=TFServeApp(MODEL_PATH,INPUT_TENSORS,OUTPUT_TENSORS,encode,decode)app.run('127.0.0.1',5000)# Host and port where the server will be runningSeeclient.pyfor full example.How to consume serverThe server supports onlyPOSTmethod to/with the input information as part of the request body.The input will be proccessed in the encode function to produce the feed_dict object that will be passed to the graph. The graph output will be processed in the decode function and the server will return whatever the decode function returns.CLItfservealso provides a CLI program with built-in encode/decode handlers:tfserve-mPATH[-iINPUTS][-oOUTPUTS][-hHANDLER][-b][-HHOST][-pPORT]-mPATH,--modelPATHpathtopbfileordirectorycontainingcheckpoint-iINPUTS,--inputsINPUTSacommaseparatedlistofinputtensors-oOUTPUTS,--outputsOUTPUTSacommaseparatedlistofoutputtensors-hHANDLER,--handlerHANDLERencode/decodehandler(deaultis'json')-b,--batchprocessmultipleinputs(defaultistoprocessoneinputperrequest)-HHOST,--hostHOSThostinterfacetobindto(0.0.0.0)-pPORT,--portPORTporttolistenon(5000)Example$tfserve-mmodels/graph.pb-ix:0-oout:0-hjson-HlocalhostRuntfservewithmodels/graph.pbmodel that takes as input the tensor with namex:0(dimesion: [?,5]) and outputs a tensor namedout:0. The server will run onhttp://localhost:5000/and will receivePOSTrequests to/.By using thejson handler, you can provide the input data as ajsonobject in the request body:{"x:0":[1,1,3,4,5]}You will receive ajsonoutput object as:{"out:0":0.48}More information about CLIRun:$tfserve--helpHelpWhat if I don't know the tensor names?You can usetfserve.helper.estimate_io_tensors(model_path)function to get a list of possible input/output tensor names. Also, you can use the CLI by running:tfserve -m [model_path] --help-modelWhat if I want to run multiple inferences at the same time?You can usebatch=Truewhen building tfserve.TFServeApp. You will then need to handle the batch dimension yourself in theencodeanddecodefunction.
Also, if using the CLI, just add the--batchflag.LimitationIt only works with one-to-one models. That is, models that need to run the graph only once to get the inference.
Other architectures of inference will be supported soon. Help is appreciated!