Package: model-server
Model-ServerA Purepython-3based flexible gRPC server for hosting Deep Learning, Machine Learning models trained on any framework! The documentation can be found herehttps://abhijit-2592.github.io/model-server/InstallationMethod 1:Installing frompython pippip3 install model-serverMethod 2:Creating wheel from githubclone the repositoryRunbash create_pip_wheel_and_upload.sh. This will prompt for userid and password. You canctrl-cthisThen install the created wheelMethod 3:No installation. Using the source code directly.If this is the case, you need to compile the protobufs. Runbash compile_protobufs.sh. Then add the project root to your$PYTHONPATH.Note:Method 2 and 3 requireslibprotoc>=3.6.0Why Model-Server?Taking deep learning models to production at scale is not a very straight forward process. If you are using Tensorflow then you haveTensorflow Serving. But, if you are using other frameworks likePyTorch,MXNet,scikit-learnetc. Taking your model to production is not very straight forward (Flask, Django and other ReST frameworks). Ideally you should be able to extend Tensorflow Serving to support models from other frameworks also but, this is extremly cumbersome! Thus, to bridge this gap we haveModel-Server: A high performance framework neutral serving solution!The idea is:if you are able to train your model inpythonyou should be  able to deploy at scale using purepythonSalient FeaturesModel-Serveris heavily inspired fromTensorflow ServingOut of box client side batching supportPure python implementation: You don't need to fiddle around with C++ to have a scalable deployment solutionFramework neutral:  UsingPyTorch,MXNetetc? Don't worry! The solution is platform neutral. If you can use a framework to train inpython-3,Model-Servercan deploy it at scaleSingle server for multi-framework and multi-models: You can host multiple models using the same framework or a mixture of multiplePyTorch,MXNet,scikit-learnTensorflowetc models!Getting startedThe core of Model Server is aServable. A servable is nothing but apython classcontaining your model's prediction definition which will be served by theModel-Server. All servables must inherit frommodel_server.Servablefor theModel-Serverto serve it.To deploy your model to production withModel Server, you just have to write a singlepython-3file containing aclasswhich inherits frommodel_server.Servableand has the following two methods:predict(self,input_array_dict)get_model_info(self,list_of_model_info_dict)Now run the floowing to start the server in5001portpython-mmodel_server.runserverpath_to_custom_servable_file.pyFor more info on  command line arguments:python-mmodel_server.runserver--helpA simple examplecreate a file calledsimple_servable.pywith the following contents:importnumpyasnpfrommodel_serverimportServableclassmy_custom_servable(Servable):def__init__(self,args):# args contains values from ArgumentParser# Thus you can pass any kwargs via command line and you get them herepassdefpredict(self,input_array_dict):"""This method is responsible for the gRPC call GetPredictions().All custom servables must define this method.Arguments:input_array_dict (dict): The PredictionRequest proto decoded as a python dictionary.# exampleinput_array_dict = {"input_tensor_name1": numpy array,"input_tensor_name2": numpy array}Returns:A python dictionary with key (typically output name) and value as numpy array of predictions# exampleoutput = {"output_tensor_name1": numpy array,"output_tensor_name2": numpy array}"""print(input_array_dict)return({"output_array1":np.array([100,200]).astype(np.float32),"output_array2":np.array(["foo".encode(),"bar".encode()]).astype(object),# you can get and pass strings encoded as bytes also})defget_model_info(self,list_of_model_info_dict):"""This method which is responsible for the call GetModelInfo()Arguments:list_of_model_info_dict (list/tuple): A list containing model_info_dictsNote:model_info_dict contains the following keys:{"name": "model name as string""version": "version as string""status": "status string""misc": "string with miscellaneous info"}Returns:list_of_model_info_dict (dict): containing the model and server info. This is similar to the function input"""return[{"name":"first_model","version":1,"status":"up"},{"name":"second_model","version":2,"status":"up","misc":"Other miscellaneous details"}]Now run:python-mmodel_server.runserverpath/to/simple_servable.pyTo start the gRPC server!Now let's define the client!importgrpcimportnumpyasnpfrommodel_serverimportserver_pb2,server_pb2_grpcfrommodel_server.utilsimportcreate_tensor_protofrommodel_server.utilsimportcreate_predict_requestfrommodel_server.utilsimportcreate_array_from_protofrommodel_server.utilsimportcreate_model_info_protochannel=grpc.insecure_channel('localhost:5001')# default port# create a stub (client)stub=server_pb2_grpc.ModelServerStub(channel)input_array_dict={"input1":create_tensor_proto(np.array([1,2]).astype(np.uint8)),"input2":create_tensor_proto(np.array([[10.0,11.0],[12.0,13.0]]).astype(np.float32)),"input3":create_tensor_proto(np.array(["Hi".encode(),"Hello".encode(),"test".encode()]).astype(object))}# create the prediction requestpredict_request=create_predict_request(input_array_dict,name="simple_call")# make the callresponse=stub.GetPredictions(predict_request)# decode the responseprint(create_array_from_proto(response.outputs["output_array1"]))# prints: array([100., 200.], dtype=float32)# Getting the model statusmodel_info_proto=create_model_info_proto([])# you can pass an empty list alsoresponse=stub.GetModelInfo(model_info_proto)Look atexamplesfolder for further examplesWork in ProgressSupport server side batching and async calls.Provide a gRPC endpoint forActive Learningso that you can plug inModel Serverwith your labeling tool and train on fly!Provide a ReST wrapperFeel free to file issues, provide suggestions and pull requests