Package: ventu
VentuServing the deep learning models easily.InstallpipinstallventuFeaturesonly need to implement Model(preprocess,postprocess,inferenceorbatch_inference)request & response data validation usingpydanticAPI document usingSpecTree(when run withrun_http)backend service usingfalconsupports both JSON andmsgpackdynamic batching withbatchingusing Unix domain socket or TCPerrors in one request won't affect others in the same batchload balancingsupport all the runtimehealth checkmonitoring metrics (Prometheus)if you have multiple workers, remember to setupprometheus_multiproc_direnvironment variable to a directoryinference warm-upHow to usedefine your request data schema and response data schema withpydanticadd examples toschema.Config.schema_extra[examples]for warm-up and health check (optional)inheritventu.Ventu, implement thepreprocessandpostprocessmethodsfor standalone HTTP service, implement theinferencemethod, run withrun_httpfor the worker behind dynamic batching service, implement thebatch_inferencemethod, run withrun_socketcheck thedocumentfor API detailsExampleThe demo code can be found inexamples.ServiceInstall requirementspip install numpy torch transformers httpximportargparseimportloggingimportnumpyasnpimporttorchfrompydanticimportBaseModel,confloat,constrfromtransformersimportDistilBertTokenizer,DistilBertForSequenceClassificationfromventuimportVentu# request schema used for validationclassReq(BaseModel):# the input sentence should be at least 2 characterstext:constr(min_length=2)classConfig:# examples used for health check and warm-upschema_extra={'example':{'text':'my cat is very cut'},'batch_size':16,}# response schema used for validationclassResp(BaseModel):positive:confloat(ge=0,le=1)negative:confloat(ge=0,le=1)classModelInference(Ventu):def__init__(self,*args,**kwargs):# initialize super class with request & response schema, configssuper().__init__(*args,**kwargs)# initialize model and other toolsself.tokenizer=DistilBertTokenizer.from_pretrained('distilbert-base-uncased')self.model=DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased-finetuned-sst-2-english')defpreprocess(self,data:Req):# preprocess a request data (as defined in the request schema)tokens=self.tokenizer.encode(data.text,add_special_tokens=True)returntokensdefbatch_inference(self,data):# batch inference is used in `socket` modedata=[torch.tensor(token)fortokenindata]withtorch.no_grad():result=self.model(torch.nn.utils.rnn.pad_sequence(data,batch_first=True))[0]returnresult.numpy()definference(self,data):# inference is used in `http` modewithtorch.no_grad():result=self.model(torch.tensor(data).unsqueeze(0))[0]returnresult.numpy()[0]defpostprocess(self,data):# postprocess a response data (returned data as defined in the response schema)scores=(np.exp(data)/np.exp(data).sum(-1,keepdims=True)).tolist()return{'negative':scores[0],'positive':scores[1]}defcreate_model():logger=logging.getLogger()formatter=logging.Formatter(fmt='%(asctime)s-%(levelname)s-%(module)s-%(message)s')handler=logging.StreamHandler()handler.setFormatter(formatter)logger.setLevel(logging.DEBUG)logger.addHandler(handler)model=ModelInference(Req,Resp,use_msgpack=True)returnmodeldefcreate_app():"""for gunicorn"""returncreate_model().appif__name__=="__main__":parser=argparse.ArgumentParser(description='Ventu service')parser.add_argument('--mode','-m',default='http',choices=('http','unix','tcp'))parser.add_argument('--host',default='localhost')parser.add_argument('--port','-p',default=8080,type=int)parser.add_argument('--socket','-s',default='batching.socket')args=parser.parse_args()model=create_model()ifargs.mode=='unix':model.run_unix(args.socket)elifargs.mode=='tcp':model.run_tcp(args.host,args.port)else:model.run_http(args.host,args.port)You can run this script as:a single thread HTTP service:python examples/app.pya HTTP service with multiple workers:gunicorn -w 2 -b localhost:8080 'examples.app:create_app()'when run as a HTTP service, can check the follow links:/metricsPrometheus metrics/healthhealth check/inferenceinference/apidoc/redocor/apidoc/swaggerOpenAPI documentan inference worker behind the batching service:python examples/app.py -m socket(Unix domain socket) orpython examples/app.py -m tcp --host localhost --port 8888(TCP) (need to run thebatching servicefirst)ClientfromconcurrentimportfuturesimporthttpximportmsgpackURL='http://localhost:8080/inference'HEADER={'Content-Type':'application/msgpack'}packer=msgpack.Packer(autoreset=True,use_bin_type=True,)defrequest(text):returnhttpx.post(URL,data=packer.pack({'text':text}),headers=HEADER)if__name__=="__main__":withfutures.ThreadPoolExecutor()asexecutor:text=['They are smart','what is your problem?','I hate that!','x',]results=executor.map(request,text)fori,respinenumerate(results):print(f'>>{text[i]}-> [{resp.status_code}]\n'f'{msgpack.unpackb(resp.content)}')