Package: ubatch
uBatchuBatchis a simple, yet elegant library for processing streams data in micro batches.uBatchallow to process multiple inputs data from different threads
as a single block of data, this is useful when process data in a batches
has a lower cost than processing it independently, for example process data
in GPU or take advantage from optimization of libraries written in C. Ideally,
the code that processes the batches should release the Python GIL for allowing
others threads/coroutines to run, this is true in many C libraries wrapped in
Python.Example>>>importthreading>>>>>>fromtypingimportList>>>fromubatchimportubatch_decorator>>>>>>>>>@ubatch_decorator(max_size=5,timeout=0.01)...defsquared(a:List[int])->List[int]:...print(a)...return[x**2forxina]...>>>>>>inputs=list(range(10))>>>>>># Run squared as usual..._=squared(inputs)[0,1,2,3,4,5,6,7,8,9]>>>>>>>>>defthread_function(number:int)->None:..._=squared.ubatch(number)...>>>>>># Multiple threads squared individual inputs...threads=[]>>>foriininputs:...t=threading.Thread(target=thread_function,args=(i,))...threads.append(t)...t.start()...[0,1,2,3,4][5,6,7,8,9]>>>fortinthreads:...t.join()The example above shows 10 threads calculating the square of a number, usinguBatchthe threads delegate the calculation task to a single
process that calculates them in batch.And with multiple parameters in user method>>>importthreading>>>>>>fromtypingimportList>>>fromubatchimportubatch_decorator>>>>>>>>>@ubatch_decorator(max_size=5,timeout=0.01)...defsquared_cube(a:List[int],mode:List[str])->List[int]:...print(a)...print(mode)...return[x**2ify=='square'elsex**3forx,yinzip(a,mode)]...>>>>>>inputs=list(range(10))>>>modes=['square'ifi%2==0else'cube'foriininputs]>>>>>># Run function as usual>>>_=squared_cube(inputs,modes)[0,1,2,3,4,5,6,7,8,9]['square','cube','square','cube','square','cube','square','cube','square','cube']>>>>>>>>>defthread_function(number:int,mode:str)->None:..._=squared_cube.ubatch(number,mode)...>>>>>># Multiple threads squared individual inputs...threads=[]>>>fori,jinzip(inputs,modes):...t=threading.Thread(target=thread_function,args=(i,j))...threads.append(t)...t.start()...[0,1,2,3,4]['square','cube','square','cube','square'][5,6,7,8,9]['cube','square','cube','square','cube']>>>fortinthreads:...t.join()This example is pretty similar to the previous one, the only difference is
that the decorated function receives an additional parameter anduBatchis able to support a variable number of parameters.If you have a function with a parameter that doesn't need to be accumulated,
with every call you can use the python "partial" tool before the use of
ubatch_decorator.>>>importthreading>>>>>>fromfunctoolsimportpartial>>>fromtypingimportList,Any>>>fromubatchimportubatch_decorator>>>>>>>>>defsquared_cube(model:Any,a:List[int],mode:List[str])->List[int]:...print(a)...print(mode)...return[x**2ify=='square'elsex**3forx,yinzip(a,mode)]...>>>squared_cube=partial(squared_cube,'This is a model')>>>squared_cube=ubatch_decorator(max_size=5,timeout=0.01)(squared_cube)......The code after that can remains like in the previous example.Installing uBatch and Supported VersionspipinstallubatchuBatch officially supports Python 3.6+.Why using uBatchWhen data is processed offline it is easy to collect data to be processed at
same time, the same does not happen when requests are attended online as
example using Flask, this is where the uBatch potential comes in.TensorFloworScikit-learnare just some of the libraries
that can take advantage of this functionality.uBatch and application serverPython application servers work like this:When the server is initialized multiple processes are created and each process
create a bunch of threads for handling requests. Taking advantage of those
threads that run in paralleluBatchcan be used to group several
inputs and process them in a single block.Let's see a Flask example:importnumpyasnpfromtypingimportList,DictfromflaskimportFlask,requestasflask_requestfromflask_restximportResource,ApifromubatchimportUBatchfrommodelimportload_modelapp=Flask(__name__)api=Api(app)model=load_model()predict_batch:UBatch[np.array,np.array]=UBatch(max_size=50,timeout=0.01)predict_batch.set_handler(handler=model.batch_predict)predict_batch.start()@api.route("/predict")classPredict(Resource):defpost(self)->Dict[str,List[float]]:received_input=np.array(flask_request.json["input"])result=predict_batch.ubatch(received_input)return{"prediction":result.tolist()}Start application server:gunicorn-kgeventapp:appAnother example usinguBatchto join multiple requests into one:importrequestsfromtypingimportList,DictfromflaskimportFlask,requestasflask_requestfromflask_restximportResource,Apifromubatchimportubatch_decoratorapp=Flask(__name__)api=Api(app)FAKE_TITLE_MPI_URL="http://my_mpi_url/predict"@ubatch_decorator(max_size=100,timeout=0.03)defbatch_fake_title_post(titles:List[str])->List[bool]:"""Post a list of titles to MPI and return responses in a list"""# json_post example: {"predict": ["title1", "title2", "title3"]}json_post={"predict":titles}# response example: {"predictions": [False, True. False]}response=requests.post(FAKE_TITLE_MPI_URL,json=json_post).json()# return: [False, True, False]return[xforxinresponse["predictions"]]@api.route("/predict")classPredict(Resource):defpost(self)->Dict[str,bool]:# title example: "Title1"title=flask_request.json["title"]# prediction example: Falseprediction=fake_title_batch.ubatch(title)return{"prediction":prediction}Start application server:gunicorn-kgeventapp:appStart developing uBatchInstall poetrycurl-sSLhttps://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py|python-Clone repositorygitclonegit@github.com:mercadolibre/ubatch.gitStart shell and install dependenciescdubatch
poetryshell
poetryinstallRun testspytestBuilding docscdubatch/docs
poetryshell
makehtmlLicensinguBatchis licensed under the Apache License, Version 2.0.
SeeLICENSEfor the full license text.