Package: auto-function-serving
auto-function-servingA python package to offload a function call to an http server running on localhost automatically using a decorator. Compatible with multiprocessing, pickle, flask, fastapi, async etc..WhyImagine a case of a multi threaded or multiprocessing application where 1 or few functions are heavily resource (cpu or memory) intensive, but the other functions can run in parallel.Example - an api call followed by tokenization and classification using a large DL model followed by further API calls.In such a case, it would make sense to create a server (generally using torchserve or tfserving) to serve requests, and replace the function call with a post request to the server.ServerHandler creates asynchronousserver and replaces any calls to the function automatically during runtime.Requests are made to 1 instance of a process running ahttp.server.HTTPServerwhich runs the function within it.AsyncServerHandler is also available which makes the requests asynchronously.Even calls made from different processes, threads, multiprocessing, flask, FastApi and async event loops are made to the same server process.UsageIn general :some code with a callablecan be replaced with an instance of Either ServerHandler or AsyncserverHandler that accepts the code as a string in it's first argument and the name of the callable as the second argument.fromauto_function_serving.ServerHandlerimportServerHandlercallable_name=ServerHandler("""some independent code with a callable""","callable_name")Example :importmodule1importmodule2deffunctionname(someinput):a=module1.function1(someinput)returnmodule2.function2(a)can be replaced withfromauto_function_serving.ServerHandlerimportAsyncserverHandlerfunctionname=AsyncServerHandler("""import module1import module2def functionname(someinput):a = module1.function1(someinput)return module2.function2(a)""","functionname",port="Any")Decorators (@AsyncserverHandler.decorator and @ServerHandler.decorator) and AsyncServerHandler details in more usage.Argumentsfromauto_function_serving.ServerHandlerimportServerHandlercallable_name=ServerHandler("""some independent code with a callable""","callable_name",port=None,backend='Popen',wait=100,backlog=1024))portif None, then the input code is hashed and a port is chosen from 50000 to 60000 using the hashif int, then int is chosenotherwise, a random open port is chosenbackend - either 'Popen' or 'multiprocessing'. Popen Should be used in general.wait - approx max number of seconds to wait for the server to run. No waiting done if set to 0, default 100backlog - max number of backlogged requests before returning errors, python default is 5, but default in ServerHandler is 1024.Featuresrunshttp.server.HTTPServer.ServerHandler and AsyncServerHandler objects can be loaded and unloaded with pickle.Uses Popen or multiprocessing to run the server.Uses only a single external dependency (aiohttp), and only for async.http, not https.chooses a port based on hash of input. (unless specified otherwise)AdvantagesMinimal code changes.Should be compatible with almost all functions in almost all CPython envs. (Not sure where it could fail? Please add an issue if you find one.)Memory leaks or errors (from the server) are extremely unlikely since it is minimal, single threaded, single process and a default component of python stdlib.Exceptions cause 5xx errors without closing the server.Even Separate Processes will make requests to 1 instance of the same server unless specified otherwise. (Because it's looking for a server on a specific port.).Can specify otherwise by set the port to any free port so that a new ServerHandler object starts a new server.http post requests : lightweight, few ms overhead, reliable.Async is a good feature.now with tests.DisadvatagesHaving a string of code as an argument to a class is not pythonic, unless the decorator is used.Importing inside functions is not ideal, even when the decorator is used.http post requests : insecure, few ms overhead.Exceptions inside the server are not sent back.No batching.No inbuilt logging. (Could be added).
Initialization delay of upto few seconds to start the server.
Async functions will not work on the server.Possible Edge casesNo auto server restart in case server closes.May leave some resources locked for a while (<1min) if not closed properly.Problems might occur if Popen or multiprocessing are not available.Possible nested async errors with jupyter or other? Please look intonest-asyncioand theissues.Warnings from somewhat hacky (but legit and completely functional) workarounds.Closing of server process indeland atexit.redister(del) fail for some reason (tested and unlikely).InstallationUse the package manager pip to installauto_function_servingpipinstallauto_function_servingHow does this work?Code for the server is stored inServerHandler.base_code and some string formatting is used to fill in the blanks.The server process is started with Popen (or multiprocessing if specified). The first thing it does is import socket and bind the port - if it's not available the code stops after an exception. Therefore only 1 instance of the server runs at a time on a machine.We know the function is ready after we can receive a valid get request from the server.Inputs and outputs are sent as bytes, converted to and from objects using pickle.If port is None in while initializing (default), a port from 50000 to 60000 is chosen by hashing the input code to make it independent of the source of a function. Collisions of different functions are possible, but unlikely. The collision of the same function in multiple processes is used to make sure only 1 server process runs at a time. The port can be specified if needed.Performance (On my machine)overhead for small input and output (few bytes) -~2ms for requests with urllib.request~4ms for async requests with aiohttp.ClientSessionoverhead for large input and output~10ms for 0.5 mb input and output (1mb total transfer).~60ms for 5 mb input and output (10 mb total transfer).~600ms for 50 mb input and output (100 mb total transfer).More UsageIt can also be used with the provided decorator for functions with no dependencies outside the function.fromauto_function_serving.ServerHandlerimportServerHandler@ServerHandler.decoratordefsomeheavyfunction(args,**kwargs):foriinrange(big_number)someexpensivecomputationimports inside the function will workfromauto_function_serving.ServerHandlerimportServerHandler@ServerHandler.decoratordefsomeheavyfunction(args,**kwargs):importnumpyasnpfromauto_function_serving.ServerHandlerimportServerHandler@ServerHandler.decoratordefsomeheavyfunction(args,**kwargs):ifnothasattr(someheavyfunction,'RunOnce'):globalnpimportnumpyasnpsetattr(someheavyfunction,'RunOnce',None)...etcWhen the somemodule does not have any expensive global loading.fromauto_function_serving.ServerHandlerimportServerHandlerfromsomemoduleimportsomeheavyfunctionsomeheavyfunction=ServerHandler.decorator(someheavyfunction)Ip address can be changed by setting ServerHandler.ip_address (default "127.0.0.1") before creating a new instance.AsyncServerHandlerAsyncServerHandler is also available which usesaiohttpto make the requests asynchronously, for use with fastapi and other async use cases.AsyncServerHandler has the same usage as ServerHandler, except calls need to be awaited or used with asyncio.run() or with asyncio.get_event_loop().run_until_complete().Number of async calls can be limited by setting AsyncServerHandler.TCPConnector_limit which controls theTCPconnectorlimit (default 100). UsingSemaphoreis also something to consider.Other things to look intoLibraries : Celery, Tfserving, Torchserve, FlaskSending globals and locals to execast treesContributingPull requests are welcome.LicenseApache License 2.0