Package: workercontext
Multi-Worker ContextThis is a small project that create a context in python which can spin up multiple workers to do a task.The only requirement is that a the function you want to give multiple workers to has only one argument that you would like to batch on. It is up to you to ensure that creating independant batches for multiple workers makes sense with this argument.This makes refactoring code for multiprocessing a lot easier and should provide quick performance wins for time consuming functions with no interdependencies between elements.Setuppip install workercontextExampleBy default the work distribution will occur on the first parameter to the function.Take the following function:fromtypingimportListdefmy_func(arr:List[int])->List[int]:return[el*2forelinarr]This can be simply split across multiple workers as follows:fromworkercontextimportMultiWorkerarr=list(range(100))withMultiWorker(my_func,n_processes=8)asf:res=f(arr)print(res)reswill be a list of lists ofintsin this case, if you would like to reduce across all workers then you can pass a reduction:fromworkercontextimportMultiWorkerfromworkercontext.reductionsimportflatten_reductionarr=list(range(100))withMultiWorker(my_func,n_processes=8,reduction=flatten_reduction)asf:res=f(arr)print(res)reswill now a list ofint.If you wanted to combine multiple reductions then you can use the reduction composition classfromworkercontextimportMultiWorkerfromworkercontext.reductionsimportReductionComposition,flatten_reduction,average_reductionreductions=ReductionComposition([flatten_reduction,average_reduction])withMultiWorker(my_func,n_processes=8,reduction=reductions)asf:res=f(arr)print(res)This makes res be a singlefloat.Using other parametersYou can batch work on other parameters by specifying them in the constructor.fromworkercontextimportMultiWorkerfromworkercontext.reductionsimportflatten_reductiondefmy_func(l1:List[int],l2:List[int])->int:foriinrange(len(l2)):forel1inl1:l2[i]+=el1returnl2arr1=list(range(100))arr2=list(range(100))withMultiWorker(my_func,batched_arg='l2',n_processes=8,reduction=flatten_reduction)asf:res=f(arr1,arr2)print(res)DocumentationMultiWorkerparametersfunction(Callable): The function to create the context for.n_processes(int): The number of processes to spawn.batched_arg(str,optional): The argument to batch on, if None the the first arg is used. Defaults to None.verbose(bool,optional): Whether or not to print information about the processing. Defaults to False.reduction(Callable[[List[Any]], Any],optional): A reduction function to be applied across the outputs of the pool. Defaults to None.Supported Reductionsflatten_reductionhistogram_reductionproduct_reductionstring concatenation_reductionbitwise and_reductionbitwise or_reductionmin_reductionmax_reductionTestingpytestFormattingblack .How it worksTL;DR it does a bunch of introspection.The args to your function are introspected.Theselfarg is remove if you passed it a method from a class.If no batched arg was specified then the first one is selected.All args are converted into kwargs using the introspected arg names and the*argsprovided.The size of the batched arg is calculated and the chunk sizes are derived.The arg is batched and batches of kwargs are created.A pool is created with a partial for a wrapper function that allows for the batching to occur on the kwargs. The last parameter to the wrapper is a callback to your function.A reduction is applied (if specified)Results are returned.When you leave the context the pools are joined and closed.