Package: blitzen
BlitzenBlitzen is a python framework built on top of python's multiprocessing module intended to maximize processor utilization on individual computers and clusters.Installationpip install blitzenUsage:Blitzen uses adispatcherto provide concurrent processing features.
Below are the dispatchers available and their use case.MulticoreDispatcherMulticore dispatcher is similar tomultiprocessing.Poolin that you initialize the dispatcher and can pass it tasks to complete concurrently.The notable differences are:Workers you create with a dispatcher will remain open until you shutdown the dispatcher.You can queue multiple tasks, and the dispatcher will complete all of them using the specified number of processes.Workers are Exception resistant, meaning if the worker subprocess crashes, the dispatcher will shut down the process and initialize a new one until the specified number of workers are actively monitor incoming tasks.Example:importtimeimportrandomfromblitzenimportMulticoreDispatcherdeff1():delay=random.randint(3,5)time.sleep(delay)print('Finished after',delay,'secs.')returndelaydeff2(delay):time.sleep(delay)raiseValueError('Throwing error')if__name__=='__main__':dispatcher=MulticoreDispatcher(workers=4)task_id=dispatcher.run(f2,4)task_ids=[dispatcher.run(f1)for_inrange(5)]All tasks are either started or queued immediately upon running thedispatcher.run()call.dispatcher.run()returns the dispatcher's task_id for the task you just passed it. This is used if you want to get specific task results from the dispatcher or tasks results in a specific order.Getting results from the dispatcher:print('Fetching Results.')results=dispatcher.get_results(task_ids)print('Recevied results from dispatcher:',results)The results are always returned in the order oftask_ids.Eventually you will have to shutdown the dispatcher to close the underlying worker processes.
You can do this two ways.dispatcher.shutdown()Terminates all worker subprocesses and deallocates memory resources.dispatcher.join()returns all results in task order as they were passed to the dispatcher, then callsdispatcher.shutdown().print('Fetching Results.')results=dispatcher.join()print('Recevied results from dispatcher:',results)Critical Note: Once you request a result from the dispatcher, it is removed the dispatchers memory by default. To override this feature use dispatcher.get_results(clear=False)A full example is visiblehere.DistributedDispatcherDistributedDispatcher lets you utilize a cluster for concurrent computing. It will handle packet synchronization between clients, servers, and drivers. It will also log all activity on the clients and servers.Define some common functionsThe clients and servers need access to the same functions, so it is likely a good idea to make a common file that will be used to import functions that arent natively in python.#common.pyimporttimedeftime_consuming_function(delay):time.sleep(delay)returndelayInitialize your serverfromblitzen.utilsimportget_local_ipfromblitzen.distributedimportDistributedDispatcherfromcommonimporttime_consuming_functionif__name__=='__main__':ip=get_local_ip()dispatcher=DistributedDispatcher(server_ip=ip)dispatcher.spawn_server(duration=30)#Run server for 30 secondsInitialize your clientsfromblitzen.utilsimportget_local_ipfromblitzen.distributedimportDistributedDispatcherfromcommonimporttime_consuming_functionif__name__=='__main__':ip='192.168.1.2'#Server IPdispatcher=DistributedDispatcher(server_ip=ip)dispatcher.spawn_client(workers=6)Run your driver code to be executed by the clientsfromblitzen.utilsimportget_local_ipfromblitzen.distributedimportDistributedDispatcherfromcommonimporttime_consuming_functionif__name__=='__main__':ip='192.168.1.2'#Server IPdispatcher=DistributedDispatcher(server_ip=ip)#With DistributedDispatcher you can specify an amount#of time each client has to finish their task#with the `timeout` keywordtask_ids=[dispatcher.run(time_consuming_function,i+5,timeout=7)foriinrange(5)]print('Requesting results.')results=dispatcher.join()print(results)LoggingDistributedDispatchers provide some logging, using the python logging module, so you can monitor your cluster. By default the logger is configured to report all logging info of levellogging.INFO. To some this can be excessive, so blitzen provides some functions to control the logger.You can get the logger usinglogger = blitzen.logging.get_logger()This logger is used by all blitzen submodules.The other blitzen.logging fuctions are:set_logfile(filename)lets you set a file to log out to.set_loglevel(level)lets you set the logging level using the same convention as python'sloggingmodule.disable()disabled the logger.enable()enables the logger.A full example visiblehere.