Package: batchflows
Batchflows for Python 3This tool will help you create and process a lot of data in an organized manner.
You can create batches of processing synchronously and asynchronously.remember it's in BETA :DGet Startedimportloggingfrombatchflows.BatchimportBatchfrombatchflows.StepimportStep#First extend Step class and implement method executeclassSaveValueStep(Step):def__init__(self,value_name,value):#Remember name is required if you want use remote stepssuper().__init__()self.value_name=value_nameself.value=value# "_context" is a dict you can use to store values that will be used in other steps.defexecute(self,_context):#do what u have to do here!_context[self.value_name]=self.value#creating a second step just to make the explanation richerclassSumCalculatorStep(Step):def__init__(self,attrs):super().__init__()self.attrs=attrsdefexecute(self,_context):calc=0.0forattrinself.attrs:calc+=_context[attr]_context['sum']=calc#Here we create our batch!batch=Batch()batch.add_step(SaveValueStep('value01',1))batch.add_step(SaveValueStep('value02',4))batch.add_step(SumCalculatorStep(['value01','value02','other_value']))#You can add something useful to your steps before starting bath!batch.context['other_value']=5#than execute your batch and be happy ;)batch.execute()logging.info(batch.context)Let's try run some parallel codeimportloggingimporttimefrombatchflows.BatchimportBatchfrombatchflows.StepimportStep,ParallelFlowsclassSomeStep(Step):defexecute(self,_context):#count to 10 slowlyc=0whilec<10:c+=1print(c)time.sleep(1)#Create your AsyncFlowlazy_counter=ParallelFlows('LazySteps01')#add steps so they run in parallellazy_counter.add_step(SomeStep('lazy01'))lazy_counter.add_step(SomeStep('lazy02'))lazy_counter2=ParallelFlows('LazySteps02')lazy_counter2.add_step(SomeStep('lazy03'))lazy_counter2.add_step(SomeStep('lazy04'))batch=Batch()batch.add_step(lazy_counter)batch.add_step(lazy_counter2)#batchfllows will wait for each step to finish before executing the next one.#In this example lazy_counter will be called first and execute steps "lazy01" and "lazy02" in parallel.#Only when both steps finish ,the batch will star lazy_counter2batch.execute()FileContextmanager and RemoteStepYou can extend RemoteStep class and make your code to run a remote batch.
Unfortunately the basic context does not allow remote steps to be performed without customization.
To solve this problem we have FileContextManagerlet's start by creating our main batchcontext_manager=FileContextManager(filepath='/tmp',#where u set where you want batch create status file#you can put a disc that all your machines shareis_remote_step=False,#default false. You saying here this is the main batchprocess_id='123ABC',#default random uuid. You can specify an id for the process.#This field is important for remote batches to be able to write the status file correctly.process_name='batch name'#default random uuid. For the main batch this field has no importance, but for your remote batch#you need put the same name of remote-step)batch=Batch(context_manager=context_manager)batch.add_step(RemoteBatchStep(name='do-a-barrel-roll',timeout=10#in seconds))batch.execute()now let's create our remote batchcontext_manager=FileContextManager(filepath='/tmp',is_remote_step=True,process_id='123ABC',#exactly the same a main batchprocess_name='do-a-barrel-roll'#exactly the same name as step)batch=Batch(context_manager=context_manager)batch.add_step(...)# lot of cool thingsbatch.add_step(...)# lot of cool thingsbatch.add_step(...)# lot of cool thingsbatch.execute()customize your ContextManagerYou can also extend the ContextManager class and create your way to run remote code.classMyContextManager(ABCContextManager):def__init__(self):self.context=dict()self.steps=[]#override this method to teach your customization how to identify if remote execution is readydefis_remote_step_done(self,name:str):raiseNotImplementedError()#override this method if you want to execute some code after batch concludedefupon_completion(self,success:bool,error:str=None):pass