Package: coinstac-computation
COINSTAC computations development made easy.A very intuitive wrapper for writing coinstac based computations:Break down your computations into simple phases with automatic transition between phases.Add as many phases as you want.Phasesalternate between local and remote automaticallyby default starting from the first phase of the local. See advanced use case example below for extras like:Run phases that needs to be run multiplelocal-remotetrips; Specifymulti_iterations=Truewhile adding a phase.Run phases that needs to be run either in local or remote without making a trip(like preprocessing, gathering final results ...); Specifylocal_only=Truewhile adding a phase.Automatic logging that saves what comes and leaves on each phase. Just setdebug=True.Installation:Runpip install coinstac-computationAdd entrycoinstac-computationto the requirements.txt.ComputationPhasesignature:fromcoinstac_computationimportComputationPhaseclassPhaseLoadData(ComputationPhase):def_initialize(self):"""Put anything that needs to be initialized only once here"""passdefcompute(self):out={}..."""To end multi-iterative phase, and go to the next phase, in local or remote set:"""out['jump_to_next']=True"""To stop the computation, In remote set:"""out['success']=TruereturnoutExample: Gather max even numbers from each siteA full working use case is in theexamples/basicdirectory where:Local sites filters out even numbers and sends to the remote.Remote finds the max across sites and returns the final result to each of the sites.Sites save final result.inputspec.json data:[{"data":{"value":[10,3,5,6,7,8,12,38,32,789,776,441]}},{"data":{"value":[12,33,88,61,37,58,103,3386,312,9,77,41]}}]Local node pipeline:importosfromcoinstac_computationimportCOINSTACPyNode,ComputationPhaseclassPhaseLoadData(ComputationPhase):defcompute(self):data=[]fordinself.input['data']:ifd%2==0:data.append(d)return{'filtered_data':data}classPhaseSaveResult(ComputationPhase):defcompute(self):withopen(f"{self.out_dir+os.sep+'results.txt'}",'w')asout:out.write(f"{self.input['aggregated_data']}")local=COINSTACPyNode(mode='local',debug=True)local.add_phase(PhaseLoadData)local.add_phase(PhaseSaveResult)Remote node pipeline:fromcoinstac_computationimportCOINSTACPyNode,ComputationPhase,PhaseEndWithSuccessclassPhaseCollectMaxEvenData(ComputationPhase):defcompute(self):data=[]forsite,site_varsinself.input.items():site_max=max(site_vars['filtered_data'])data.append(site_max)return{'aggregated_data':data}remote=COINSTACPyNode(mode='remote',debug=True)remote.add_phase(PhaseCollectMaxEvenData)remote.add_phase(PhaseEndWithSuccess)Entry point:importcoinstacfromlocal_pipelineimportlocalfromremote_pipelineimportremotecoinstac.start(local,remote)Run:cd examples/basic/
~/coinstac-computation/examples/basic/$ docker build -t base . && coinstac-simulatorAdvanced use cases:1. Multiple local <---> remote iterationsexample:Each sites cast a vote(positive vote if number is even) for multiple(default=51) times.Remote gathers the votes and returns the final voting result to all sites at the end.Sites save the final result.Overview:Specify when to end the iterative phase with a phase jump signal asjump_to_next=True:classPhaseSubmitVote(ComputationPhase):def_initialize(self):"""This method runs only once"""self.cache['data_index']=0self.cache['data']=[]forlineinopen(self.base_dir+os.sep+self.input_args['data_source']).readlines():self.cache['data'].append(float(line.strip()))defcompute(self):out={'vote':self.cache['data'][self.cache['data_index']]%2==0,}self.cache['data_index']+=1"""Send a jump to next phase signal"""out['jump_to_next']=self.cache['data_index']>len(self.cache['data'])-1returnoutAdd the phase as multi-iterations:local.add_phase(PhaseSubmitVote,multi_iterations=True)2. Send data across local <---> remoteexample:To make it simple, we send a matrix of size 1000 by 1000 to remote, aggregate it by averaging, and return to each site.Local:importosfromcoinstac_computationimportCOINSTACPyNode,ComputationPhaseimportnumpyasnpclassPhaseLoadData(ComputationPhase):defcompute(self):out={}data=np.random.randn(*self.input['matrix_shape'])out.update(**self.send("site_matrix",data))returnoutclassPhaseSaveResult(ComputationPhase):defcompute(self):data=self.recv('averaged_matrix')np.save(self.out_dir+os.sep+"averaged_matrix.npy",data)local=COINSTACPyNode(mode='local',debug=True)local.add_phase(PhaseLoadData)local.add_phase(PhaseSaveResult)Remote:fromcoinstac_computationimportCOINSTACPyNode,ComputationPhase,PhaseEndWithSuccessimportnumpyasnpclassPhaseAggregateMatrix(ComputationPhase):defcompute(self):out={}data=self.recv("site_matrix")mean_data=np.array(data).mean(0)out.update(**self.send("averaged_matrix",mean_data))returnoutremote=COINSTACPyNode(mode='remote',debug=True)remote.add_phase(PhaseAggregateMatrix)remote.add_phase(PhaseEndWithSuccess)Sample logs from local0[INPUT] 14:27:36 02/04/2022 
  ->{'data_source': 'data_file.txt'}
[CACHE] 14:27:36 02/04/2022 
  ->{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 0, 'PhaseSaveResult': 0}}}
  <-{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 1, 'PhaseSaveResult': 0}}, 'input_args': {'data_source': 'data_file.txt'}, 'next_phase': 'PhaseSubmitVote', 'data_index': 1, 'data': [712.0, 309.0, 574.0, 838.0, 296.0, 349.0, 781.0, 749.0, 360.0, 702.0, 253.0, 831.0, 911.0, 14.0, 259.0, 805.0, 494.0, 501.0, 549.0, 624.0, 919.0, 836.0, 362.0, 373.0, 563.0, 134.0, 610.0, 875.0, 328.0, 299.0, 874.0, 387.0, 743.0, 233.0, 834.0, 870.0, 685.0, 342.0, 79.0, 270.0, 314.0, 42.0, 364.0, 902.0, 755.0, 248.0, 815.0, 4.0, 21.0, 423.0, 302.0], 'PHASE:PhaseSubmitVote': True}
[OUTPUT] 14:27:36 02/04/2022 
  <-{'output': {'vote': True, 'jump_to_next': False}}

[INPUT] 14:27:37 02/04/2022 
  ->{}
[CACHE] 14:27:37 02/04/2022 
  ->{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 1, 'PhaseSaveResult': 0}}, 'input_args': {'data_source': 'data_file.txt'}, 'next_phase': 'PhaseSubmitVote', 'data_index': 1, 'data': [712.0, 309.0, 574.0, 838.0, 296.0, 349.0, 781.0, 749.0, 360.0, 702.0, 253.0, 831.0, 911.0, 14.0, 259.0, 805.0, 494.0, 501.0, 549.0, 624.0, 919.0, 836.0, 362.0, 373.0, 563.0, 134.0, 610.0, 875.0, 328.0, 299.0, 874.0, 387.0, 743.0, 233.0, 834.0, 870.0, 685.0, 342.0, 79.0, 270.0, 314.0, 42.0, 364.0, 902.0, 755.0, 248.0, 815.0, 4.0, 21.0, 423.0, 302.0], 'PHASE:PhaseSubmitVote': True}
  <-{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 2, 'PhaseSaveResult': 0}}, 'input_args': {'data_source': 'data_file.txt'}, 'next_phase': 'PhaseSubmitVote', 'data_index': 2, 'data': [712.0, 309.0, 574.0, 838.0, 296.0, 349.0, 781.0, 749.0, 360.0, 702.0, 253.0, 831.0, 911.0, 14.0, 259.0, 805.0, 494.0, 501.0, 549.0, 624.0, 919.0, 836.0, 362.0, 373.0, 563.0, 134.0, 610.0, 875.0, 328.0, 299.0, 874.0, 387.0, 743.0, 233.0, 834.0, 870.0, 685.0, 342.0, 79.0, 270.0, 314.0, 42.0, 364.0, 902.0, 755.0, 248.0, 815.0, 4.0, 21.0, 423.0, 302.0], 'PHASE:PhaseSubmitVote': True}
[OUTPUT] 14:27:37 02/04/2022 
  <-{'output': {'vote': False, 'jump_to_next': False}}

[INPUT] 14:27:37 02/04/2022 
  ->{}
[CACHE] 14:27:37 02/04/2022 
  ->{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 2, 'PhaseSaveResult': 0}}, 'input_args': {'data_source': 'data_file.txt'}, 'next_phase': 'PhaseSubmitVote', 'data_index': 2, 'data': [712.0, 309.0, 574.0, 838.0, 296.0, 349.0, 781.0, 749.0, 360.0, 702.0, 253.0, 831.0, 911.0, 14.0, 259.0, 805.0, 494.0, 501.0, 549.0, 624.0, 919.0, 836.0, 362.0, 373.0, 563.0, 134.0, 610.0, 875.0, 328.0, 299.0, 874.0, 387.0, 743.0, 233.0, 834.0, 870.0, 685.0, 342.0, 79.0, 270.0, 314.0, 42.0, 364.0, 902.0, 755.0, 248.0, 815.0, 4.0, 21.0, 423.0, 302.0], 'PHASE:PhaseSubmitVote': True}
  <-{'PIPELINE:LOCAL': {'index': 0, 'iterations': {'PhaseSubmitVote': 3, 'PhaseSaveResult': 0}}, 'input_args': {'data_source': 'data_file.txt'}, 'next_phase': 'PhaseSubmitVote', 'data_index': 3, 'data': [712.0, 309.0, 574.0, 838.0, 296.0, 349.0, 781.0, 749.0, 360.0, 702.0, 253.0, 831.0, 911.0, 14.0, 259.0, 805.0, 494.0, 501.0, 549.0, 624.0, 919.0, 836.0, 362.0, 373.0, 563.0, 134.0, 610.0, 875.0, 328.0, 299.0, 874.0, 387.0, 743.0, 233.0, 834.0, 870.0, 685.0, 342.0, 79.0, 270.0, 314.0, 42.0, 364.0, 902.0, 755.0, 248.0, 815.0, 4.0, 21.0, 423.0, 302.0], 'PHASE:PhaseSubmitVote': True}
[OUTPUT] 14:27:37 02/04/2022 
  <-{'output': {'vote': True, 'jump_to_next': False}}
...Sample logs from remote[INPUT] 14:27:37 02/04/2022 
  ->{'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}
[CACHE] 14:27:37 02/04/2022 
  ->{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 0, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}}
  <-{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 1, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}, 'input_args': {'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}, 'next_phase': 'PhaseCollectVote', 'vote_ballot': [[1, 3]], 'PHASE:PhaseCollectVote': True}
[OUTPUT] 14:27:37 02/04/2022 
  <-{'output': {}, 'success': False}

[INPUT] 14:27:37 02/04/2022 
  ->{'local0': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': True, 'jump_to_next': False}, 'local3': {'vote': True, 'jump_to_next': False}}
[CACHE] 14:27:37 02/04/2022 
  ->{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 1, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}, 'input_args': {'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}, 'next_phase': 'PhaseCollectVote', 'vote_ballot': [[1, 3]], 'PHASE:PhaseCollectVote': True}
  <-{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 2, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}, 'input_args': {'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}, 'next_phase': 'PhaseCollectVote', 'vote_ballot': [[1, 3], [3, 1]], 'PHASE:PhaseCollectVote': True}
[OUTPUT] 14:27:37 02/04/2022 
  <-{'output': {}, 'success': False}

[INPUT] 14:27:37 02/04/2022 
  ->{'local0': {'vote': True, 'jump_to_next': False}, 'local3': {'vote': True, 'jump_to_next': False}, 'local2': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': True, 'jump_to_next': False}}
[CACHE] 14:27:37 02/04/2022 
  ->{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 2, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}, 'input_args': {'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}, 'next_phase': 'PhaseCollectVote', 'vote_ballot': [[1, 3], [3, 1]], 'PHASE:PhaseCollectVote': True}
  <-{'PIPELINE:REMOTE': {'index': 0, 'iterations': {'PhaseCollectVote': 3, 'PhaseSendGlobalResults': 0, 'PhaseEndWithSuccess': 0}}, 'input_args': {'local0': {'vote': True, 'jump_to_next': False}, 'local1': {'vote': False, 'jump_to_next': False}, 'local2': {'vote': False, 'jump_to_next': False}, 'local3': {'vote': False, 'jump_to_next': False}}, 'next_phase': 'PhaseCollectVote', 'vote_ballot': [[1, 3], [3, 1], [4, 0]], 'PHASE:PhaseCollectVote': True}
[OUTPUT] 14:27:37 02/04/2022 
  <-{'output': {}, 'success': False}
...Development notes:Make sure you have:dockerinstalled and running.nodejsinstalled.coinstac-simulatorpackage installed.npm install --global coinstac-simulatorMust setdebug=Falsewhile deploying.Backward compatible to the older library(compspecVersion=1):Add the following snippet at the end of local and remote pipeline scripts.if__name__=="__main__":local.to_stdout()Useversion 1.0compspec format.Comment out lineCMD ["python", "entry.py"]in theDockerfile.You can also use aremote debuggerin pycharm ashere.Thanks!