Package: datamultiproc
datamultiproc: A Python multiprocessing data pipeline libraryCreate Processors, compose them into pipelines and process your data using Python's
multiprocessing library.ConceptsSampleA Sample is a data object that is passed through the pipeline.
A custom sample can be simply implemented by inheriting fromBaseSampleand adding
the desired data fields. Note that andef __init__(self, ...)method is not required.
'BaseSample' is build in top of PydanticsBaseModeland inherits its functionality.
For more information, seePydantic.Data fields can be of any type and also Optional. The latter is especially handy when they are filled
during the execution of the pipeline.importnumpyasnpfromdatamultiprocimportBaseSamplefromtypingimportOptionalclassCustomSample(BaseSample):data:intoptional_data:Optional[str]=Nonearray:np.ndarrayBaseSamplehas two special fields (idandprocessing_history).
The first has to be filled during instantiation of aSample, the latter is appended to during execution
of the pipeline, in order to keep track of all processing steps.classBaseSample(Aggregate):processing_history:List[Tuple[str,str]]=[]id:strProcessorAProcessorcarries out an actual processing step on aSample.
To implement a customProcessor, you simply inherit from theProcessorclass and implement theprocess()method.# Processor without argumentsclassCustomProcessor(Processor):defprocess(self,sample:Sample)->Sample:# do something with sample, e.g.sample.data=sample.date+1returnsample# Processor with argumentsclassCustomProcessor(Processor):step_size:int_hidden_arg:int=1defprocess(self,sample:Sample)->Sample:# do something with sample, e.g.sample.data=sample.date+self.step_size+self._hidden_argreturnsampleComposeCompose chains multipleProcessortogether.
They are executed in the order they are passed to theComposeconstructor.process=Compose(CustomProcessor(),CustomProcessor(step_size=2),CustomProcessor(step_size=3),)samples=[...]# some list of samplesprocessed_samples=[process(s)forsinsamples]Cache and SaveCacheandSaveare specialProcessorsthat can be used to store intermediate results.CachestoresSamplesafter the processing step, that it is wrapped around as a '.pickle' file.
The files are stored inside a subfolder of the specifiedsave_to_dirfolder.
The subfolder is named after the "Processor" class.
If the pipeline is executed again,Cachewill first check if the file already exists
and if so, it will load theSamplesfrom the file instead of executing the wrappedProcessoragain.Saveworks similarly, but does not load samples from file, if they already exist.Cacheis especially useful for storing intermediate results, that are expensive to compute.Saveis useful for storing the final results of a pipeline.process=Compose(Cache(processor=CustomProcessor(),cache_dir="path/to/cache"),CustomProcessor(step_size=2),CustomProcessor(step_size=3),Save(save_to_dir="path/to/save"),)MultiprocessingThe library can be used with Python'smultiprocessinglibrary, as follows.
A full working example can be found inexample.py.frommultiprocessingimportProcess,Queue# number of processor cores of your machineNUM_PROCESSES=8# list of samplessamples=[...]process=Compose([SomeProcessor(),AnotherProcessor(),Save(save_to_dir="path/to/save"),])# put the samples into a multiprocessing Queuesamples_queue=Queue()forsinsamples:samples_queue.put(s)defdo_processing(process:Callable,queue:Queue):whilenotqueue.empty():sample=queue.get()try:sample=process(sample)exceptProcessingError:print(f"failed processing:{sample.id}")processes=[]for_inrange(NUM_PROCESSES):p=Process(target=do_processing,args=(process,samples_queue))p.start()processes.append(p)forpinprocesses:p.join()