Package: simple-pypeliner
Pypelinerpypelineris a simple python framework for building data processing pipelines.you can check out the documentationhere.and checkout some exampleshere.Installationit can be simply installing usingpip:pipinstallsimple-pypelinerGetting StartedLet's say we want to build a pipeline for processing textual data coming from a file.
we will need 3 main components to achieve this using pypeliner.Readerfirst we will need a reader for reading the file, and it can be defined like the following:frompypeliner.readers.baseimportBaseReaderclassFileReader(BaseReader):def__init__(self,file_path:str)->None:self.file_path=file_pathdefread(self)->str:withopen(self.file_path,"r")asfile_handler:returnfile_handler.read()inheriting fromBaseReadermakes sure that thereadmethod is implemented to be used later within theReader.
the code basically is reading a file and returning its content as a string.Processorsnext we will define our processing logic, lets say we want to make all letters in the text file lower case, then remove
stop words i.g: but, and, or, etc after that remove numbers.to achieve that we will write a processor for each one of these tasks.each processor should define its ownprocessmethod. a processor will receive thestateit needs to process from theRunner.ifstateis passed the that processor will use thestateinstead of the state passed by the runner,
this is achieved using thesupercall forprocessmethod and passingstatethere.
it is recommended to always callsuperfor theprocessmethod as a best practice.PROCESSOR_NAMEis a verbose name for a processor.1- Lower Case ProcessorclassLowerCaseProcessor(BaseProcessor):PROCESSOR_NAME="Lower Case Processor"defprocess(self,state:Any=None)->Any:returnstate.lower()2- Remove Stop Words ProcessorclassRemoveStopWordsProcessor(BaseProcessor):PROCESSOR_NAME="Remove Stop Words Processor"defprocess(self,state:Any=None)->Any:stop_words=["the","to","and","a","in","it","is","am","I","that","had","on","for","be","were","was","of","or","it","an",]result=""forstop_wordinstop_words:result=re.sub(rf"\W+{stop_word.title()}\W+"," ",re.sub(rf"\W+{stop_word}\W+"," ",state))returnresult3- Remove Numbers ProcessorclassRemoveNumbersProcessor(BaseProcessor):PROCESSOR_NAME="Remove Numbers Processor"defprocess(self,state:Any=None)->Any:returnre.sub(r"\d+","",state)Runnerthe runner is the place where everything comes together, it will use theReaderto load the file
and define the running loop for theProcessorwe'll be using the built inBaseRunnerfromrunners.baseimportBaseRunnerrunner=BaseRunner(processors=[LowerCaseProcessor(),RemoveStopWordsProcessor(),RemoveNumbersProcessor(),],reader=FileReader(file_path="test.txt"),configuration=RunnerConfiguration(post_processors=[],post_processors=[],verbose=True,run_timers=True))print(runner.run())processorsparameter will set the processors of the pipeline and also its order.readerparameter will set the reader of the pipeline to be the file reader we defined earlier.configurationparameter will set the configuration for the runner.verbosewill print the current processor that is running andrun_timerswill print the time consumed by each processor to run and you can also setpre-processorsandpost-processorsthat will run before and after each processor.for more information visit the documentation atpypeliner