Package: tanbih-pipeline
Pipeline provides an unified interface to set up data stream processing systems with Kafka, Pulsar,
RabbitMQ, Redis and many more. The idea is to free developer from the dynamic change of technology
in deployment, so that a docker image released for a certain task can be used with Kafka or Redis
through changes of environment variables.Featuresa unified interface from Kakfa to Pulsar, from Redis to MongoDBcomponents connection controlled via command line, or environment variablessupport file and in-memory for testingRequirementsPython 3.7, 3.8Installation$pipinstalltanbih-pipelineYou can install the required backend dependencies with:$pipinstalltanbih-pipeline[redis]$pipinstalltanbih-pipeline[kafka]$pipinstalltanbih-pipeline[pulsar]$pipinstalltanbih-pipeline[rabbitmq]$pipinstalltanbih-pipeline[elastic]$pipinstalltanbih-pipeline[mongodb]If you want to support all backends, you can:$pipinstalltanbih-pipeline[full]ProducerProducer is to be used when developing a data source in our pipeline. A source
will produce output without input. A crawler can be seen as a producer.>>>fromtypingimportGenerator>>>frompydanticimportBaseModel>>>frompipelineimportProducerasWorker,ProducerSettingsasSettings>>>>>>classOutput(BaseModel):...key:int>>>>>>classMyProducer(Worker):...defgenerate(self)->Generator[Output,None,None]:...foriinrange(10):...yieldOutput(key=i)>>>>>>settings=Settings(name='producer',version='0.0.0',description='')>>>producer=MyProducer(settings,output_class=Output)>>>producer.parse_args("--out-kind MEM --out-topic test".split())>>>producer.start()>>>[r.get('key')forrinproducer.destination.results][0,1,2,3,4,5,6,7,8,9]ProcessorProcessor is to be used to process input. Modification will be in-place. A processor
can produce one output for each input, or no output.>>>frompydanticimportBaseModel>>>frompipelineimportProcessorasWorker,ProcessorSettingsasSettings>>>>>>classInput(BaseModel):...temperature:float>>>>>>classOutput(BaseModel):...is_hot:bool>>>>>>classMyProcessor(Worker):...defprocess(self,content,key):...is_hot=(content.temperature>25)...returnOutput(is_hot=is_hot)>>>>>>settings=Settings(name='processor',version='0.1.0',description='')>>>processor=MyProcessor(settings,input_class=Input,output_class=Output)>>>args="--in-kind MEM --in-topic test --out-kind MEM --out-topic test".split()>>>processor.parse_args(args)>>>processor.start()SplitterSplitter is to be used when writing to multiple outputs. It will take a function to
generate output topic based on the processing message, and use it when writing output.>>>frompipelineimportSplitterasWorker,SplitterSettingsasSettings>>>>>>classMySplitter(Worker):...defget_topic(self,msg):...return'{}-{}'.format(self.destination.topic,msg.get('id'))>>>>>>settings=Settings(name='splitter',version='0.1.0',description='')>>>splitter=MySplitter(settings)>>>args="--in-kind MEM --in-topic test --out-kind MEM --out-topic test".split()>>>splitter.parse_args(args)>>>splitter.start()UsageChoosing backend technology:kinddescriptionmulti-
readershared
readerdata
expireLREDISRedis ListXXreadXREDISRedis StreamXXlimitKAFKAKafkaXXreadPULSARPulsarXXttlRABBITMQRabbitMQXreadELASTICElasticSearchMONGODBMongoDBFILE*json,csvMEM*memoryFILE accepts jsonl input on stdin and with filename, it also accepts csv file.
Both format can be gzipped.MEM read and write to memory, designed for unit tests.# check command line arguments for certain input and outputworker.py--in-kindFILE--help# orIN_KIND=FILEworker.py# orexportIN_KIND=FILEworker.py--help# process input from file and output to stdout (--in-content-only is
# needed for this version)worker.py--in-kindFILE--in-filenamedata.jsonl--in-content-only\--out-kindFILE--out-filename-# read from file and write to KAFKAworker.py--in-kindFILE--in-filenamedata.jsonl--in-content-only\--out-kindKAFKA--out-namespacetest--out-topicarticles\--out-kafkakafka_url--out-configkafka_config_jsonArgumentscommondebug
monitoringkind
namespace
topicinput:FILEScriptspipeline-copyis a script to copy data from a source to a destination. It can
be used to inject data from a file to a database, or from a database to another
database. It is implemented as a Pipeline worker.Since JSON format does not support datetimes, in order forpipeline-copyto
treat datetime field as datetime instead of string, you can provide a model
definition via argument–model-definition. An example of such model definition
is as following (the class name needs to beModel):fromdatetimeimportdatetimefromtypingimportOptionalfrompydanticimportBaseModelclassModel(BaseModel):hashtag:strusername:strtext:strtweet_id:strlocation:Optional[str]created_at:datetimeretweet_count:intEnvironment VariablesApplication accepts following environment variables
(Please note, you will need to add prefixIN_,–in-andOUT_,–out-to these variables to indicate the option for
input and output). Please refer to backend documentation for
available arguments/environment variables.Customize SettingsclassCustomSettings(Settings):new_argument:str=Field("",title="a new argument for custom settings")classCustomProcessor(Processor):def__init__(self):settings=CustomSettings("worker","v0.1.0","custom processor")super().__init__(settings,input_class=BaseModel,output_class=BaseModel)ErrorsPipelineError will be raised when error occursContributeUsepre-committo runblackandflake8# install pre-commitpre-commitinstallCreditsYifan Zhang (yzhang at hbku.edu.qa)