Package: unipipeline
unipipelinesimple way to build the declarative and distributed data pipelines.Why you should use itDeclarative strict configScaffoldingFully typedPython support 3.6+Brokers supportkafkarabbitmqinmemory simple pubsubInterruption handling = safe user code transactionsCLIHow to Install$pip3installunipipelineExample# dag.yml
---

service:
  name: "example"
  echo_colors: true
  echo_level: error


external:
  service_name: {}


brokers:
  default_broker:
    import_template: "unipipeline.brokers.uni_memory_broker:UniMemoryBroker"

  ender_broker:
    import_template: "example.brokers.uni_log_broker:LogBroker"


messages:
  __default__:
    import_template: "example.messages.{{name}}:{{name|camel}}"

  input_message: {}

  inetermediate_message: {}

  ender_message: {}


cron:
  my_super_task:
    worker: my_super_cron_worker
    when: 0/1 * * * *

  my_mega_task:
    worker: my_super_cron_worker
    when: 0/2 * * * *

  my_puper_task:
    worker: my_super_cron_worker
    when: 0/3 * * * *


waitings:
  __default__:
    import_template: example.waitings.{{name}}_wating:{{name|camel}}Waiting

  common_db: {}


workers:
  __default__:
    import_template: "example.workers.{{name}}:{{name|camel}}"

  my_super_cron_worker:
    input_message: uni_cron_message

  input_worker:
    input_message: input_message
    waiting_for:
      - common_db

  intermediate_first_worker:
    input_message: inetermediate_message
    output_workers:
      - ender_second_worker
    waiting_for:
      - common_db

  intermediate_second_worker:
    input_message: inetermediate_message
    external: service_name
    output_workers:
      - ender_frist_worker

  ender_frist_worker:
    input_message: ender_message

  ender_second_worker:
    input_message: ender_message
    broker: ender_broker
    waiting_for:
      - common_dbGet Startedcreate./unipipeline.ymlsuch as example aboverun cli commandunipipeline-f./unipipeline.ymlscaffoldIt should create all structure of your workers, brokers and so onremove error raising from workerscorrect message structure for make more usefullcorrect broker connection (if need)run cli command to run your consumerunipipeline-f./unipipeline.ymlconsumeinput_workeror with pythonfromunipipelineimportUniu=Uni(f'./unipipeline.yml')u.init_consumer_worker(f'input_worker')u.initialize()u.start_consuming()produce some message to the message broker by your self or with toolsunipipeline-f./unipipeline.ymlproduce--workerinput_worker--data='{"some": "prop"}'or with python# main.pyfromunipipelineimportUniu=Uni(f'./unipipeline.yml')u.init_producer_worker(f'input_worker')u.initialize()u.send_to(f'input_worker',dict(some='prop'))DefinitionServiceservice:name:some_name# need for health-check file nameecho_level:warning# level of uni console logs (debug, info, warning, error)echo_colors:true# show colors in consoleExternalexternal:
  some_name_of_external_service: {}no propsit needs for declarative grouping the external workers with serviceWorkerworkers:__default__:# each worker get this default props if definedretry_max_count:10some_worker_name:retry_max_count:3# just counter. message move to /dev/null if limit has reachedretry_delay_s:1# delay before retrytopic:"{{name}}"# template stringerror_payload_topic:"{{topic}}__error__payload"# template stringerror_topic:"{{topic}}__error"# template stringbroker:"default_broker"# broker name. reference to message transportexternal:null# name of external service. reference in this config fileack_after_success:true# automatic ack after process messagewaiting_for:# list of references-some_waiting_name# name of block. this worker must wait for connection of this external service if needoutput_workers:# list of references-some_other_worker_name# allow worker sending messages to this workerinport_template:"some.module.hierarchy.to.worker.{{name}}:{{name|camel}}OfClass"# required module and classname for importinput_message:"name_of_message"# required reference of input message typeWaitingwaitings:some_blocked_service_name:retry_max_count:3# the same semantic as worker.retry_max_countretry_delay_s:10# the same semantic as worker.retry_delay_simport_template:"some.module:SomeClass"# required. the same semantic as worker.import_templateBrokerbrokers:some_name_of_broker:retry_max_count:3# the same semantic as worker.retry_max_countretry_delay_s:10# the same semantic as worker.retry_delay_scontent_type:application/json# content typecompression:null# compression (null, application/x-gzip, application/x-bz2, application/x-lzma)import_template:"some.module:SomeClass"# required. the same semantic as worker.import_templateMessagemessages:name_of_message:import_template:"some.module:SomeClass"# required. the same semantic as worker.import_templatebuild in messages:messages:uni_cron_message:import_template:unipipeline.messages.uni_cron_message:UniCronMessageCLIunipipelineusage:unipipeline--help

UNIPIPELINE:simplewaytobuildthedeclarativeanddistributeddatapipelines.thisisclitoolforunipipeline

positionalarguments:{check,scaffold,init,consume,cron,produce}sub-commandscheckcheckloadingofallmodulesscaffoldcreateallmodulesandclassesifitisabsent.noargsinitinitializebrokertopicsforworkersconsumestartconsumingworkers.connecttobrokersandwaitingformessagescronstartcronjobs,Thatdefinedinconfigfileproducepublishmessagetobroker.sendittoworker

optionalarguments:-h,--helpshowthishelpmessageandexit--config-fileCONFIG_FILE,-fCONFIG_FILEpathtounipipelineconfigfile(default:./unipipeline.yml)--verbose[VERBOSE]verboseoutput(default:false)unipipeline checkusage: 
    unipipeline -f ./unipipeline.yml check
    unipipeline -f ./unipipeline.yml --verbose=yes check

check loading of all modules

optional arguments:
  -h, --help  show this help message and exitunipipeline initusage: 
    unipipeline -f ./unipipeline.yml init
    unipipeline -f ./unipipeline.yml --verbose=yes init
    unipipeline -f ./unipipeline.yml --verbose=yes init --workers some_worker_name_01 some_worker_name_02

initialize broker topics for workers

optional arguments:
  -h, --help            show this help message and exit
  --workers INIT_WORKERS [INIT_WORKERS ...], -w INIT_WORKERS [INIT_WORKERS ...]
                        workers list for initialization (default: [])unipipeline scaffoldusage: 
    unipipeline -f ./unipipeline.yml scaffold
    unipipeline -f ./unipipeline.yml --verbose=yes scaffold

create all modules and classes if it is absent. no args

optional arguments:
  -h, --help  show this help message and exitunipipeline consumeusage: 
    unipipeline -f ./unipipeline.yml consume
    unipipeline -f ./unipipeline.yml --verbose=yes consume
    unipipeline -f ./unipipeline.yml consume --workers some_worker_name_01 some_worker_name_02
    unipipeline -f ./unipipeline.yml --verbose=yes consume --workers some_worker_name_01 some_worker_name_02

start consuming workers. connect to brokers and waiting for messages

optional arguments:
  -h, --help            show this help message and exit
  --workers CONSUME_WORKERS [CONSUME_WORKERS ...], -w CONSUME_WORKERS [CONSUME_WORKERS ...]
                        worker list for consumingunipipeline produceusage: 
    unipipeline -f ./unipipeline.yml produce --worker some_worker_name_01 --data {"some": "json", "value": "for worker"}
    unipipeline -f ./unipipeline.yml --verbose=yes produce --worker some_worker_name_01 --data {"some": "json", "value": "for worker"}
    unipipeline -f ./unipipeline.yml produce --alone --worker some_worker_name_01 --data {"some": "json", "value": "for worker"}
    unipipeline -f ./unipipeline.yml --verbose=yes produce --alone --worker some_worker_name_01 --data {"some": "json", "value": "for worker"}

publish message to broker. send it to worker

optional arguments:
  -h, --help            show this help message and exit
  --alone [PRODUCE_ALONE], -a [PRODUCE_ALONE]
                        message will be sent only if topic is empty
  --worker PRODUCE_WORKER, -w PRODUCE_WORKER
                        worker recipient
  --data PRODUCE_DATA, -d PRODUCE_DATA
                        data for sendingunipipeline cronusage: 
    unipipeline -f ./unipipeline.yml cron
    unipipeline -f ./unipipeline.yml --verbose=yes cron

start cron jobs, That defined in config file

optional arguments:
  -h, --help  show this help message and exitContributingTODO LISTRPC Gateways: http, tcp, udpClose/Exit uni by call methodAsync producerCommon Error HandlingAsync get_answerServer of Message layoutPrometheus apireq/res Sdkrequest tasks result registryAsync consumerAsync by defaultMulti-threading start with run-groups