Package: pypette
pypette (to be read as pipette) is a module which makes building pipelines
ridiculously simple, allowing users to control the flow with minimal
instructions.FeaturesRidiculously simple interface.Ability to view pipeline structure within the comfort of a terminal.Run pipeline in exception resilient way if needed.Create dependencies on pipelines easily.Generate a easy to view/understand report within the comfort of a terminal.SetupUsing pippipinstallpypetteDirectly from the repositorygitclonehttps://github.com/csurfer/pypette.gitpythonpypette/setup.pyinstallDocumentationDetailed documentation can be found athttps://csurfer.github.io/pypetteStructuresJobThe basic unit of execution, say a python method or a callable.frompypetteimportJobdefprint_hello():print("Hello!")defprint_hello_msg(msg):print("Hello "+msg+"!")# Job without argumentsj1=Job(print_hello)# Job with arguments specified as argument listj2=Job(print_hello_msg,args=("pypette is simple",))# Job with arguments specified as key word argumentsj3=Job(print_hello_msg,kwargs={"msg":"pypette is simple"})BashJobThe basic unit of execution, which runs a bash command.frompypetteimportBashJob# Job with bash commandsb1=BashJob(['ls','-l'])b2=BashJob(['pwd'])PipeStructure to specify the flow in which the jobs need to be executed. The whole
interface consists of only 4 methods.frompypetteimportPipe# 1. Create a new Pipep=Pipe('TestPipe')# 2. Add jobs to execute. (Assuming job_list is a list of python/bash jobs)# To run the jobs in job_list in order one after the other where each job# waits for the job before it to finish.p.add_jobs(job_list)# To run the jobs in job_list parallelly and run the next step only after# all jobs in job list finish.p.add_jobs(job_list,run_in_parallel=True)# Add jobs in a builder format.p.add_stage(job1).add_stage(job2)# To add jobs in series.p.add_stage(job1,job2)# To add jobs in parallel.Building complex pipelinesJobs submitted to pipeline should be callables i.e. structures which can be
run. This means python methods, lambdas etc qualify.What about Pipe itself?Of course, it is a callable and you can submit a pipe object to be run along
with regular jobs. This way you can build small pipelines which achieve a
specific task and then combine them to create more complex pipelines.frompypetteimportBashJob,Job,Pipedefwelcome():print("Welcome user!")defhavefun():print("Have fun!")defgoodbye():print("Goodbye!")# Build a simple pipelinep1=Pipe('Fun')p1.add_jobs([Job(havefun),])# Include simple pipeline into a complicated pipelinep2=Pipe('Overall')p2.add_jobs([Job(welcome),p1,Job(goodbye),BashJob(['ls','-l']),BashJob(['pwd'])])p2.run()# This first runs welcome, then runs p1 pipeline then runs goodbye.Example pipelineAn example pipeline and its code are included inexamplesfolder.Visualizing the pipeline using graph()Pipeline objects have a method calledgraph()which helps visualize the
pipeline within the comfort of your terminal. The graph is recursive in nature
and it visualizes everything that will be run if we callrun()on the pipe
object.Visualizing the top-level pipeline inexamples/basic.pyled to the
following visualization.Running the entire pipeline.The only thing you need to do at this point to run the entire pipeline is to
callrun()on your pipeline object.Reporting the entire pipeline.The only thing you need to do at this point to get the report of entire
pipeline is to callreport()on your pipeline object.ContributingBug Reports and Feature RequestsPlease useissue trackerfor reporting bugs or feature requests.DevelopmentPull requests are most welcome. Kindly follow the steps suggested below:Checkout the repository.Make your changes and add/update relavent tests.Install`poetry`using`pip install poetry`.Run`poetry install`to create project’s virtual environment.Run tests using`poetry run tox`(Any python versions which you don’t have checked out will fail this). Fix failing tests and repeat.Make documentation changes that are relavant.Install`pre-commit`using`pip install pre-commit`and run`pre-commit run –all-files`to do lint checks.Generate documentation using`poetry run sphinx-build -b html docs/ docs/_build/html`.Generate`requirements.txt`for automated testing using`poetry export –dev –without-hashes -f requirements.txt > requirements.txt`.Commit the changes and raise a pull request.Buy the developer a cup of coffee!If you found the utility helpful you can buy me a cup of coffee using