Package: pyppl
PyPPL- APythonPiPeLine frameworkDocumentation|API|Change logFeaturesProcess caching.Process reusability.Process error handling.Runner customization.Easy running profile switching.Plugin system.InstallationpipinstallPyPPLPlugin gallery(*) shipped withPyPPLpyppl_report: Generating reports for PyPPL pipelinespyppl_flowchart: Generating flowchart for PyPPLpyppl_export*: Exporting outputs generated by PyPPL pipelinepyppl_echo*: Echoing script output to PyPPL logs"pyppl_rich*: Richer information in logs for PyPPLpyppl_strict*: More strict check of job success for PyPPLpyppl_lock*: Preventing running processes from running again for PyPPLpyppl_annotate: Adding long description/annotation for processespyppl_require: Checking and installing requirements for processespyppl_jobtime: Job running time statistics for PyPPLpyppl_notify: Email notifications for PyPPLpyppl_runcmd: Allowing to run local command before and after each process for PyPPLpyppl_runners: Common runners for PyPPLWriting pipelines with predefined processesLet's say we are implementing theTCGA DNA-Seq Re-alignment Workflow(The very left part of following figure).
For demonstration, we will skip the QC and the co-clean parts here.demo.py:frompypplimportPyPPL,Channel# import predefined processesfromTCGAprocsimportpBamToFastq,pAlignment,pBamSort,pBamMerge,pMarkDups# Load the bam filespBamToFastq.input=Channel.fromPattern('/path/to/*.bam')# Align the reads to reference genomepAlignment.depends=pBamToFastq# Sort bam filespBamSort.depends=pAlignment# Merge bam filespBamMerge.depends=pBamSort# Mark duplicatespMarkDups.depends=pBamMerge# Export the resultspMarkDups.config.export_dir='/path/to/realigned_Bams'# Specify the start process and run the pipelinePyPPL().start(pBamToFastq).run()Implementing individual processesTCGAprocs.py:frompypplimportProcpBamToFastq=Proc(desc='Convert bam files to fastq files.')pBamToFastq.input='infile:file'pBamToFastq.output=['fq1:file:{{i.infile | stem}}_1.fq.gz','fq2:file:{{i.infile | stem}}_2.fq.gz']pBamToFastq.script='''bamtofastq collate=1 exclude=QCFAIL,SECONDARY,SUPPLEMENTARY\filename= {{i.infile}} gz=1 inputformat=bam level=5\outputdir= {{job.outdir}} outputperreadgroup=1 tryoq=1\outputperreadgroupsuffixF=_1.fq.gz\outputperreadgroupsuffixF2=_2.fq.gz\outputperreadgroupsuffixO=_o1.fq.gz\outputperreadgroupsuffixO2=_o2.fq.gz\outputperreadgroupsuffixS=_s.fq.gz'''pAlignment=Proc(desc='Align reads to reference genome.')pAlignment.input='fq1:file, fq2:file'#                             name_1.fq.gz => name.bampAlignment.output='bam:file:{{i.fq1 | stem | stem | [:-2]}}.bam'pAlignment.script='''bwa mem -t 8 -T 0 -R <read_group> <reference> {{i.fq1}} {{i.fq2}} |\samtools view -Shb -o {{o.bam}} -'''pBamSort=Proc(desc='Sort bam files.')pBamSort.input='inbam:file'pBamSort.output='outbam:file:{{i.inbam | basename}}'pBamSort.script='''java -jar picard.jar SortSam CREATE_INDEX=true INPUT={{i.inbam}}\OUTPUT={{o.outbam}} SORT_ORDER=coordinate VALIDATION_STRINGENCY=STRICT'''pBamMerge=Proc(desc='Merge bam files.')pBamMerge.input='inbam:file'pBamMerge.output='outbam:file:{{i.inbam | basename}}'pBamMerge.script='''java -jar picard.jar MergeSamFiles ASSUME_SORTED=false CREATE_INDEX=true\INPUT={{i.inbam}} MERGE_SEQUENCE_DICTIONARIES=false OUTPUT={{o.outbam}}\SORT_ORDER=coordinate USE_THREADING=true VALIDATION_STRINGENCY=STRICT'''pMarkDups=Proc(desc='Mark duplicates.')pMarkDups.input='inbam:file'pMarkDups.output='outbam:file:{{i.inbam | basename}}'pMarkDups.script='''java -jar picard.jar MarkDuplicates CREATE_INDEX=true INPUT={{i.inbam}}\OUTPUT={{o.outbam}} VALIDATION_STRINGENCY=STRICT'''Each process is indenpendent so that you may also reuse the processes in other pipelines.Pipeline flowchart# When try to run your pipline, instead of:#   PyPPL().start(pBamToFastq).run()# do:PyPPL().start(pBamToFastq).flowchart().run()Then an SVG file endswith.pyppl.svgwill be generated under current directory.
Note that this function requiresGraphvizandgraphviz for python.See plugindetails.Pipeline reportSee plugindetailspPyClone.report="""## {{title}}PyClone[1] is a tool using Probabilistic model for inferring clonal population structure from deep NGS sequencing.![Similarity matrix]({{path.join(job.o.outdir, "plots/loci/similarity_matrix.svg")}})```tablecaption: Clustersfile: "{{path.join(job.o.outdir, "tables/cluster.tsv")}}"rows: 10```[1]: Roth, Andrew, et al. "PyClone: statistical inference of clonal population structure in cancer." Nature methods 11.4 (2014): 396."""# or use a template filepPyClone.report="file:/path/to/template.md"PyPPL().start(pPyClone).run().report('/path/to/report',title='Clonality analysis using PyClone')Full documentationReadTheDocs