Package: labxpipe
Integrated withLabxDB: all required annotations (labels, strand, paired etc) are retrieved from LabxDB. This is optional.Based on existing robust technologies. No new language.LabxPipe pipelines are defined in JSON text files.LabxPipe is written in Python. Using norms, such as input and output filenames, insures compatibility between tasks.Simple and complex pipelines.By default, pipelines are linear (one step after the other).Branching is easily achieved be defining a previous step (usingstep_inputparameter) allowing users to create any dependency between tasks.Parallelized using robust asynchronous threads from the Python standard library.ExamplesSee JSON files inconfig/pipelinesof this repository.Pipeline JSON filemrna_seq.jsonmRNA-seq.mrna_seq_profiling_bam.jsonmRNA-seq. Genomic coverage profiles usingGeneAbacus. BAM and SAM outputs.mrna_seq_no_db.jsonmRNA-seq. NoLabxDB.mrna_seq_with_plotting.jsonmRNA-seq. Plotting non-mapped reads. Demonstratestep_input.mrna_seq_cufflinks.jsonmRNA-seq. Replaces GeneAbacus by Cufflinks.chip_seq.jsonChIP-seq.Bowtie2andSamtoolsto uniquify reads.chip_seq_user_function.jsonChIP-seq.Bowtie2andSamtoolsto uniquify reads. Genomic coverage profiles usingGeneAbacus. Peak-calling usingMACS3employing auser-defined step/function.Following demonstrates how to applymrna_seq.jsonpipeline. It requires:LabxDBFASTQ files for sample namedAGR000850andAGR000912/plus/data/seq/by_run/AGR000850
├── 23_009_R1.fastq.zst
└── 23_009_R2.fastq.zst
/plus/data/seq/by_run/AGR000912
├── 65_009_R1.fastq.zst
└── 65_009_R2.fastq.zstNote:mrna_seq_no_db.jsondemonstrates how to use LabxPipewithoutLabxDB: it only requires FASTQ files (inpath_seq_rundirectory, see above).Requirements:LabxDB. Alternatively,mrna_seq_no_db.jsondoesn't require LabxDB.ReadKneadto trim reads.STARand genome index in directory definedpath_star_index.GeneAbacusto count reads and generate genomic profile for tracks.Start pipeline:lxpiperun--pipelinemrna_seq.json\--worker2\--processor16Output is written inpath_outputdirectory.Create report:lxpipereport--pipelinemrna_seq.jsonReport filemrna_seq.xlsxshould be created in same directory asmrna_seq.json.Extract output file(s) to use them directly, for instance to load them in IGV. For example:To extract BAM files and rename them using the sample label:lxpipeextract--pipelinemrna_seq.json\--filesaligning,accepted_hits.sam.zst\--labelTo extract BigWig profile files and rename them using the sample label and reference in addition to the original filename used as filename suffix:lxpipeextract--pipelinemrna_seq.json\--filesprofiling,genome_plus.bw\--label\--reference\--suffixUse-d/--dry_runto test the extract command before applying it.Merge gene/mRNA counts generated byGeneAbacusincountingdirectory:lxpipemerge-count--pipelinemrna_seq.json\--stepcountingCreate a trackhub. Requirements:ChromosomeMappingsfile (to map chromosome names from Ensembl/NCBI to UCSC)Tabulated file (with chromosome name and length)Execute in a separate directory:lxpipetrackhub--runsAGR000850,AGR000912\--species_ucscdanRer11\--path_genome/plus/scratch/sai/annots/danrer_genome_all_ensembl_grcz11_ucsc_chroms_chrom_length.tab\--path_mapping/plus/scratch/sai/annots/ChromosomeMappings/GRCz11_ensembl2UCSC.txt\--input_sam\--bam_namesaccepted_hits.sam.zst\--make_config\--make_trackhub\--make_bigwig\--processor16Directory is ready to be shared by a web server for display in theUCSC genome browser.ConfigurationParameters can be definedglobally. See inconfigdirectory of this repository for examples.Writing pipelinesParameters are defined first globally (see above), then per pipeline, then per replicate/run, and then per step/function. The latest definition takes precedence:path_seq_rundefined in/etc/hts/labxpipe.jsonis used by default, but ifpath_seq_runis defined in the pipeline file, it will be used instead.Main parametersParameterTypenamestringpath_outputstringpath_seq_runstringpath_local_stepsstringpath_annotsstringpath_bowtie2_indexstringpath_bwa-mem2_indexstringpath_minimap2_indexstringpath_star_indexstringfastq_exts[]stringsadaptors{}logging_levelstringrun_refs[]stringsreplicate_refs[]stringsref_info_source[]stringsref_infos{}analysis[{}, {}, ...]Parameters for all stepsParameterTypestep_namestringstep_functionstringstep_descstringforcebooleanStep-specific parametersStepSynonymParameterTypereadkneadpreparingoptions[]stringsops_r1[{}, {}, ...]ops_r2[{}, {}, ...]plot_fastq_inbooleanplot_fastqbooleanfastq_outbooleanzip_fastq_outstringbowtie2genomic_aligningoptions[]stringsindexstringoutputstringoutput_unfilteredstringcompress_sambooleancompress_sam_cmdstringcreate_bam◆booleanindex_bam◆booleanbwa-mem2options[]stringsindexstringoutputstringcompress_outputbooleancompress_output_cmdstringcreate_bam◆booleanindex_bam◆booleanminimap2options[]stringsindexstringoutputstringcompress_outputbooleancompress_output_cmdstringcreate_bam◆booleanindex_bam◆booleanstaraligningoptions[]stringsindexstringoutput_type[]stringscompress_sambooleancompress_sam_cmdstringcompress_unmappedbooleancompress_unmapped_cmdstringcufflinksoptions[]stringsinputs[{}, {}, ...]features[{}, {}, ...]geneabacuscountingoptions[]stringsinputs[{}, {}, ...]path_annotsstringfeatures[{}, {}, ...]samtools_sortoptions[]stringssort_by_name_bambooleansamtools_uniquifyoptions[]stringssort_by_name_bambooleanindex_bambooleancleaningsteps[{}, {}, ...]◆ indicates exclusive options. For example, eithercreate_bamorindex_bamcan be used, but not both.Sample-specific parameters. Automatically populated if using LabxDB or sourced fromref_infos. These parameters can be changed manually in any step (for example settingpairedtofalsewill ignore second reads in that step).ParameterTypelabel_shortstringpairedbooleandirectionalbooleanr1_strandstringquality_scoresstringUser-defined stepIn addition to the provided steps/functions, i.e.bowtie2,starorgeneabacus, users can defined their own step, usable in the LabxPipe pipelines. LabxPipe will import user-defined steps:Written in PythonOne step per file with the.pyextension located in the directory defined bypath_local_stepsEach step defined in individual file requires:Afunctionsvariable listing the step name(s)A function namedrunwith the 3 parameterspath_in,path_outandparamsFor example:functions=['macs3']defrun(path_in,path_out,params):...Example of a user-defined function providing peak-calling usingMACS3is available inconfig/user_steps/macs3.pyin this repository.Example of a pipeline using theMACS3step is available inconfig/pipelines/chip_seq_user_function.jsonin this repository.Demultiplexing sequencing reads:lxpipe demultiplexDemultiplex reads based on barcode sequences from theSecond barcodefield inLabxDBDemultiplexing usingReadKnead. The most important for demultiplexing is the ReadKnead pipeline. Pipelines are identified using theAdapter 3'field in LabxDB.Example for simple demultiplexing. The first nucleotides at the 5' end of read 1 are used as barcodes (theAdapter 3'field is set tosRNA 1.5in LabxDB for these samples) with the following pipeline:{"sRNA 1.5":{"R1":[{"name":"demultiplex","end":5,"max_mismatch":1}],"R2":null}}The barcode sequences are added by LabxPipe using theSecond barcodefield inLabxDB.Example for iCLIP demultiplexing. InVejnar et al., iCLIP is demultiplexed (theAdapter 3'field is set toTruSeq-DMS+A Indexin LabxDB for these samples) using the following pipeline:{"TruSeq-DMS+A Index":{"R1":[{"name":"clip","end":5,"length":4,"add_clipped":true},{"name":"trim","end":3,"algo":"bktrim","min_sequence":5,"keep":["trim_exact","trim_align"]},{"name":"length","min_length":6},{"name":"demultiplex","end":3,"max_mismatch":1,"length_ligand":2},{"name":"length","min_length":15}],"R2":null}}Pipeline is stored indemux_truseq_dms_a.json. The barcode sequences are added by LabxPipe using theSecond barcodefield inLabxDB. (NB: published demultiplexed data were generated using"algo": "align"with a minimum score of 80 instead of"algo": "bktrim")Then pipeline was tested running:lxpipedemultiplex--bulkHHYLKADXX\--path_demux_opsdemux_truseq_dms_a.json\--path_seq_preparedprepared\--demux_nozip\--processor1\--demux_verbose_level20\--no_readonlyThis output isvery verbose: for every read, output from every step of the demultiplexing pipeline is reported. To get consistent output,--processormust be set to1. Output is written in local directoryprepared.And finally, once pipeline is validated (data is written inpath_seq_prepareddirectory, seehere):lxpipedemultiplex--bulkHHYLKADXX\--path_demux_opsdemux_truseq_dms_a.json\--processor10LicenseLabxPipeis distributed under the Mozilla Public License Version 2.0 (see /LICENSE).Copyright © 2013-2023 Charles E. Vejnar