Package: piperna
piperna==========A python wrapper for processing of bulk RNA seq dataRequirementsPython > 3.5 (piperna uses the 'six' package but will attempt to install if not already installed)Computing cluster with PBS or SLURMModules installed for python, STAR or kallisto,R Modules for GenomicAlignments, rtracklayer, and Rsamtools if running SUMMARIZEInstallationInstallation can probably be done correctly many different ways.  Here are the methods that have worked for us.  We recommend that piperna be installed with pipx.At SCRI do the followingmoduleloadpython
python3-mpipinstall--userpipx
python3-mpipxensurepath
pipxinstall--include-deps--pip-args'--trusted-host pypi.org --trusted-host files.pythonhosted.org'pipernaAt the FHCRC do the following...moduleloadPython/3.6.7-foss-2016b-fh1
python3-mpipinstall--userpipx
python3-mpipxensurepath
pipxinstall--include-depspipernaYou should then be able to test installation by calling piperna.  After running the folllowing, you should see the help screen displayed.pipernaUsagepipernausage:AwrapperforrunningRNASeqAlignment[-h][--bam_folderBAM_FOLDER][--fastq_folderFASTQ_FOLDER][--organized_by{folder,file}][--genome_keyGENOME_KEY][--split_charSPLIT_CHAR][--R1_charR1_CHAR][--R2_charR2_CHAR][--extEXT][--selectSELECT][--sample_flagSAMPLE_FLAG][--runsheetRUNSHEET][--typeofseq{single,pe}][--software{STAR,kallisto}][--outputOUTPUT][--debug][--cluster{PBS,SLURM}][--userUSER][--threadsTHREADS][--gb_ramGB_RAM][--additional_headerADDITIONAL_HEADER][--mflMFL][--sflSFL][--count][--installINSTALL][--outSAMtypeOUTSAMTYPE][--addSTARstringADDSTARSTRING][--log_prefixLOG_PREFIX][--flow_cell_foldersFLOW_CELL_FOLDERS][--verbose]{MAKERUNSHEET,ALIGN,SUMMARIZE,CONCATFASTQ,GENOMESFILE,ENVIRONSFILE,UNBAM}positionalarguments:{MAKERUNSHEET,ALIGN,SUMMARIZE,CONCATFASTQ,GENOMESFILE,ENVIRONSFILE,UNBAM}arequiredstringdenotingsegmentofpipelinetorun.1)"MAKERUNSHEET"-toparseafolderoffastqs;2)"ALIGN"-toperformalignmentusingSTARorKALLISTO;3)"SUMMARIZE"-tosummarizeandcountreads,4)"CONCATFASTQ"-functiontoconcatenatefastqfiles-i.e.forSRAupload;5)"GENOMESFILE"-printlocationofandcatgenomes.jsonfile;6)"ENVIRONSFILE"-printlocationofandcatenvirons.jsonfile;7)"UNBAM"-convertbamtofastq

optionalarguments:-h,--helpshowthishelpmessageandexit--bam_folderBAM_FOLDER,-bfBAM_FOLDERForUNBAMonly:Pathnameofbamlocations--fastq_folderFASTQ_FOLDER,-fqFASTQ_FOLDERForMAKERUNSHEETonly:Pathnameoffastqfolder--organized_by{folder,file},-b{folder,file}Optiontospecifyhowfastqorunbamfolderisorganized--genome_keyGENOME_KEY,-gkGENOME_KEYForMAKERUNSHEETonly:abbreviationtouse"installed"genomesintherunsheet(SeeREADME.mdformoredetails--split_charSPLIT_CHAR,-scSPLIT_CHARCharacterbywhichtosplitthefastqfilenameintosamples,OPTIONALandforMAKERUNSHEETonly--R1_charR1_CHAR,-r1cR1_CHARCharacterbywhichtosplitthefastqfilenameintoread1,OPTIONALandforMAKERUNSHEETonly--R2_charR2_CHAR,-r2cR2_CHARCharacterbywhichtosplitthefastqfilenameintoread2,OPTIONALandforMAKERUNSHEETonly--extEXT,-eEXTsuffixoffastqfiles,OPTIONALandforMAKERUNSHEETonly--selectSELECT,-sSELECTToonlyruntheselectedrowintherunsheet,OPTIONALandforMAKERUNSHEETonly--sample_flagSAMPLE_FLAG,-fSAMPLE_FLAGFORMAKERUNSHEETonlystringtoidentifysamplesofinterestinafastqfolder--runsheetRUNSHEET,-rRUNSHEETtab-delimfilewithsamplefieldsasdefinedinthescript.-ornameofrunsheettosaveifusingMAKERUNSHEET--typeofseq{single,pe},-t{single,pe}Typeofsequencingperformed-REQUIREDforMAKERUNSHEET,UNBAMandCONCATFASTQ--software{STAR,kallisto},-so{STAR,kallisto}Tosetdesiredsoftware,requiredandusedforMAKERUNSHEETonly--outputOUTPUT,-oOUTPUTTosetoutputpath,requiredforMAKERUNSHEET,UNBAM;OPTIONALforSUMMARIZE--defaultforSUMMARIZEis./SummarizedExperiment.RDS--debug,-dToprintcommands(Fortestingflow)--cluster{PBS,SLURM},-c{PBS,SLURM}Clustersoftware.OPTIONALCurrentlysupported:PBSandSLURM--userUSER,-uUSERuserforsubmittingjobs-defaultstousername.OPTIONAL--threadsTHREADS,-thTHREADSTosetnumberofcores--gb_ramGB_RAM,-gbGB_RAMTosetgb_ram--additional_headerADDITIONAL_HEADER,-ahADDITIONAL_HEADERAdditionalbashheaderlines--mflMFL,-mfMFLMeanfragmentlength(kallistoONLY)--sflSFL,-sfSFLSDfragmentlength(kallistoONLY)--count,-coRunCount(STAROnly)--installINSTALL,-iINSTALLFORGENOMESFILE:locationoffiletoinstallasanewgenomes.jsonfile,existinggenomes.jsonwillbeerased--outSAMtypeOUTSAMTYPE,-stOUTSAMTYPETodefinetypeofSAM/BAMoutput(STAROnly)--addSTARstringADDSTARSTRING,-aADDSTARSTRINGAdditionalSTARargumentstoberunonalljobsinrunsheet(STAROnly)--log_prefixLOG_PREFIX,-lLOG_PREFIXPrefixspecifyinglogfilesforpipernaoutputfromhenipipecalls.OPTIONAL--flow_cell_foldersFLOW_CELL_FOLDERS,-fcFLOW_CELL_FOLDERSForCONCATFASTQonly:Comma-sepratedlocationofflowcellfolders-i.e.asoutputfromCellRangerREQUIREDforCONCATFASTQ--verbose,-vRunwithsomeadditionalouput-notmuchthough...OPTIONALRunsheetThe runsheet is the brains of the piperna workflow.  You can make a runsheet using the MAKERUNSHEET command.  This command will parse a directory of fastq folder (specified using the -fq flag; fastq files should be organized in subfolders named by sample) and will find fastq mates (R1 and R2 - Currently only PE sequencing is supported).  Running piperna MAKERUNSHEET will find and pair these fastqs for you and populate the runsheet with genome index locations (see below) and output filenames with locations as specified using the -o flag.  Note that piperna output will default to the current working directory if no location is otherwise specified.  There is an option for selecting only folders that contain a specific string (using the -sf flag).After generation of a runsheet (csv file), you should take a look at it in Excel or Numbers to make sure things look okay...Here are the columns that you can include.  Order is irrelevant.  Column names (headers) exactly as written below are required.Example Runsheetabsolute pathnames are required for runsheetssampleindexfastq1fastq2outputsoftwaregtfmys1pathpathpathpathSTARpathmys2pathpathpathpathSTARpath'sample' name of the sample REQUIRED.'index' location of the indexed fasta file REQUIRED.'fastq1' a tab seperated string of filenames denoting location of all R1 files for a sample REQUIRED if paired end.'fastq2' a tab seperated string of filenames denoting location of all R2 files for a sample REQUIRED if paired end.'fastqs' a tab seperated string of filenames can be used for single end reads REQUIRED if single end.'output' name of the location for the aligned and sorted bam file.'software' either 'STAR' or 'kallisto'.  REQUIRED'gtf' a location for annotation file in gtf format.  REQUIRED for SUMMARIZE.Genomes and adding genome locationsyou should have a previously indexed (by the software package of your choosing) location of your genome accessible to piperna.  This location is referred to in piperna as the 'index'.piperna provides an easy way to add these locations to your system for repeated use using the --genome_key (-gk) option during MAKERUNSHEET commands.  A file called genomes.json can be found in the 'data' directory of the piperna install folder.  This file can be edited to include those locations you want to regularly put in the runsheet.  The following shows an example of a genomes.json file.  The files "top level" is a name that can be used in the --genome_key field (-gk) during runsheet generation to populate the columns of the runsheet with locations associated with that genome_key.  The 'default' key will be used when no genome_key is specified.{"default":{"fasta":"/path/path/hg38/STAR_index","gtf":"/path/path/hg38/hg38.gtf"},"default_kallisto":{"fasta":"/path/path/hg38/kallisto_index","gtf":"/path/path/hg38/hg38.gtf"}}Cluster specific parametersPiperna could be modified to run on any cluster  Templates are provided for SLURM and PBSLocation of the environs file can be found by running:pipernaENVIRONSFILEAs an example the SLURM script generation  environs is shown below.  Broadly it's parameters is divided into three groups: popen, script_lines, and resources.popen - an entry that lists the command to open a submission job ('sbatch' in this case)
script_lines - provides the header of the bash script
resources - gives job-specific parameters{"SLURM":{"popen":["sbatch"],"script_lines":{"1":["#!/bin/bash\n#SBATCH --output=outtmp\n#SBATCH --error=errtmp",""],"2":["#SBATCH --job-name=<--0-->","JOB_NAME"],"3":["#SBATCH --ntasks=1\n#SBATCH --cpus-per-task=<--0-->","THREADS"],"4":["#SBATCH --mem-per-cpu=<--0-->000","RAM"],"5":["<--0-->","HEADER"],"6":["{\necho '<--1-->';<--0-->;<--1-->;} 2>&1 | tee <--2-->","MODULES|COMMAND|TEMP_LOG_FILE"],"7":["sed -e 's@^@[PIPERNA-<--0-->] JOB: <--1-->:\t\t@' <--2--> >> <--3-->","TIME|JOB_NAME|TEMP_LOG_FILE|LOG_FILE"],"8":["rm <--0-->\n","TEMP_LOG_FILE"]},"resources":{"PIPERNA_STAR":{"ram":"8","threads":"4","modules":"module load STAR/2.7.6a-foss-2019b"},"PIPERNA_KALLISTO":{"ram":"8","threads":"4","modules":"module load kallisto"},"PIPERNA_SUMMARIZE":{"ram":"16","threads":"8","modules":"module load R"},"PIPERNA_UNBAM":{"ram":"2","threads":"8","modules":"module load SAMtools\nmodule load BEDTools"}}}}Doing a piperna runSay your fastqs live within within subfolders of a folder 'fastq' in the folder 'data'.  So if you were to...cd/data/fastq
ls... you'd get a bunch of folders, each of which would be filled with fastqs.  Each folder name should correspond to a sample name.To run piperna, do the following...Make a new output directory 'piperna'.Go into that directory and make a runsheet pointing to the fastq folder i.e. the folder level above.  (at the command line, piperna is cool with either relative or absolute pathnames; but as stated earlier, absolute pathnames are required for the runsheet.)Optionally you can only select directories of fastq files that contain in their name the string denoted using the -sf flag.After inspecting and completing the runsheet, run ALIGN, NORM, and SEACR.cd..
mkdirpipernacdpiperna
pipernaMAKERUNSHEET-fq../fastq-sfMySampleDirectoriesStartWithThisString-o.
pipernaALIGN-rrunsheet.csv
pipernaSUMMARIZE-rrunsheet.csv-oSummarizedExperiment.RDSInterfacing with DESeq2After running piperna, the SummarizedExperiment.RDS file can be input directly into DESeq2 like thisse<-readRDS(file.path(DATA_DIR,"summarizedExperiment.RDS"))colnames(se)<-colData(se)$samplese$group<-c("GroupA","GroupA","GroupB","GroupB","GroupB","GroupA")#as an exampledds<-DESeqDataSet(se,design=~group)keep<-rowSums(counts(dds))>=10dds<-dds[keep,]dds<-DESeq(dds,parallel=T)AcknowledgementsWritten by Scott Furlan.