Package: henipipe
henipipe==========Version 2.4.14A python wrapper for processing of sequencing data generated using CutnRun or CutnTag (developed by the Henikoff lab FHCRC).  Now with a single-cell option ('SC') for processing CutnTag data generated using the iCell8 platform (Takara).New in version 2.4.xAdded a --version functionIn version 2.4.9 - environs.json was properly configured for PBS and see below for how to run henipipe with a project code (PBS only)New in version 2.3Henipipe adds a new function, DEDUP.  DEDUP will remove presumed PCR duplicates from the bed file but leave a record of how many were detected.  The DEDUP funciton is meant to be used after ALIGN and before other downstream functions.  It will replace the bed file with a collapsed bedfile with a new column that counts the number of presumed duplicates found.  Note that read pairs will be collapsed regardless of strand and all strand data will be changed to positive.Henipipe adds a new function, BLACKLIST.  BLACKLIST will remove bed entries that overlap with a user-specified blacklist file designated with the '-bl' flag.Changes to MACS2 function were made in this version to allow custom parameters to be passed to MACS2 peak calling using the -M2p flag.Fixed a bug in SCALE functionNew in version 2.2Henipipe now will now make Bigwig files from bedgraphsNew in version 2.0Henipipe now will process single cell data generated by the Takara iCell8 platform (See instructions below).  The pipeline will output the followingA bigwig file of aggregated SC dataSEACR peak calls (v1.3 - v1.4 coming soon) using both the 'relaxed' and 'stringent' SEACR modesA bedgraph fileA fragments.tsv.gz file for input into single cell software such as Signac and ArchRA fragments.tsv.gz.tbi (Tabix indexed file of #4)RequirementsPython > 3.5 (henipipe uses the 'six' package but will attempt to install if not already installed)Computing cluster with PBS or SLURMModules installed for python, bowtie2, samtools, bedtools, RMACS2 is required for MACS2 functionModules installed for kenttools and htslib are required for SC optionInstallationInstallation can probably be done correctly many different ways.  Here are the methods that have worked for us.  We recommend that henipipe be installed with pipx.At SCRI do the followingmoduleloadpython
python3-mpipinstall--userpipx
python3-mpipxensurepath
pipxinstall--include-deps--pip-args'--trusted-host pypi.org --trusted-host files.pythonhosted.org'henipipeAt the FHCRC do the following...moduleloadPython/3.7.4-foss-2019b-fh1
python3-mpipinstall--userpipx
python3-mpipxensurepath
pipxinstall--include-depshenipipeYou should then be able to test installation by calling henipipe.  After running the folllowing, you should see the help screen displayed.henipipeHelp (-h)usage:Awrapperforrunninghenipipe[-h][--sample_flagSAMPLE_FLAG][--fastq_folderFASTQ_FOLDER][--trim_folderTRIM_FOLDER][--trim_argsTRIM_ARGS][--organized_by{folder,file}][--genome_keyGENOME_KEY][--split_charSPLIT_CHAR][--R1_charR1_CHAR][--R2_charR2_CHAR][--extEXT][--filter_highFILTER_HIGH][--filter_lowFILTER_LOW][--outputOUTPUT][--runsheetRUNSHEET][--log_prefixLOG_PREFIX][--selectSELECT][--debug][--bowtie_flagsBOWTIE_FLAGS][--cluster{PBS,SLURM,local}][--threadsTHREADS][--gb_ramGB_RAM][--installINSTALL][--norm_method{coverage,read_count,spike_in}][--userUSER][--SEACR_norm{non,norm}][--SEACR_fdrSEACR_FDR][--SEACR_stringency{stringent,relaxed}][--keep_files][--verbose]{SC,MAKERUNSHEET,ALIGN,SCALE,MERGE,SEACR,MACS2,AUC,GENOMESFILE,FASTQC,TRIM}positionalarguments:{SC,MAKERUNSHEET,ALIGN,SCALE,MERGE,SEACR,MACS2,AUC,GENOMESFILE,FASTQC,TRIM}arequiredstringdenotingsegmentofpipelinetorun.1)"MAKERUNSHEET"-toparseafolderoffastqs;2)"ALIGN"-toperformalignmentusingbowtieandoutputbedfiles;3)"SCALE"-tonormalizedatatoreference(spikein);4)"MERGE"-tomergebedgraphs5)"SEACR"-toperformSEACR;6)"MACS"-toperformMACS2;7)"AUC"-tocalculateAUCbetweennormalizedbedgraphusingapeakfile;8)"GENOMESFILE"-printlocationofgenomes.jsonfile;9)"FASTQC"-runfastqconcluster;10)runtrimmotaticoncluster;optionalarguments:-h,--helpshowthishelpmessageandexit--sample_flagSAMPLE_FLAG,-sfSAMPLE_FLAGFORMAKERUNSHEETonlystringtoidentifysamplesofinterestinafastqfolder--fastq_folderFASTQ_FOLDER,-fqFASTQ_FOLDERForSCandMAKERUNSHEETonly:Pathnameoffastqfolder(filesmustbeorganizedinfoldersnamedbysample)--trim_folderTRIM_FOLDER,-tfTRIM_FOLDERREQURIED,ForTRIMonly:Pathnameofoutputfolder;Notethatalltrimmedfastqswillbeplacedinthesamefolder--trim_argsTRIM_ARGS,-taTRIM_ARGSOPTIONAL,ForTRIMonly:Argstopasstotrimmomatic--organized_by{folder,file},-by{folder,file}Optiontospecifyhowfastqorunbamfolderisorganized--genome_keyGENOME_KEY,-gkGENOME_KEYForMAKERUNSHEETonly:abbreviationtouse"installed"genomesintherunsheet(SeeREADME.mdformoredetails--split_charSPLIT_CHAR,-scSPLIT_CHARCharacterbywhichtosplitthefastqfilenameintosamples,OPTIONALandforMAKERUNSHEETonly--R1_charR1_CHAR,-r1cR1_CHARCharacterbywhichtosplitthefastqfilenameintoread1,OPTIONALandforMAKERUNSHEETonly;default="_R1_"--R2_charR2_CHAR,-r2cR2_CHARCharacterbywhichtosplitthefastqfilenameintoread2,OPTIONALandforMAKERUNSHEETonly;default="_R2_"--extEXT,-eEXTsuffixoffastqfiles,OPTIONALandforMAKERUNSHEETonly--filter_highFILTER_HIGH,-fhFILTER_HIGHForALIGNonly:upperlimitoffragmentsizetoexclude,defaultsisnoupperlimit.OPTIONAL--filter_lowFILTER_LOW,-flFILTER_LOWForALIGNonly:lowerlimitoffragmentsizetoexclude,defaultsisnolowerlimit.OPTIONAL--outputOUTPUT,-oOUTPUTPathnametooutputfolder(notethisfoldermustexistalready!!),Defaultstocurrentdirectory--runsheetRUNSHEET,-rRUNSHEETtab-delimfilewithsamplefieldsasdefinedinthescript.-REQUIREDforalljobsexceptMAKERUNSHEET--log_prefixLOG_PREFIX,-lLOG_PREFIXPrefixspecifyinglogfilesforhenipipeoutputfromhenipipecalls.OPTIONAL--selectSELECT,-sSELECTToonlyruntheselectedrowintherunsheet,OPTIONAL--debug,-dToprintcommands(Fortestingflow).OPTIONAL--bowtie_flagsBOWTIE_FLAGS,-bBOWTIE_FLAGSForALIGN:bowtieflags,OPTIONAL--cluster{PBS,SLURM,local},-c{PBS,SLURM,local}Clustersoftware.OPTIONALCurrentlysupported:PBS,SLURMandlocal--threadsTHREADS,-tTHREADSnumberofthreads;default:8--gb_ramGB_RAM,-gbGB_RAMgigabytesofRAMperthread--installINSTALL,-iINSTALLFORGENOMESFILE:locationoffiletoinstallasanewgenomes.jsonfile,existinggenomes.jsonwillbeerased--norm_method{coverage,read_count,spike_in},-n{coverage,read_count,spike_in}ForALIGNandSCALE:Normalizationmethod,by"read_count","coverage",or"spike_in".Ifmethodis"spike_in",HeniPipewillaligntothespike_inreferencegenomeprovidedinrunsheet.OPTIONAL--userUSER,-uUSERuserforsubmittingjobs-defaultstousername.OPTIONAL--SEACR_norm{non,norm},-Sn{non,norm}ForSEACR:Normalizationmethod;defaultis"non"-normalized,select"norm"tonormalizeusingSEACR.OPTIONAL--SEACR_fdrSEACR_FDR,-SfSEACR_FDRForSEACR:UsedtosetFDRthresholdwhencontrolisnotused.OPTIONAL--SEACR_stringency{stringent,relaxed},-Ss{stringent,relaxed}FORSEACR:Defaultwillrunas"stringent",otheroptionis"relaxed".OPTIONAL--keep_files,-kFORALIGN:usethisflagtoturnoffpiping(Willgenerateallfiles).--verbose,-vRunwithsomeadditionalouput-notmuchthough...OPTIONALRunsheet (workflow for bulk processing only)The runsheet is the brains of the henipipe workflow. You can make a runsheet using the MAKERUNSHEET command.  This command will parse a directory of fastq folder (specified using the -fq flag; fastq files should be organized in subfolders named by sample) and will find fastq mates (R1 and R2 - Currently only PE sequencing is supported).  Running henipipe MAKERUNSHEET will find and pair these fastqs for you and populate the runsheet with genome index locations (see below) and output filenames with locations as specified using the -o flag.  Note that thenipie output will default to the current working directory if no location is otherwise specified.  There is an option for selecting only folders that contain a specific string (using the -sf flag).After generation of a runsheet (csv file), you should take a look at it in Excel or Numbers to make sure things look okay...Here are the columns that you can include.  Order is irrelevant.  Column names (headers) exactly as written below are required.Example Runsheetabsolute pathnames are required for runsheetssamplefastaspikein_fastafastq1fastq2bed_outspikein_bed_outgenome_sizesbedgraphSEACR_keySEACR_outmys1pathpathpathpathpathpathpathpath4JSpathmys2pathpathpathpathpathpathpathpath4JS_CONTROLpath'sample' name of the sample REQUIRED.'fasta' location of the Bowtie2 indexed fasta file REQUIRED.'spikein_fasta' location of the Bowtie2 indexed fasta file for spike_in normalization OPTIONAL.'fastq1' a tab seperated string of filenames denoting location of all R1 files for a sample REQUIRED.'fastq2' a tab seperated string of filenames denoting location of all R2 files for a sample REQUIRED.'bed_out' name of the location for the aligned and sorted bam file REQUIRED.'spikein_bed_out' name of the location for the aligned and sorted bam file OPTIONAL.'genome_sizes' REQUIRED.'bedgraph' file name of normalized bedgraph REQUIRED.'SEACR_key' sample key corresponding to sample groups to be run against an IgG (or other) contol.  all samples to be run against a control are given the same name and the control is labeleled with the an additional string underscore + 'CONTROL' (i.e. 4JS_CONTROL) OPTIONAL.'SEACR_out' file name of SEACR output OPTIONAL.Genomes and adding genome locations (required for bulk and SC)Henipipe uses Bowtie2 for alignment.  As such, you should have a previously indexed location of your genome accessible to henipipe.  This location is referred to in henipipe as the 'fasta'.  Similarly, one should provide the location of the spike_in indexed reference genome in the 'spikein_fasta' field.  For bedgraph conversion, a text file of genome sizes text file is also needed.  See the following for a discussion on how to make a 'genome_sizes' filehttps://www.biostars.org/p/173963/.Henipipe provides an easy way to add these locations to your system for repeated use using the --genome_key (-gk) option during MAKERUNSHEET commands.  A file called genomes.json can be found in the 'data' directory of the henipipe install folder.  This file can be edited to include those locations you want to regularly put in the runsheet.  The following shows an example of a genomes.json file.  The files "top level" is a name that can be used in the --genome_key field (-gk) during runsheet generation to populate the columns of the runsheet with fasta, spikein_fasta, and genome_sizes locations associated with that genome_key.  The 'default' key will be used when no genome_key is specified.{"default":{"fasta":"/path/path/hg38/bowtie2_index","genome_sizes":"/path/path/hg38/genome_sizes.txt","spikein_fasta":"/path/path/Ecoli/bowtie2_index"},"my_hg38":{"fasta":"/shared/biodata/ngs/Reference/iGenomes/Homo_sapiens/UCSC/hg38/Sequence/Bowtie2Index/genome","genome_sizes":"/shared/ngs/illumina/henikoff/solexa/databases/human/hg38/chr_lens.txt","spikein_fasta":"/shared/ngs/illumina/henikoff/Bowtie2/Ecoli"}Doing a bulk henipipe runSay your fastqs live within within subfolders of a folder 'fastq' in the folder 'data'.  So if you were to...cd/data/fastq
ls... you'd get a bunch of folders, each of which would be filled with fastqs.  Each folder name should correspond to a sample name.To run henipipe, do the following...Make a new output directory 'henipipe'.Go into that directory and make a runsheet pointing to the fastq folder i.e. the folder level above.  (at the command line, henipipe is cool with either relative or absolute pathnames; but as stated earlier, absolute pathnames are required for the runsheet.)Optionally you can only select directories of fastq files that contain in their name the string denoted using the -sf flag.After inspecting and completing the runsheet, run ALIGN, NORM, SEACR, and AUC.cd..
mkdirhenipipecdhenipipe
henipipeMAKERUNSHEET-fq../fastq-sfMySampleDirectoriesStartWithThisString-o.
henipipeALIGN-rrunsheet.csv
henipipeSCALE-rrunsheet.csv
henipipeBIGWIG-rrunsheet.csv
henipipeSEACR-rrunsheet.csv
mkdirauc
henipipeAUC-rrunsheet.csv-oaucTry our test bulk dataClone this repo.  There you will find a folder called 'test_data'.  Our test fastq files are in a folder creatively named 'fastq'.Note that it is probably easier and faster to simply clone the henipipe repo than to find the data embedded the pipx install.  As such, I would encourage any user who would like to use these test data for troubleshooting purposes to install normally using pipx (as above), then separately clont the repo by invoking 'git clone' as below.  This obviously duplicates the repo, but I have made the test files pretty small (< 70 MB total).  I would then recommend that after all testing is complete, the user then delete the folder created using 'git clone'.gitclonehttps://github.com/scfurl/henipipe.gitcd***Yourspecificlocation***/henipipe/test_data

mlPython# or otherwise load pythonmlR# or otherwise load Rmkdirhenipipecdhenipipe

henipipeMAKERUNSHEET-fq../fastq#edit the resulting runsheet.csv file in Excel or Numbers - Numbers is better - export as CSV and rename (runsheet_fixed.csv)#make sure it looks okayawk-F',''{print $1, $2}'runsheet_fixed.csv#proceed with henipipe stepshenipipeALIGN-t16-rrunsheet_fixed.csv-nspike_in
henipipeSCALE-rrunsheet_fixed.csv-nspike_in
henipipeSEACR-rrunsheet_fixed.csvDoing a SC henipipe run (New in >= Version 2.0)Say your fastqs exist in a 'fastq' in a folder 'data'.ls/data/fastqFor the current version of henipipe, these fastq filesmustbe named such that each pair of fastqs (R1 and R2) has a unique name that contains the cell barcode in the name of the file.  This typically is the output for demuxed iCell8 Takara runs...  For example:A2_1_TCTTATTACCTGCGGG_S221_R1_001.fastq.gz
A2_1_TCTTATTACCTGCGGG_S221_R2_001.fastq.gzTo run henipipes sc, do the following...From within the 'data' directory (in the above example where the fastq files are located in data/fastq); Make a new output directory, say 'henipipe_sc'.Run henipipe specifying the following: the location of the fastqs (here 'fastq') with the '-fq' flag and the location of the desired output (here 'henipipe_sc') with the '-o' flag.  Don't forget to add a genome key with the '-gk' flag!mkdirhenipipe_sc
henipipeSC-fqfastq-ohenipipe_sc-gkFH_mm10_unmaskedHenipipe SC options and disclaimersHenipipe SC was written to accomodate iCell8 output.  As such, it should be able to parse the fastq folder and find paired fastq files.  That being said, it has not been exhaustively debugged.  Let us know if you have an issue using by submitting an issue through github.  Henipipe SC peak files will be generated using SEACR 1.3 (1.4 coming soon), but only by specifying an FDR value (See SEACR for more information).  The FDR is set using the '--SEACR_fdr' flag with a default of 0.05.  Future versions of henipipe will allow for use of this setting for bulk data but currently henipipe SEACR functionality in bulk still requires a control input file.Using a project codeHenipipe is not engineered to accept project codes.  However for PBS job submission systems, henipipe file will capture a global environmental variable called PROJECT and run jobs with that specific project code.For example (note this code is made up)exportPROJECT=e05c3b6h-5a82-4943-uj87-2659cea32a54
henipipeALIGN-t16-rrunsheet.csv-cPBSAcknowledgementsWritten by Scott Furlan with code inspiration from Andrew Hill's cellwrapper; Henipipe includes a python script samTobed.py which takes code from a fantastic sam reader "simplesam" -https://github.com/mdshw5/simplesam.  samTobed.py uses specific sam-sorting parameters similar to those written in Jorja Henikoff's PERL script.