Package: ilus
IlusEnglish|简体中文Ilusis a lightweight, scalable, handysemi-automatedvariant calling pipeline generator for Whole-genome sequencing (WGS) and Whole exom sequencing (WES) analysis.Introductionilusis a pipeline generator, which used to generate WGS/WES analysis pipeline，butiluscan't excute the jobs, which means users needs to submit the jobs by hands, and the processing don't rely onilusany more，that's why we called it as a semi-automated tools.ilushas 3 main modelues：First、WGSanalysis module. This module based onGATK Best Practice，usebwa-mem + GATK, the most mainstream way to build an analysis process. It integrates 5 complete processes, including alignment, sorting, and multi-lane merging of the same sample, Markduplicates, HaplotypeCaller gvcf, Joint-calling ,and Variant quality  score recalibrator (VQSR). This module also works for WES analysis, Just set the configuration filevariant_calling_intervalparameter to the exon capture interval of WES (detailed below).By default, the pipeline currently performs the follwing:- Map reads to reference by `bwa mem`;
- Mark duplicates by `GATK MarkDuplicates`;
- Base quality score recalibration by `GATK BaseRecalibrator` and `GATK ApplyBQSR`;
- Preprocessing quality control by `samtools stats`;
- Calculatiing the base coverage by `bedtools cvg`;
- Variant calling by `GATK HaplotypeCaller`;
- Variants join-calling by `GATK GenotypeGVCFs`;
- Variant quality recalibration by `GATK VariantRecalibrator` and `GATK ApplyVQSR`;
- Annotation by `ensembl VEP`.Second,genotype-joint-callingmodule. This module is separate fromilus WGS，in order to call genotype  fromgvcfdirectly. Or when you need to complete the WGS/WES data analysis in multiple batches, you can generategvcfin batches, and finally organize a totalgvcffile list, and then use this function to complete the subsequent steps. this can increase the flexibility of the running process.Third,VQSRmodule，also separate fromilus WGS，to help us to quality control on the mutation results.Note that ilus don’t include QC on raw fastq data.By default, the sequencing data you input is clean data.ilusdoes not directly run tasks for the following two considerations:First, avoid making optimizations about task scheduling outside of the core functionality ofilus. Different computing clusters (local and cloud), jobs are scheduled in various ways. If these situations are taken into account,iluswill become bloated and complicated, and it may not be possible to do it. Well, this will cause some people to be unable to useiluseffectively, and even easily lose the focus ofilusin the process of managing task delivery. As a lightweight tool, I designediluswithout the ability to automatically post and run tasks in mind from the start. I hope it can be used as a framework program to generate process scripts that meet your analysis needs strictly based on your input data and configuration file information.If you want to achieve automatic delivery and monitoring of jobs,ilusIt is hoped that it can be implemented in the form of external plugins in the future, but currently these scripts need to be manually delivered.Second, increase flexibility and maneuverability. The processes generated byilushave a characteristic, that is, they are completely independent ofilusand will no longer rely on any functions ofilus. It doesn't matter if it is completely uninstalled and deleted. In addition, each line in the execution script (shell) generated byilus can run independently without affecting each other.The advantage of this is that you can split the script into several sub-scripts according to the characteristics of the computer cluster (if you have few samples or insufficient cluster resources, you can not split them), and then deliver the tasks separately, which can greatly reduce the number of sub-scripts. Speed up the completion of tasks,and, this is currently the only way to deliver ilus tasks in parallel.How many subscripts to split into depends on your own specific situation. For example, if you have a total of 10 samples, thexxx.step1.bwa.shcomparison script in the first step has a total of 10 comparison commands, and each line is abwaof a sample.Since these 10 commands are independent of each other, you can split this step into 10 (or less) subscripts and then manually post these tasks. As for how to split a complete execution script into multiple ones, you can either write your own program or use theyhbatch_slurm_jobs, but note that the program I provide here is based on the slurm system, which may not meet your needs, if you think this the program does not directly meet your needs, you can modify it. Althoughilusdoes not automate the delivery and execution of management tasks, this method can also increase the flexibility and control of process control.It is very important to monitor the task completion status, especially when there are thousands of samples to be analyzed. This process, if checked manually, would be inefficient and error-prone.ilushas taking this into account, an identifiable completion mark will be added to each task when the process is generated. We only need to check whether each task has this mark (for details, please refer toWGSbelow for details. example).Even so, once we have a large number of tasks, it would be too troublesome to manually open the file every time to check whether the corresponding task has been successfully completed. Therefore, I implemented a program inilusthat can be used to check the completion status of the task job. For the specific usage, please refer to the example ofWGSbelow.Installationilusis based on Python and supports both Python2.7+ and Python3.7+. The stable version of the code has been released to PyPI. So to useilus, install it directly throughpipthe Python package management tool:$pipinstallilusIn addition to the main programilus, this command will automatically install other Python packages thatilusdepends on. After the installation is complete, executeiluson the command line. If you can see something similar to the following, then the installation is successful.$ilus
usage:ilus[-h]{WGS,genotype-joint-calling,VQSR}...
ilus:error:toofewargumentsQuick StartBy executingilus --helpyou can see three function modules :WGS,genotype-joint-callingandVQSR.$ilus--help
usage:ilus[-h][-v]{WGS,genotype-joint-calling,VQSR,split-jobs,check-jobs}...

ilus(Version=1.3.2):AWGS/WESanalysispipelinegenerator.

optionalarguments:-h,--helpshowthishelpmessageandexit-v,--versionshowtheversionofilusandexit.

iluscommands:{WGS,genotype-joint-calling,VQSR,split-jobs,check-jobs}WGSCreateapipelineforWGS(fromFASTQtogenotypeVCF).genotype-joint-callingGenotypefromGVCFs.VQSRVQSRsplit-jobsSplitthewholeshellintomultiplejobs.check-jobsCheckthejobshavefinishedornot.

That'showyoucanuseilusBelow, we will introduce how to use these three modules by examples.WGSThe run scripts of the WGS Analysis Pipeline are generated byilus WGSand are used as follows:$ilusWGS--help
usage:ilusWGS[-h][-nPROJECT_NAME]-CSYSCONF-OOUTDIR[-f][--use-sentieon]-LFASTQLIST[-c][-PWGS_PROCESSES][-dr]optionalarguments:-h,--helpshowthishelpmessageandexit-nPROJECT_NAME,--namePROJECT_NAMENameoftheproject.(Default:test)-CSYSCONF,--confSYSCONFYAMLconfigurationfilespecifyingsystemdetails.-OOUTDIR,--outdirOUTDIROutputdirectoryforresults.-f,--force-overwriteForceoverwriteexistingshellscriptsandfolders.--use-sentieonUsesentieon(doc:https://support.sentieon.com/manual)tocreateanalysispipeline.-LFASTQLIST,--fastqlistFASTQLISTListofalignmentFASTQfiles.-c,--cramConvertBAMtoCRAMafterBQSRandsavealignmentfilestorage.-PWGS_PROCESSES,--processWGS_PROCESSESSpecifyoneormoreprocesses(separatedbycomma)ofWGSpipeline.Possiblevalues:align,markdup,BQSR,gvcf,combineGVCFs,genotype,VQSR-dr,--dry-runDryrunthepipelinefortesting.-C,-Land-Oarerequired parameters, and the rest are optional parameters according to actual needs. The-Oparameter is the output directory, if the directory does not exist,iluswill be created automatically. The most important are-Cand-Lparameters, the former is the configuration file ofilus, without this fileiluscannot generate the analysis process correctly, so it is very important; the latter is Input file,The format of this file has fixed requirements, a total of 5 columns, each column is the necessary information for the process.Below, I will explain the format of these two files respectively:The first is the configuration file. We need to write the program path used in the analysis process, theGATK bundlefile path, the path of the reference sequence and the parameters corresponding to each key step in the file.It should be noted that the prefix of the comparison index file ofbwa memshould be the same as the prefix of{resources}{reference}of the configuration file, and placed in the same folder. as follows:/path/human_reference/GRCh38/|--human_GRCh38.fa|--human_GRCh38.dict|--human_GRCh38.fa.amb|--human_GRCh38.fa.ann|--human_GRCh38.fa.bwt|--human_GRCh38.fa.fai|--human_GRCh38.fa.pac`--human_GRCh38.fa.saThe configuration file should be written inYaml syntax, here I provide aconfiguration file template.In the configuration file,bwa,samtools,bcftools,bedtools,gatk,bgzipandtabixare all necessary bioinformatics software, which need to be installed in advance, and then fill in the path to in the corresponding parameters (as shown in the template).verifyBamID2is only used to calculate whether there is pollution in the sample,it is not a required parameter, if your configuration file does not have this parameter, it means The process does not calculate the contamination of the sample. If there is, you have to install and download theresourcedata supporting it. I also tell you where to download the relevant data in the template.Note that thevariant_calling_intervalparameter in the configuration file. This is a parameter specifically used to specify the variation detection interval. For example, in the example of the above configuration file, I gave 25 chromosomes fromchr1tochrM, which means to tell the process to perform mutation detection on these 25 chromosomes . If you list only one chromosome in this parameter, or only give a chromosome interval, such aschr1:1-10000, theniluswill also only perform variant detection in the interval you give.This is a very flexible and useful parameter. Thevariant_calling_intervalinterval can be specified arbitrarily. In addition to the assignment method given in my example, you can also assign the intervalfile pathto this parameter.We know that many steps of WGS and WES are exactly the same, and there are only differences in the interval of variant detection ------ WES data is not necessary and cannot beused for variant detection on the whole chromosome, only in the exon capture area.You only need to put the file of the exon capture area,which in the.bedfile format, the example is as follows:chr16369763697chr1101158101158chr1103241103241chr1104108104108chr1185336185336chr1261495261495chr1598862598862chr1601606601606chr1700596700596chr1725086725086You can also refer toGATK's instructions, here you do not need to manually split them into one by one. It is enough to assign the path of the file to this parameter, and then the process becomes the WES analysis process. This is whyilusis called a WGS and WES analysis pipeline generator.Also,ilusrequired public datasets are:gatk bundleand genome reference sequences.[Note] If the sample size of your project is less than 10, then GATK will not calculate the value ofInbreedingCoeff. In this case,vqsr_optionsin the configuration file does not need to set-an InbreedingCoeff, you can remove it.Next is the input file specified by the-Lparameter. The file contains all the sequencing data information necessary for theWGS/WESanalysis process.This file needs to be prepared by you, the format of each column of the file as follows:[1] SAMPLE，Sample name[2] RGID，Read Group，when usingbwa mem's -R parameter[3] FASTQ1，Fastq1's file directory[4] FASTQ2，Fastq2's file directory，if it'sSingle Endsequencing，the column is replaced with a space[5] LANE，fastq‘slaneidin these file info,RGIDis the most error-prone, andRGIDmust be set correctly (refer to the following example for the correct way of writing), otherwise the analysis process will go wrong.In addition, if the sequencing volume of a sample is relatively large, resulting in a sample with multiplelanesequencing data, or the samelanedata is split into multiple sub-files, at this time, you do not need to manually analyze theselanedata. To merge fastq data, you only need to write the input file according to the sequencing information.Those data belonging to the same sample, even if the inputfastqhas been split into thousands of copies, the process will automatically merge after each sub-data is compared and sorted.
Below I give an example of an input file, in which there is a case of the data splitting output of the sample:#SAMPLE RGID    FASTQ1  FASTQ2  LANEHG002"@RG\tID:CL100076190_L01\tPL:COMPLETE\tPU:CL100076190_L01_HG002\tLB:CL100076190_L01\tSM:HG002"/path/HG002_NA24385_son/BGISEQ500/BGISEQ500_PCRfree_NA24385_CL100076190_L01_read_1.clean.fq.gz/path/HG002_NA24385_son/BGISEQ500/BGISEQ500_PCRfree_NA24385_CL100076190_L01_read_2.clean.fq.gzCL100076190_L01
HG002"@RG\tID:CL100076190_L02\tPL:COMPLETE\tPU:CL100076190_L02_HG002\tLB:CL100076190_L02\tSM:HG002"/path/HG002_NA24385_son/BGISEQ500/BGISEQ500_PCRfree_NA24385_CL100076190_L02_read_1.clean.fq.gz/path/HG002_NA24385_son/BGISEQ500/BGISEQ500_PCRfree_NA24385_CL100076190_L02_read_2.clean.fq.gzCL100076190_L02
HG003"@RG\tID:CL100076246_L01\tPL:COMPLETE\tPU:CL100076246_L01_HG003\tLB:CL100076246_L01\tSM:HG003"/path/HG003_NA24149_father/BGISEQ500/BGISEQ500_PCRfree_NA24149_CL100076246_L01_read_1.clean.fq.gz/path/HG003_NA24149_father/BGISEQ500/BGISEQ500_PCRfree_NA24149_CL100076246_L01_read_2.clean.fq.gzCL100076246_L01
HG003"@RG\tID:CL100076246_L02\tPL:COMPLETE\tPU:CL100076246_L02_HG003\tLB:CL100076246_L02\tSM:HG003"/path/HG003_NA24149_father/BGISEQ500/BGISEQ500_PCRfree_NA24149_CL100076246_L02_read_1.clean.fq.gz/path/HG003_NA24149_father/BGISEQ500/BGISEQ500_PCRfree_NA24149_CL100076246_L02_read_2.clean.fq.gzCL100076246_L02
HG004"@RG\tID:CL100076266_L01\tPL:COMPLETE\tPU:CL100076266_L01_HG004\tLB:CL100076266_L01\tSM:HG004"/path/HG004_NA24143_mother/BGISEQ500/BGISEQ500_PCRfree_NA24143_CL100076266_L01_read_1.clean.fq.gz/path/HG004_NA24143_mother/BGISEQ500/BGISEQ500_PCRfree_NA24143_CL100076266_L01_read_2.clean.fq.gzCL100076266_L01
HG004"@RG\tID:CL100076266_L02\tPL:COMPLETE\tPU:CL100076266_L02_HG004\tLB:CL100076266_L02\tSM:HG004"/path/HG004_NA24143_mother/BGISEQ500/BGISEQ500_PCRfree_NA24143_CL100076266_L02_read_1.clean.fq.gz/path/HG004_NA24143_mother/BGISEQ500/BGISEQ500_PCRfree_NA24143_CL100076266_L02_read_2.clean.fq.gzCL100076266_L02
HG005"@RG\tID:CL100076244_L01\tPL:COMPLETE\tPU:CL100076244_L01_HG005\tLB:CL100076244_L01\tSM:HG005"/path/HG005_NA24631_son/BGISEQ500/BGISEQ500_PCRfree_NA24631_CL100076244_L01_read_1.clean.fq.gz/path/HG005_NA24631_son/BGISEQ500/BGISEQ500_PCRfree_NA24631_CL100076244_L01_read_2.clean.fq.gzCL100076244_L01The following example illustrates the use and process structure ofilus WGS.Example 1: Generating a WGS analysis pipeline from scratch$ilusWGS-c-nmy_wgs-Cilus_sys.yaml-Linput.list-Ooutput/This command means that the project (-n)my_wgsgenerates a WGS analysis process in the output directoryoutputbased on the configuration file (-C)ilus_sys.yamland the input data (-L)input.list. At the same time, the process automatically convertsBAMto (-c)CRAMformat after completing the analysis.CRAMis more space efficient thanBAM, if the-cparameter is not set, the originalBAMfile is kept.After the above command is successfully executed, there are a total of 4 folders in the output directoryoutput(as follows):00.shell/01.alignment/02.gvcf/03.genotype/They are used to store different types of data generated by the process, including:00.shelltheshellscript collection directory;01.alignmentstores the alignment results in units of samples;02.gvcfstores thegvcfresults of each sample;03.genotypeholds the result of the last variant detection.There are various execution scripts and log directories of the analysis process in the00.shelldirectory:/00.shell
├──loginfo
│├──01.alignment
│├──01.alignment.e.log.list
│├──01.alignment.o.log.list
│├──02.markdup
│├──02.markdup.e.log.list
│├──02.markdup.o.log.list
│├──03.BQSR
│├──03.BQSR.e.log.list
│├──03.BQSR.o.log.list
│├──04.gvcf
│├──04.gvcf.e.log.list
│├──04.gvcf.o.log.list
│├──05.genotype
│├──05.genotype.e.log.list
│├──05.genotype.o.log.list
│├──06.VQSR
│├──06.VQSR.e.log.list
│└──06.VQSR.o.log.list
├──my_wgs.step1.bwa.sh
├──my_wgs.step2.markdup.sh
├──my_wgs.step3.bqsr.sh
├──my_wgs.step4.gvcf.sh
├──my_wgs.step5.genotype.sh
└──my_wgs.step6.VQSR.shWhen submit the task running process, execute it in sequence fromstep1tostep6. Theloginfo/folder records the running status of all steps of each sample. You can check the.o.log.listlog file of each task to get the mark of whether each sample ended successfully.If successful, you can see a marker like[xxxx] xxxx doneat the end of the log file. You can easily know which samples (steps) have been well done, and which ones haven't. This script will help you to output all those unfinished tasks, which is convenient to check for problems and re-execute this part of unfinished tasks.check_jobs_statusThe usage is as follows:$pythoncheck_jobs_status.pyloginfo/01.alignment.o.log.list>bwa.unfinish.listif all tasks have done, this list is empty，and print** All Jobs done **.How to submit batch jobsThe process scripts generated byilushave a feature, that is, they are completely independent ofilusand will no longer depend on any functions ofilus. At this time, you even useilusIt doesn't matter if you uninstall or delete it.And each line in the execution script (shell) can be run independently.The advantage of this is that you can split the script into several sub-scripts according to the characteristics of the cluster you are using (if you have few samples or insufficient cluster resources, you can not split them), and then deliver the tasks independently, so it will be more efficient,this is also the only way to submit ilus tasks in parallel.As for how many subscripts to be split into, it depends on your own needs. For example, if you have a total of 10 samples, there are a total of 10 comparison commands in thexxx.step1.bwa.shmapping script in the first step. A line is abwatask of a sample. Since these 10 commands are independent of each other, you can split the step into 10 (or less) subscripts, and then manually post these 10 tasks. As for how to split a complete execution script into multiple ones, you can either write your own program or use theyhbatch_slurm_jobs programprovided byilus, but please note that the program I provide here is based on the slurm system, which may not meet your needs,  and you can modify it, if you think This program does not meet your needs.Example 2: Generate only one/some steps in theWGSprocessSometimes, we do not intend (or do not have to) to complete theWGSprocess from start to the end. For example, we only want to perform the steps fromfastqcomparison togvcfgeneration, and do not want to performgenotypeandVQSR.The-Pparameter of ilus can does this:$ilusWGS-c-nmy_wgs-Cilus_sys.yaml-Linput.list-Palign,markdup,BQSR,gvcf-O./outputThis only generates execution scripts frombwatogvcf, which is useful for projects that need to do analysis in batches. Moreover, the output results ofilusare distinguished by sample units, so in the same output directory, as long as the sample numbers are different, the data of different batches will not overlap each other.In addition, the-Pparameter has another purpose, that is, if aWGSstep is wrong and needs to be adjusted, and then re-update the corresponding step, then you can use-Pto rerun specific steps. For example, if I need to regenerate the running script of theBQSRstep, I can do this:$ilusWGS-c-nmy_wgs-Cilus_sys.yaml-Linput.list-PBQSR-O./outputHowever, it should be noted thatiluswill only keep the totalBAM/CRAMfile afterBQSRfor each sample in order to save the project's storage space consumption. Therefore, if you want to re-runBQSRyou need to make sure that theBAMfile from the previous step ofBQSR(ie,markdup) has not been deleted.If you have been usingilusthen you don't need to worry about this problem, becauseilushas "atomic properties" when executing tasks, that is to say, only when all processes in the step are successfully completed will the Delete files that are completely unnecessary. Therefore, if theBQSRstep does not finish normally, theBAMfile of the previousmarkdupwill be preserved.-P
The analysis module specified by the parameter must belong to one or more of「align,markdup,BQSR,gvcf,genotype,VQSR」and be separated by commas.genotype-joint-callingIf we already havegvcfdata for each sample, and now use thesegvcfto do multi-sample joint variants callingJoint-calling, then you can usegenotype-joint-callingto implement. The specific usage is as follows:$ilusgenotype-joint-calling--help
usage:ilusgenotype-joint-calling[-h][-nPROJECT_NAME]-CSYSCONF-OOUTDIR[-f][--use-sentieon]-LGVCFLIST[--as_pipe_shell_order]optionalarguments:-h,--helpshowthishelpmessageandexit-nPROJECT_NAME,--namePROJECT_NAMENameoftheproject.(Default:test)-CSYSCONF,--confSYSCONFYAMLconfigurationfilespecifyingsystemdetails.-OOUTDIR,--outdirOUTDIROutputdirectoryforresults.-f,--force-overwriteForceoverwriteexistingshellscriptsandfolders.--use-sentieonUsesentieon(doc:https://support.sentieon.com/manual)tocreateanalysispipeline.-LGVCFLIST,--gvcflistGVCFLISTListofGVCFfiles.Onegvcf_fileper-rowandtheformatshouldlookslike:[intervalgvcf_file_path].Column[1]isasymbolwhichcouldrepresentthegenomeregionofthegvcf_fileandcolumn[2]shouldbethepath.--as_pipe_shell_orderKeeptheshellnameastheorderof`WGS`.-Lis the input parameter ofilus genotype-joint-calling, which input agvcf listfile, which consists of two columns, the first column is the correspondinggvcffile The interval or chromosome number, the second column is the path of thegvcffile. Currentlyilusrequires thegvcfof each sample to be separated by major chromosomes (1-22, X, Y, M), for example:$ilusgenotype-joint-calling-nmy_project-Cilus_sys.yaml-Lgvcf.list-Ogenotype--as_pipe_shell_orderThe format ofgvcf.list:chr1/path/sample1.chr1.g.vcf.gz
chr1/paht/sample2.chr1.g.vcf.gz
chr2/path/sample1.chr2.g.vcf.gz
chr2/path/sample2.chr2.g.vcf.gz
...
chrM/path/sample1.chrM.g.vcf.gz
chrM/path/sample2.chrM.g.vcf.gzIn this examplegvcf.listhas only two samples. The parameter--as_pipe_shell_orderis optional(the default is not added), its only function is to output the name of the executed script according to theilus WGSprocess, maintaining the same order and the same as theWGSprocess The output directory structure of .VQSRThis function is only used to generate execution scripts based onGATK VQSR. If we already have the final mutation detection (VCF format) results, and now we just want to useGATK VQSRto do quality control of this mutation data, then you can use this module, the usage is similar togenotype-joint-calling, as follows:$ilusVQSR--help
usage:ilusVQSR[-h][-nPROJECT_NAME]-CSYSCONF-OOUTDIR[-f][--use-sentieon]-LVCFLIST[--as_pipe_shell_order]optionalarguments:-h,--helpshowthishelpmessageandexit-nPROJECT_NAME,--namePROJECT_NAMENameoftheproject.(Default:test)-CSYSCONF,--confSYSCONFYAMLconfigurationfilespecifyingsystemdetails.-OOUTDIR,--outdirOUTDIROutputdirectoryforresults.-f,--force-overwriteForceoverwriteexistingshellscriptsandfolders.--use-sentieonUsesentieon(doc:https://support.sentieon.com/manual)tocreateanalysispipeline.-LVCFLIST,--vcflistVCFLISTVCFsfilelist.Onefileper-row.--as_pipe_shell_orderKeeptheshellnameastheorderof`WGS`.Different fromgenotype-joint-calling，ilus VQSRinputVCFlist，andeach line is a VCF path，for example：/path/chr1.vcf.gz
/path/chr2.vcf.gz
...
/path/chrM.vcf.gzOther parameters is the same asgenotype-joint-calling. Also, the vcf in the file list does not need to be manually merged in advance,ilus VQSRwill do that. A complete example is provided below:$ilusVQSR-Cilus_sys.yaml-Lvcf.list-Ogenotype--as_pipe_shell_order-- The End --