Package: espresso-caller
Automated and reproducible tool for identifying genomic variants at scaleProcessing high-throughput sequencing data was the biggest challenge in the past, dealing with small datasets of large sample data.
Genomics projects with thousands of samples have became popular due to decreasing in sequencing costs.
Now we are facing a new challenge: how to process and store large-scale datasets efficiently.TheSequence Alignment/Map Format Specification (SAM)and its binary version (BAM) is the most popular and recommended file format to store sequence alignment data. However, due to the increasing throughput of sequencing platforms, BAM files have lost their efficiency, as we are dealing more frequently with deeper whole-genome sequencing (WGS), whole-exome sequencing (WES) and other large-scale genomics data.TheCRAM file formatis an  improved version of SAM format. Instead of storing DNA sequences aligned to a reference genome, it stores only thedifferencesbetween the sample and the genome sequence. With this feature, itreduces dramatically the file sizes. To use this new format, common tools, likeSamtools, require the original genome file (normally in FASTA format).On another front, different computational tools have been developed to address different problems and they are often chained together in a pipeline to solve a bigger problem. For example, a basic variant caller workflow will:map the reads from a FASTQ file to a reference genome;use the allelic counts at each site to call a genotype.It is often the case that each of the aforementioned steps will be performed by a different tool (like BWA + GATK). To assist on this matter (combining different tools, linking the output of one step to the input of the following), languages that are agnostic to the choice of platform were developed. These languages allow us todescribetasksand combine them intoworkflows, using a dialect closer to the research domain and distant from the computer science domain.Common Workflow Language (CWL),NextflowandWorkflow Description Language (WDL)are the most popular languages to define workflows for processing genomics data. It is important to highlight that these languages only describe what needs to be done.As an actual user, one expects that the aforementioned tasks/workflows areexecutedto produce results of interest. For this reason, researchers developedworkflow execution systems, which combine workflow files (.cwl,.nfor.wdl) with parameter files (.jsonof.yaml) and input data to generate adependency treeand actuallyexecute the processing tasks.
The most popular workflow executors are:ToilandCromwell.
These tools are flexible enough to support differentcomputing backendssuch as local, cluster, custom environments and cloud.Distributing bioinformatics tools and keeping track of their versions between several workflows and computing nodes have become a big problem for both system administrators (system updates breaks user tools) and users (updated version o tool outputsdifferent results, compromising the reproducibility of the research -- for more information seeFAIR Guiding Principles).
To solve this problem the research community adopted container technologies, mainlySingularityandDocker. Projects likeBioContainersprovide thousands of container images of bioinformatics tools.
Also,Docker Hubis the main repository for popular open-source projects.Back to workflows... the workflow file + inputs file have been working well for small datasets. With ourwftoolssoftware, we can submit workflows (and their inputs) to execution services, such as Cromwell in server mode, check workflow executions status and logs, and collect output files copying them to desirable directory. The input JSON file must contain workflow-specific parameter to work, for example, a workflow that require FASTQ files as input we have to provide alist of absolute paths to FASTQ files(two list if paired-end), other workflows may require several resource files that can be specific for different genome versions. It is important to note that Cromwell checks the file existence immediately prior to its use within the task, and this is associated with several downstream errors that force the system to abort the execution of the workflow. Additionally, the input JSON files are manually produced.Espresso-Caller is a tool that automates execution of workflows for identification of genomic variants from whole-exome and whole-genome datasets.
It collects raw sequencing paired-end FASTQ or raw gVCF files, together with resources files (b37orhg38versions), assessing their existence, to generate the JSON file that is used as input of workflows for processing WES/WGS data.
Our software takes some conventions to automatically find the required files and to define sample metadata.
Internally it submits WDL files to Cromwell server throughCromwell API.
The next section introduces the workflows, next section provides examples of command lines and explanation of arguments, conventions and outputs.
The last section shows advanced uses ofespressoThis document ends with the actual usage help.Workflowsespressoprovides three workflows:hc(HaplotypeCalling) takes raw FASTQ files and their metadata as input plus resources files and generates per-sample raw gVCF alongside unmapped BAM and analysis-ready CRAM files.joint(JointDiscovery) takes raw gVCF files and merge into a single unified VCF;allexecutes all data processing steps: from raw FASTQs to unified VCF.Our tool bundlesin-house workflowsand workflows defined by the Broad Institute asGATK Best Practicesdefined in WDL format.
The workflow files are stored inside the tool package.
The list below presents each workflow in the order that is executed by this tool.HaplotypeCallingruns the following workflows:Convert raw paired-end FASTQ files to unmapped BAM format (uBAM)Map sequences to reference genome, recalibrate alignments and generate analysis-ready BAM filesProduce sample-specific sufficient statistics to call variantsValidate BAM filesConvert BAM files to CRAM formatJointDiscovery, we combine the sample-specific sufficient statistics to produce final genotype calls. For this step, we use the original WDL file produced by the Broad Institute.Espresso-Caller follows someconvention over configuration(where configuration is the inputs JSON file).
Your data files must files the following conventions to work:Forallandhc:Partial FASTQ files were previously merged into a single oneEach sample is paired-end sequencing data divided in two FASTQ files by strand: forward and reverseFASTQ files are located at the same directory, one directory for each library name/batchFASTQ file names matches this pattern:(sample_name)_R?[12].fastq(.gz)?FASTQ sequence headers match this pattern:@_:_:(sample_id):(flowcell):_:_:_:_:_:(primer)which is merged assample_id.flowcell.primerResource files, including reference genome files, are in the same directory, one directory for each versionResource files have the same name from download URLDestination directory does not exist or is empty, any conflicting filewill be overwrittensilentlyRelative paths are allowed in command argumentsResource files can be downloaded using the following scripts.
Downloaded files will have the right names to work withespresso.b37hg38To download resource files run.
It is important to inform theabsolute pathto destination directory.bashdownload_resources_b37_files.sh/home/data/ref/b37
bashdownload_resources_hg38_files.sh/home/data/ref/hg38The following command will runespressowith two directories containing raw FASTQ files.
Samples files can be separated by directories to use different metadata for each group of samples (or batch).
If two--fastq <fastq directory>argument is defined you have to inform the following arguments twice.--library <library name>Library (LBSAM tag)--dateSequencing run date(DTSAM tag) -Must be ISO8601 format (YYYY-MM-DD)!--platformSequencing platform (PLSAM tag)--centerSequencing center (CNSAM tag)Also, sample name (SM), platform unit (PU) are extracted from FASTQ automatically by using predefined conventions.Next we have to inform the path to resources files (--reference <resources directory>) and the reference genome version (onlyb37andhg38are supported),--version <genome version>.Some computing environments do not support container technology.
In this case we have to inform theabsolute pathsto this software.
Seeinstalling required software script.--gatk_path_overridepath toGATK version 4, it must point togatkwrapper script (not the Jar file).--gotc_path_overridepath to directory containing all softwares (bwa,picard.jar, etc)--samtools_path_overridepath tosamtools.The--dont_runflag doesespressotonot submit workflows to Cromwell server.
In this mode, the tool will check all required files, copy required workflow files and generate inputs JSON file writing both to destination directory.
It is very recommend that you run our tool in this mode and check JSON filebeforerunning in production.
Also, it is useful when there are some change to do in the default JSON file and then submit workflow usingespressoinstead.The optional--moveflag will tellespressotomoveoutput files from Cromwell execution directory to destination directory.
It is useful for processing large-scale genomics datasets avoiding file duplication.The last two arguments are required: callset name and destination directory to write all files.espressoall\--fastq~/raw/batch1\--fastq~/raw/batch2\--libraryBatch1\--libraryBatch2\--date2018-10-19\--date2019-02-07\--platformILLUMINA\--platformILLUMINA\--centerMyCenter\--centerMyCenter\--reference~/ref/b37\--versionb37\--gatk_path_override~/bin/gatk-4.1.0.0/gatk\--gotc_path_override~/bin\--samtools_path_override~/bin/samtools-1.9/samtools\--move\--dont_run\MyDataset.b37\~/res/my_datasetStarting the haplotype-calling workflow with reference genome version b37
Workflow file: /home/data/res/my_dataset/haplotype-calling.wdl
Inputs JSON file: /home/data/res/my_dataset/haplotype-calling.b37.inputs.json
Workflow will not be submitted to Cromwell. See workflow files in /home/data/res/my_datasetIf you run without--dont_runargument you will see the text below.
As you can see, after workflow is submitted no output is presented until execution is finished.
Then the tool will print the collected output files and exit.In this versionallrunhcfirst thenjoint. TODO: write WDL file that do bothStarting haplotype-calling workflow with reference genome version b37
Workflow file: /home/data/res/my_dataset/haplotype-calling.wdl
Inputs JSON file: /home/data/res/my_dataset/haplotype-calling.b37.inputs.json
Workflow submitted to Cromwell Server (http://localhost:8000)
Workflow id: 9977f400-d1b6-41ff-ab92-7ebbbf7a30a8Expected outputsAt the end of executionespressowillcollect output filescopying them to destination directory.
These are the file you expect to see.{callset}.vcf.gzis thefinal unified VCF filewherecallsetwas previously defined (index{callset}.vcf.gz.tbi)haplotype-calling.wdlishcworkflow in WDLjoint-discovery-gatk4-local.wdlisjointworkflow in WDLhaplotype-calling.{version}.inputs.jsonis inputs forhcwhereversionisb37orhg38joint-discovery.{version}.inputs.jsonis inputs forjointout.intervalsis an intervals files used byjointto call variants on all samples{callset}.variant_calling_detail_metricsis the variant call metrics files{callset}.variant_calling_summary_metricssummary variant calling metrics fileFor each sample it should have.{sample}.{version}.g.vcf.gzis raw gVCF file wheresamplewas extracted from FASTQ file name (index{sample}.{version}.g.vcf.gz.tbi){sample}.{version}.cramis the analysis-ready CRAM file (contains both aligned sequences and sequences that weren't mapped at the end of file){sample}.{version}.duplicate_metricsis the duplication metrics file{sample}.{version}.validation_.txtis the BAM validation output file{sample}.unmapped.bamis the unmapped sequences in BAM format (exactly the same sequences from FASTQ but with recalibrated qualities){sample}.{version}.recal_data.csvis the file used to recalibrate PHRED qualitiesRunninghcorjointindividuallyIt is possible to run onlyhcbyespresso hc ...andjointbyespresso joint ....
Runninghcis useful when we don't want to generate the unified VCF for theses FASTQ files.
Thejointis useful when we already have raw gVCF files and we want to merge to or more data from batches or studies.
However there are some conventions that your VCF files must follow (only if they weren't generated byespresso).VCF file names must match this pattern:(sample_name)(.version)?.g.vcf(.gz)?(it must have.g.vcfextension to skip unified VCFs that may exist in the same directory).Index files with the same name plus.tbiextension in the same directory.Reproduce data processingWith WDL and JSON files written in theresult directoryit is possible to re-execute data processingwithoutespresso.
To do that we need the Cromwell binary file.
It should also works on different workflow engine such asminiwdlor workflow submission toolm such aswf.Run workflow Cromwell in Local modejava-jarcromwell.jarrun\-i/home/data/res/my_dataset/haplotype-calling.b37.inputs.json\/home/data/res/my_dataset/haplotype-calling.wdl