Package: joint-calling
Joint calling pipelineAHailbased pipeline for post-processing and filtering of large scale genomic variant calling datasets.Combines GVCFs (generated by GATK4) to a Hail Matrix Table.Performs sample-level QC.Performs variant QC using a allele-specific VQSR model.UsageThe workflow should be run using theCPG analysis runner.Install the CPG analysis runner:mambainstall-ccpg-cconda-forgeanalysis-runnerAssuming the name of the dataset isfewgenomes,exportDATASET=fewgenomesTest the workflow using the analysis runner on a given dataset:# Assuming we already changed to the dataset repository rootDATASET=$(basename$(pwd))analysis-runner\--dataset${DATASET}\--output-dir"joint-calling/test"\--description"Joint calling, test"\--access-leveltest\pythonbatch_workflow.py\--namespacetest\--analysis-projectfewgenomes\--input-projectfewgenomes\--release-related\--output-versionv1-0\--assume-gvcfs-are-ready\--scatter-count50This command will use thetestaccess level, which means finding the input GVCFs in thetestnamespace (e.g.gs://cpg-$DATASET-test/gvcf/*.g.vcf.gz), writing the resulting matrix tables to thetest-tmpbucket,gs://cpg-fewgenomes-test-tmp/mt/v0.mt, and writing plots togs://cpg-fewgenomes-test-tmp/joint-calling/v0.--scatter-countcontrols the number of secondary workers for Dataproc clusters, as well as the number of shards to parition data for the AS-VQSR analysis.To use themainbucket for input and output, run the workflow with thefullaccess level:DATASET=$(basename$(pwd))analysis-runner\--dataset${DATASET}\--output-dir"gs://cpg-${DATASET}-hail/joint-calling"\--description"Joint calling, full"\--access-levelfull\joint-calling/driver_for_analysis_runner.shworkflows/batch_workflow.py\--scatter-count200\--frommain\--tomain\--callset${DATASET}\--version$(VERSION)\--batchbatch1\--batchbatch2\--batchbatch3It will find input GVCFs in themainbucket, assuming the batch IDs arebatch1,batch2,batch3:gs://cpg-$DATASET-main/gvcf/{batch1,batch2,batch3}/*.g.vcf.gz; write the resulting matrix table to themainbucket:gs://cpg-fewgenomes-main/mt/v0.mt, and plots togs://cpg-fewgenomes-analysis/joint-calling/v0.Overview of the pipeline stepsFind inputs. According to the specified--datasetand--batchparameters, look atgs://cpg-<dataset>-main/gvcf/<batch-id>/*.g.vcf.gz(orgs://cpg-<dataset>-test/gvcf/<batch-id>/*.g.vcf.gz) for the GVCFs and a CSV file with QC metadata.Post-process the GVCFs:Run GATK ReblockGVCFs to annotate with allele-specific VCF INFO fields required for recalibration (QUALapprox, VarDP, RAW_MQandDP),Subset GVCF to non-alt chromosomes.Run the GVCF combiner usingscripts/combine_gvcfs.py. The script merges GVCFs into a sparse Matrix Table usingHail's vcf_combiner.Run thescripts/sample_qc.pyscript, that performs thesample-level QC, and generates a Table with the filtered sample IDs, as well as a metadata Table with metrics that were used for filtering (coverage, sex, ancestry, contamination, variant numbers/distributions, etc.).Run theallele-specific VQSR approachto perform the variant filtration.Sample QCThe sample QC and random forest variant QC pipelines are largely a re-implementation and orchestration ofthe Hail methods used for the quality control of GnomAD release. Good summaries of gnomAD QC pipeline can be found in gnomAD update blog posts:https://macarthurlab.org/2017/02/27/the-genome-aggregation-database-gnomadhttps://macarthurlab.org/2018/10/17/gnomad-v2-1https://macarthurlab.org/2019/10/16/gnomad-v3-0https://gnomad.broadinstitute.org/blog/2020-10-gnomad-v3-1-new-content-methods-annotations-and-data-availability/#sample-and-variant-quality-controlhttps://blog.hail.is/whole-exome-and-whole-genome-sequencing-recommendations/Here we give a brief overview of the sample QC steps:Compute sample QC metrics using Hail’ssample_qcmodule on all autosomal bi-allelic SNVs.Filter outlier samples using the following cutoffs. Note that the most up to date cutoffs are speified in the configuration filefilter_cutoffs.yaml, which can be overridden with--filter-cutoffs-file.Filter using BAM-level metrics was performed when such metrics were available. We removed samples that were outliers for:Contamination: freemix > 5% (call-UnmappedBamToAlignedBam/UnmappedBamToAlignedBam/*/call-CheckContamination/*.selfSM/FREEMIX)Chimeras: > 5% (call-AggregatedBamQC/AggregatedBamQC/*/call-CollectAggregationMetrics/*.alignment_summary_metrics/PCT_CHIMERAS)Duplication: > 30% (call-UnmappedBamToAlignedBam/UnmappedBamToAlignedBam/*/call-MarkDuplicates/*.duplicate_metrics/PERCENT_DUPLICATION)Median insert size: < 250 (call-AggregatedBamQC/AggregatedBamQC/*/call-CollectAggregationMetrics/*.insert_size_metrics/MEDIAN_INSERT_SIZE)Median coverage < 18X (calculated from the GVCFs).Sex inferred for each sample with Hail'simpute_sex. Filter samples with sex chromosome aneuploidies or ambiguous sex assignment.Note that all filtering above makes it exclude samples from the variant QC modelling, as well as from the AC/AF/AN frequency calculation. However, it keeps the samples in the final matrix table, with labels inmt.meta.hardfilter.Relatedness inferred between samples using Hail'spc_relate. Identified pairs of 1st and 2nd degree relatives. Filter to a set of unrelated individuals using Hail'smaximal_independent_setthat tries to keep as many samples as possible. When multiple samples could be selected, we kept the sample with the highest coverage.PCA was a ran on high-quality variants, and RF was trained using 16 principal components as features on samples with known ancestry. Ancestry was assigned to all samples for which the probability of that ancestry was >75%.Hailsample_qcwas used stratified by 8 ancestry assignment PCs. Within each PC, outliers were filtered if they are 4 median absolute deviations (MADs) away from the median for the following metrics:n_snp,r_ti_tv,r_insertion_deletion,n_insertion,n_deletion,r_het_hom_var,n_het,n_hom_var,n_transition,n_transversion, or 8 MADs away from the median number of singletons (n_singletonmetric).Allele-specific variant quality score recalibration (AS-VQSR)Export variants into a sites-only VCF and split it into SNPs and indels, as well as region-wise for parallel processing.Run Gnarly Genotyper to perform "quick and dirty" joint genotyping.Create SNP and indel recalibration models using the allele-specific version of GATK Variant Quality Score RecalibrationVQSR, using the standard GATK training resources (HapMap, Omni, 1000 Genomes, Mills indels), with the following features:SNVs:AS_FS,AS_SOR,AS_ReadPosRankSum,AS_MQRankSum,AS_QD,AS_MQIndels:AS_FS,AS_SOR,AS_ReadPosRankSum,AS_MQRankSum,AS_QDNo sample had a high quality genotype at this variant site (GQ>=20, DP>=10, and AB>=0.2 for heterozygotes) (all fields are populated by GATK)InbreedingCoeff< -0.3 (there was an excess of heterozygotes at the site compared to Hardy-Weinberg expectations) (InbreedingCoeffis populated by GATK)Apply the models to the VCFs and combine them back into one VCF.Import the VCF back to a matrix table.VQSR pipeline is a compilation from the following 2 WDL workflows:hail-ukbb-200k-callset/GenotypeAndFilter.AS.wdlTheBroad VQSR workflowdocumentedhere, translated from WDL with a help ofJanis.Random forest variant QCGather information for the random forest modelImpute missing entriesSelect variants for training examplesTrain random forests modelTest resulting model on chr20Save training data with metadata describing random forest parameters usedApply random forest model to the full variant set.