Package: vembrane
vembrane: variant filtering using python expressionsvembrane allows to simultaneously filter variants based on anyINFOorFORMATfield,CHROM,POS,ID,REF,ALT,QUAL,FILTER, and the annotation fieldANN. When filtering based onANN, annotation entries are filtered first. If no annotation entry remains, the entire variant is deleted.vembrane relies onpysamfor reading/writing VCF/BCF files.For a comparison with similar tools have a look at thevembrane benchmarks.Installationvembrane is available inbiocondaand can either be installed into an existing conda environment withmamba install -c bioconda vembraneor into a new named environmentmamba create -n environment_name -c bioconda vembrane.
Alternatively, if you are familiar with git andpoetry, clone this repository and runpoetry install.
Seedocs/develop.mdfor further details.vembrane filterUsagevembrane takes two positional arguments: The filter expression and the input file; the latter may be omitted to read fromstdininstead, making it easy to use vembrane in pipe chains.usage: vembrane filter [options] expression [input vcf/bcf]

options:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
                        Output file. If not specified, output is written to STDOUT.
  --output-fmt {vcf,bcf,uncompressed-bcf}, -O {vcf,bcf,uncompressed-bcf}
                        Output format.
  --annotation-key FIELDNAME, -k FIELDNAME
                        The INFO key for the annotation field. Defaults to "ANN".
  --aux NAME=PATH, -a NAME=PATH
                        Path to an auxiliary file containing a set of symbols.
  --keep-unmatched      Keep all annotations of a variant if at least one of them
                        passes the expression (mimics SnpSift behaviour).
  --preserve-order      Ensures that the order of the output matches that of the input.
                        This is only useful if the input contains breakends (BNDs)
                        since the order of all other variants is preserved anyway.Filter expressionThe filter expression can be any valid python expression that evaluates tobool. However, functions and symbols available have been restricted to the following:all,anyabs,len,max,min,round,sumenumerate,filter,iter,map,next,range,reversed,sorted,zipdict,list,set,tuplebool,chr,float,int,ord,strAny function or symbol frommathAny function fromstatisticsRegular expressions viarecustom functions:without_na(values: Iterable[T]) -> Iterable[T](keep only values that are notNA)replace_na(values: Iterable[T], replacement: T) -> Iterable[T](replace values that areNAwith some other fixed value)genotype related:count_hom,count_het,count_any_ref,count_any_var,count_hom_ref,count_hom_varis_hom,is_het,is_hom_ref,is_hom_varhas_ref,has_varAvailable fieldsThe following VCF fields can be accessed in the filter expression:NameTypeInterpretationExample expressionINFODict[str, Any¹]INFO field -> ValueINFO["DP"] > 0ANNDict[str, Any²]ANN field -> ValueANN["Gene_Name"] == "CDH2"CHROMstrChromosome NameCHROM == "chr2"POSintChromosomal position (1-based)24 < POS < 42ENDintChromosomal end position (1-based, inclusive, NA for breakends); also accessible viaINFO["END"]24 < END < 42IDstrVariant IDID == "rs11725853"REFstrReference alleleREF == "A"ALTstrAlternative allele³ALT == "C"QUALfloatQualityQUAL >= 60FILTERList[str]Filter tags"PASS" in FILTERFORMATDict[str, Dict[str, Any¹]]Format -> (Sample -> Value)FORMAT["DP"][SAMPLES[0]] > 0SAMPLESList[str][Sample]"Tumor" in SAMPLESINDEXintIndex of variant in the fileINDEX < 10¹ depends on type specified in VCF header² for the usual snpeff and vep annotations, custom types have been specified; any unknown ANN field will simply be of typestr. If something lacks a custom parser/type, please consider filing an issue in theissue tracker.³ vembrane does not handle multi-allelic records itself. Instead, such files should be
preprocessed by either of the following tools (preferably even before annotation):bcftools norm -m-any […]gatk LeftAlignAndTrimVariants […] --split-multi-allelicsvcfmulti2oneallele […]ExamplesOnly keep annotations and variants where gene equals "CDH2" and its impact is "HIGH":vembranefilter'ANN["Gene_Name"] == "CDH2" and ANN["Annotation_Impact"] == "HIGH"'variants.bcfOnly keep variants with quality at least 30:vembranefilter'QUAL >= 30'variants.vcfOnly keep annotations and variants where feature (transcript) is ENST00000307301:vembranefilter'ANN["Feature"] == "ENST00000307301"'variants.bcfOnly keep annotations and variants where protein position is less than 10:vembranefilter'ANN["Protein_position"].start < 10'variants.bcfOnly keep variants where mapping quality is exactly 60:vembranefilter'INFO["MQ"] == 60'variants.bcfOnly keep annotations and variants where consequence contains the word "stream" (matching "upstream" and "downstream"):vembranefilter're.search("(up|down)stream", ANN["Consequence"])'variants.vcfOnly keep annotations and variants where CLIN_SIG contains "pathogenic", "likely_pathogenic" or "drug_response":vembranefilter\'any(entry in ANN["CLIN_SIG"]for entry in ("pathogenic", "likely_pathogenic", "drug_response"))'\variants.vcfUsing set operations, the same may also be expressed as:vembranefilter\'not {"pathogenic", "likely_pathogenic", "drug_response"}.isdisjoint(ANN["CLIN_SIG"])'\variants.vcfFilter on sample specific values:by sample name:vembranefilter'FORMAT["DP"]["specific_sample_name"] > 0'variants.vcfby sample index:vembranefilter'FORMAT["DP"][0] > 0'variants.vcfby sample name based on the index in the list ofSAMPLES:vembranefilter'FORMAT["DP"][SAMPLES[0]] > 0'variants.vcfusing all or a subset ofSAMPLES:vembranefilter'mean(FORMAT["DP"][s] for s in SAMPLES) > 10'variants.vcfFilter on genotypes for specific samples (named "kid", "mom", "dad"):vembranefilter\'is_het("kid") and is_hom_ref("mom") and is_hom_ref("dad") and \all(FORMAT["DP"][s] > 10 for s in ["kid", "mom", "dad"])'\variants.vcfExplicitly access theGTfield for the first sample in the file:vembranefilter'FORMAT["GT"][0] == (1, 1)'variants.vcfCustomANNtypesvembraneparses entries in the annotation field as outlined indocs/ann_types.md.Missing values in annotationsIf a certain annotation field lacks a value, it will be replaced with the special value ofNA. Comparing with this value will always result inFalse, e.g.ANN["MOTIF_POS"] > 0will always evaluate toFalseifthere was no value in the "MOTIF_POS" field of ANN (otherwise the comparison will be carried out with the usual semantics).For fields with custom types, such asANN["Protein_position"]which is of typePosRangewith attributesstart,endandlength, trying to accessANN["Protein_position"].startwill result inNAif there was no value for"Protein_position"in the annotation of the respective record, i.e. the access will returnNAinstead of raising anAttributeError.
In general, any attribute access onNAwill result inNA(and issue a warning to stderr).Since you may want to use the regex module to search for matches,NAalso acts as an emptystr, such thatre.search("nothing", NA)returns nothing instead of raising an exception.Explicitlyhandling missing/optional values in INFO or FORMAT fields can be done by checking for NA, e.g.:INFO["DP"] is NA.Handling missing/optional values in fields other than INFO or FORMAT can be done by checking for None, e.gID is not None.Sometimes, multi-valued fields may contain missing values; in this case, thewithout_nafunction can be convenient, for example:mean(without_na(FORMAT['DP'][s] for s in SAMPLES)) > 2.3. It is also possible to replaceNAwith some constant value with thereplace_nafunction:mean(replace_na((FORMAT['DP'][s] for s in SAMPLES), 0.0)) > 2.3Auxiliary filesvembranesupports additional files, such as lists of genes or ids with the--aux NAME=path/to/fileoption. The file should contain one item per line and is parsed as a set. For examplevembrane filter --aux genes=genes.txt "ANN['SYMBOL'] in AUX['genes']" variants.vcfwill keep only records where the annotated symbol is in the set specified ingenes.txt.vembrane tagWhilevembrane filterremoves/skips records which do not pass the supplied expression,vembrane tagapplies tags to records which do pass the expression.
In other words,tagis a non-destructive version offilter, which only adds tags to records (hence never removes information from the input).
As such, multiple tags can be applied to a single record in the same run.Note that the VCF specification explicitly defines theFILTERfield as a "list of codes for filters thatfail" (emphasis by us).
As such,PASSshouldindicate that all filters have passed, and.(missing)shouldindicate that no filters have been applied.
Custom tags should therefore indicate whether a record failed a certain filter, and not whether it passed.However, the default behaviour is to tag records which pass the expression, and not those which fail.
To toggle between applying tags in a positive (tagging records thatpass) or negative (tagging records thatfail) sense, use the--tag-mode [pass|fail]/-m [pass|fail]flag.Tag names must neither contain whitespace nor semicolons. Additionally, the tag name must also not be"0".ExamplesTag records with quality at least 30 with a tag namedquality_at_least_30:vembranetag--tagquality_at_least_30="QUAL >= 30"variants.vcfTag records with quality at least 30 with a tag namedquality_at_least_30and records which have "Illumina" in their list of platforms with a tag namedillumina:vembranetag-tquality_at_least_30="QUAL >= 30"-tillumina"'Illumina' in INFO['platformnames']"variants.vcfTag records with qualityless than30 with a tag namedq_below_30using the--tag-mode failsetting:vembranetag--tag-modefail--tagq_below_30="QUAL >= 30"variants.vcfTag records with qualityless than30 with a tag namedq_below_30by negating the expression itself:vembranetag--tagq_below_30="not (QUAL >= 30)"variants.vcfvembranetag--tagq_below_30="QUAL < 30"variants.vcfvembrane tableIn addition to thefiltersubcommand, vembrane (≥ 0.5) also supports writing tabular data with thetablesubcommand.
In this case, an expression which evaluates totupleis expected, for example:vembranetable'CHROM, POS, 10**(-QUAL/10), ANN["CLIN_SIG"]'input.vcf>table.tsvWhen handlingmulti-sample VCFs, you often want to iterate over all samples in a record by looking at aFORMATfield for all of them.
However, if you use a standard Python list comprehension (something like[FORMAT['DP'][sample] for sample in SAMPLES]), this would yield a single column with a list containing one entry per sample (something like[25, 32, 22]for three samples with the respective depths).In order to have a separate column for each sample, you can use thefor_each_sample()function in both the mainvembrane tableexpression and the--headerexpression.
It should contain onelambda expressionwith exactly one argument, which will be substituted by the sample names in the lambda expression.For example, you could specifiy expressions for the--headerand the main VCF record evaluation like this:vembranetable--header'CHROM, POS, for_each_sample(lambda sample: f"{sample}_depth")''CHROM, POS, for_each_sample(lambda s: FORMAT["DP"][s])'input.vcf>table.tsvGiven a VCF file with samplesSample_1,Sample_2andSample_3, the header would expand to be printed as:CHROM  POS   Sample_1_depth   Sample_2_depth   Sample_3_depthand the expression to evaluate on each VCF record would become:(CHROM,POS,FORMAT['DP']['Sample_1'],FORMAT['DP']['Sample_2'],FORMAT['DP']['Sample_3'])When not supplying a--headerexpression, the entries of the expanded main expression become the column names in the header.
When supplying a header via--header,  itsfor_each_sample()expects an expression which can be evaluated tostrand must have the same number of fields as the main expression.Please note that, as anywhere in vembrane, you can use arbitrary Python expressions infor_each_sample()lambda expressions.
So you can for example perform computations on fields or combine multiple fields into one value:vembranetable'CHROM, POS, for_each_sample(lambda sample: FORMAT["AD"][sample] / FORMAT["DP"][sample] * QUAL)'input.vcf>table.tsvInstead of using thefor_each_sample(wide format) machinery, it is also possible to generate the data in long format by specifying the--longflag.
In this case, the first column will always be calledSAMPLEand there's an additional variable of the same name available for the expressions.
For example:vembranetable--long'CHROM, POS, FORMAT["AD"][SAMPLE] / FORMAT["DP"][SAMPLE] * QUAL'input.vcf>long_table.tsvvembrane annotatevembrane is able to annotate vcf files with a given table-like file. In addition to the vcf and annotation file, the user has to provide a configuration file.Configuration (Example):## example.yamlannotation:file:"example.tsv"# the table-like annotation file column with headercolumns:chrom:"chrom"# column name of the annotation file refering to the chromosomestart:"chromStart"# column name of the annotation file refering to the chromosome startstop:"chromEnd"# column name of the annotation file refering to the chromosome enddelimiter:"\t"# delimiter of the columnsvalues:-value:# a new annotation entry in the info field of the vcfvcf_name:"genehancer_score"# the name of annotation entrynumber:"1"# number of values for each entrydescription:"Scorefromgenehancer."# description of this entry in the headertype:"Float"# type of the valuesexpression:"DATA['score'][0]"# any python expression to calculate the value(s)# DATA['score'] refers to the 'score' column of the annotation field-value:# a second annotation entry to annotatevcf_name:"genehancer_score2"number:"1"description:"Scorefromgenehancer."type:"Float"expression:"log(max(DATA['score'])*2)"example.tsv (Example):chrom	chromStart	chromEnd	name	score
chr10	76001	77000	HJSDHKD	463
chr10	120054	130024	HJSJHKD	463
chr10	432627	492679	IDASJLD	327
chr10	540227	872071	SZAGHSD	435
chr10	654480	1000200	HSJKJSD	12Exemplary invocation:vembrane annotate example.yaml example.bcf > annotated.vcf.Internally for each vcf record the overlapping regions of the annotation file are determined and stored inDATA. The expression may then access theDATAobject and its columns by the columns names to generate a single or multiple values of cardinalitynumberof typetype. These values are stored in the new annotation entry under the namevcf_nameand with header descriptiondescription.CitationCheck the "Cite this repository" entry in the sidebar for citation options.Also, please readshould-I-cite-this-softwarefor background.AuthorsMarcel Bargull (@mbargull)Jan Forster (@jafors)Till Hartmann (@tedil)Johannes Köster (@johanneskoester)Elias Kuthe (@eqt)David Lähnemann (@dlaehnemann)Felix Mölder (@felixmoelder)Christopher Schröder (@christopher-schroeder)