Package: zorp
ZORP: A helpful GWAS parserWhy?ZORP is intended to abstract away differences in file formats, and help you work with GWAS data from many
different sources.Provide a single unified interface to read text, gzip, or tabixed dataSeparation of concerns between reading and parsing (with parsers that can handle the most common file formats)Includes helpers to auto-detect data format and filter for variants of interestWhy not?ZORP provides a high level abstraction. This means that it is convenient, at the expense of speed.For GWAS files, ZORP does not sort the data for you, because doing so in python would be quite slow. You will still
need to do some basic data preparation before using.InstallationBy default, zorp installs with as few python dependencies as practical. For more performance, and to use special
features, install the additional required dependencies as follows:$ pip install zorp[perf,lookups]The snp-to-rsid lookup requires a very large file in order to work efficiently. You can download the pre-generated file
using thezorp-assetscommand line script, as follows.
(use "--no-update" to skip warnings about already having the latest version)$zorp-assetsdownload--typesnp_to_rsid--taggenome_buildGRCh37--no-update
$zorp-assetsdownload--typesnp_to_rsid--taggenome_buildGRCh37Or build it manually (which may require first downloading a large source file):$ zorp-assets build --type snp_to_rsid --tag genome_build GRCh37Assets will be downloaded to the least user-specific location available, which may be overridden by setting the
environment variableZORP_ASSETS_DIR. Runzorp-assets show --allto see the currently selected asset directory.A note on rsID lookupsWhen developing on your laptop, you may not wish to download 16 GB of data per rsID lookup. A much smaller "test"
dataset is available, which contains rsID data for a handful of pre-selected genes of known biological functionality.$ zorp-assets download --type snp_to_rsid_test --tag genome_build GRCh37To use it in your python script, simply add an argument to the SnpToRsid constructor:rsid_finder = lookups.SnpToRsid('GRCh37', test=True)If you have generated your own lookup using the code in this repo (make_rsid_lookup.py), you may also replace
the genome build with a hardcoded path to the LMDB file of lookup data. This use case is fairly uncommon, however.UsagePythonfromzorpimportlookups,readers,parsers# Create a reader instance. This example specifies each option for clarity, but sniffers are provided to auto-detect#   common format options.sample_parser=parsers.GenericGwasLineParser(marker_col=1,pvalue_col=2,is_neg_log_pvalue=True,delimiter='\t')reader=readers.TabixReader('input.bgz',parser=sample_parser,skip_rows=1,skip_errors=True)# After parsing the data, values of pre-defined fields can be used to perform lookups for the value of one field#  Lookups can be reusable functions with no dependence on zorprsid_finder=lookups.SnpToRsid('GRCh37')reader.add_lookup('rsid',lambdavariant:rsid_finder(variant.chrom,variant.pos,variant.ref,variant.alt))# Sometimes a more powerful syntax is needed- the ability to look up several fields at once, or clean up parsed data#   in some way unique to this datasetreader.add_transform(lambdavariant:mutate_entire_variant(variant))# We can filter data to the variants of interest. If you use a domain specific parser, columns can be referenced by namereader.add_filter('chrom','19')# This row must have the specified value for the "chrom" fieldreader.add_filter(lambdarow:row.neg_log_pvalue>7.301)# Provide a function that can operate on all parsed fieldsreader.add_filter('neg_log_pvalue')# Exclude values with missing data for the named field# Iteration returns containers of cleaned, parsed data (with fields accessible by name).forrowinreader:print(row.chrom)# Tabix files support iterating over all or part of the fileforrowinreader.fetch('X',500_000,1_000_000):print(row)# Write a compressed, tabix-indexed file containing the subset of variants that match filters, choosing only specific#   columns. The data written out will be cleaned and standardized by the parser into a well-defined format.out_fn=reader.write('outfile.txt',columns=['chrom','pos','pvalue'],make_tabix=True)# Real data is often messy. If a line fails to parse, the problem will be recorded.fornumber,message,raw_lineinreader.errors:print('Line{}failed to parse:{}'.format(number,message))Command line file conversionThe file conversion feature of zorp is also available as a command line utility. Seezorp-convert --helpfor details
and the full list of supported options.This utility is currently in beta; please inspect the results carefully.To auto-detect columns based on a library of commonly known file formats:$ zorp-convert --auto infile.txt --dest outfile.txt --compressOr specify your data columns exactly:$ zorp-convert infile.txt --dest outfile.txt --index  --skip-rows 1 --chrom_col 1 --pos_col 2 --ref_col 3 --alt_col 4 --pvalue_col 5 --beta_col 6 --stderr_beta_col 7 --allele_freq_col 8The--indexoption requires that your file be sorted first. If not, you can tabix the standard output format manually
as follows.$ (head -n 1 <filename.txt> && tail -n +2 <file> | sort -k1,1 -k 2,2n) | bgzip > <filename.sorted.gz>
$ tabix <filename.sorted.gz> -p vcfDevelopmentTo install dependencies and run in development mode:pip install -e '.[test,perf,lookups]'To run unit tests, use$flake8zorp
$mypyzorp
$pytesttests/