Package: bed-lookup
Allows very fast searching of a bed file of any size by gene/snp location.For example:frombed_lookupimportBedFileb=BedFile('my_bed.bed')gene=b.lookup('chr3',1000104)This module requires cython, and should work with recent versions of
python2 and python3.It can also be used with a pandas dataframe directly:df['new_col']=b.lookup_df(df,'chrom','pos')Note: with large dataframes, this function can be very slow, but there is
a nice trick to speed it up:importnumpyasnpimportpandasaspdfrommultiprocessingimportPool,cpu_countpool=Pool()b=BedFile('my_bed.bed')df=pd.read_csv('big_table.txt.gz',sep='\t',compression='gzip')dfs=np.array_split(df,cpu_count())run=[]out=[]# Our chromsome column is 'chrom' and position column is 'pos'.fordindfs:run.append(pool.apply_async(b.lookup_df,(d,'chrom','pos')))forrinrun:out.append(r.get())df['new_col']=pd.concat(out)InstallationInstallation follows the standard python syntax:gitclonehttps://github.com/MikeDacre/python_bed_lookupcdpython_bed_lookuppythonsetup.pybuildsudopythonsetup.pyinstallIf you do not have root permission on you device, replace the last line with:python setup.py install --userRunning from the command lineThere is a command line script calledbed_location_lookupthat will be installed
in/usr/binif you install globally or in~/.local/usr/binif you install for
your user only. The sytax for that script is:bed_location_lookup <bed_file> chr1_1000134 chr2_1859323 ....It will work for any number of gene coordinate arguments. Be aware, that there is a
file opening delay when the script is run (for small bed files this will be very
small, but for large files it can be a few seconds). It is therefore much more
efficient to call a single instance ofbed_location_lookupwith a long list of
coordinates than it is to call it once per coordinate. For a large number of
coordinates this difference can be substantial.bed_location_lookuphas a few other options also, to get those run:bed_location_lookup -hNote: if you know the bed file is large and a database already exists, you can
get considerable speed up by passing the database file instead of the raw bed
file. e.g. passbedfile.bed.dbinstead ofbedfile.bed. This bypasses the
file length check.Backend information and customizationIt makes use of a cython optimized dictionary lookup for small bed files
and sqlite for larger bed files. Which backend is being used is transparent
to the user, simply use thelookup()function as demonstrated in the
example above. The default file size cutoff is ~5 million lines in the bed
file, which results in a memory use of 1.2GB for a 5 million line long file.
The memory use scales linearly, so setting the limit at 1 million lines will
result in about 240MB of memory use. To change the file size cutoff edit the_max_lenvariable inbed_lookup/__init__.py. Be aware that the file
size limit is actually measured in bytes, for speed purposes. A dictionary of
size to file length maps is provided in the__init__.pyfile, the default
should work fine on most systems.Note that the sqlite backed is very slightly slower for lookups, however the
sqlite backend requires that a database exists already. If one does not exist
(the expected name is the bed file name followed by a.db) already then one
is created, and this step can be very slow. Hypothetically this should only be
done once.As noted above, when creating a BedFile object, a file length lookup is performed.
This lookup can be costly, particularly for gzipped files. To skip this step,
simply pass the database file to BedFile(), instead of the bedfile itself.Note: this code will work with either plain text or gzipped files, gzipped files
will be slightly slower at load due to the overhead of decompression. For large
files where an sqlite database already exists, there will be only a very slight
delay relative to the uncompressed bed file (due to file length counting).As the BedFile object is only generated once, any lookups after the creation of
this object will be very fast (less than a second) foranylength of bed file.
Smaller files will obvious result in even quicker lookups.