Package: vpsearch
vpsearch - Fast Vantage-Point Tree Search for Sequence DatabasesThis is a package for indexing and querying a sequence database for fast
nearest-neighbor search by means ofvantage point
trees. For reasonably large
databases, such asRDP, this results in sequence
lookups that are typically 5-10 times faster than other alignment-based lookup
methods.Vantage-point tree search uses global-to-global alignment to compare sequences,
rather than seed-and-extend approximative methods as used for example by
BLAST.UsageGiven a sequence database (in FASTA format),vpsearch buildconstructs an
optimized vantage point search tree. Building the tree is a one-time operation
and doesn't have to be done again unless the database changes. As an
illustration, we build a vantage point tree for the RDP database of bacterial
16S sequences. This database contains 281261 sequences of which 39237 are
duplicates. After removing these duplicates, we are left with 242024 unique
sequences. Building a tree for these sequences is done with:$vpsearchbuildrdp_download_281261seqs_dedup.faBuildingfor242024sequences...done.Linearizing...done.Databasecreatedinrdp_download_281261seqs_dedup.dbFor the RDP database of full length sequences, this takes about 20 minutes on a
standard machine. When only selected regions of the sequences are considered,
the time needed to build a tree can be much reduced. For example, vantage point
trees for the v1-v2 hypervariable region (350 base pairs) or the v3-v4 region
(250 base pairs) of the RDP 16S sequencese can be built in 30 seconds to 1
minute.Once a tree has been built, unknown sequences can be looked up using thevpsearch querycommand. Here we supply a query file with a single sequencevpsearchqueryrdp_download_281261seqs_dedup.faquery.faqueryS00014371599.54152900115241152907546queryS00408592399.08152900115241152607481queryS00408592299.08152900115241152607481queryS00408592598.50153100115241152707386By default, thevpsearch querycommand outputs the best four matches in the
database per query sequence (the number of matches can be changed with the-kparameter). Lookup is done one query sequence at a time, but multiple queries
can be considered in parallel by enabling multiple threads; use the-joption
to specify the number of threads.Thevpsearch querycommand attempts to output its results in the standard
BLAST tabular format. The interpretation of the columns is as follows:Column nameExampleNotesquery IDquerysubject IDS000143715% identity99.54alignment length1529mismatches0currently not implementedgap openings0currently not implementedquery start1query end1524subject start1subject end1529E-value0N/A (always 0)bit score7546interpreted as the alignment scoreNote that the number of mismatches and gap openings are currently not displayed
in the result output. This will be addressed in a future version of the
package.InstallationUsing EDMUsers of theEnthought Deployment Manager(EDM)can install the necessary prerequisites (Click, Cython, Numpy, and Parasail) by
importing an EDM environment from the bundle file shipped with this repositoryedmenvimport-f<bundle.json>vpsearchwhere<bundle.json>is one ofvpsearch_py3.6_osx-x86_64.jsonorvpsearch_py3.6_rh6-x86_64.json, depending on your platform.When this is done, activate the environment, and install this package. From the
root of this repository, runedmshell-evpsearchpipinstall-e.Using Pip, Conda, etc.Users of other package installation tools, such as Pip or Conda, need to
install theParasaillibrary following
the instructions on the Parasail web page. Once that is done, the Python
dependencies can be installed using the appropriate command for your package
manager. For pip, for example, this can be done withpipinstall-rrequirements.txtOnce that is done, activate your virtual environment, and install this package
viapipinstall-e.Using DockerIt is possible to build a Docker image that contains vpsearch as well as all of
its dependencies. This is useful, for example, when integrating vpsearch into a
workflow manager, like Snakemake, CWL, or WDL.To build the image, run the following command from the root of this repository:dockerbuild.-tvpsearch-imageOnce the image has been built, vpsearch can then be run from within a
container. Assuming you have a FASTA file of target sequences in the filedatabase.fastain the current directory, run the following to build a
vpsearch index:dockerrun-it-v$PWD:/data-tvpsearch-imagevpsearchbuild/data/database.fastaTo query the index for a given FASTA filequery.fastaof query sequences,
run:dockerrun-it-v$PWD:/data-tvpsearch-imagevpsearchquery/data/database.db/data/query.fastaTroubleshootingThe vpsearch package relies on the Parasail C library for alignment. If
building the package fails because the Parasail library cannot be found, you
can manually specify the location of the Parasail include files and shared
object libraries by setting thePARASAIL_INCLUDE_DIRandPARASAIL_LIB_DIRenvironment variables before building the package:exportPARASAIL_INCLUDE_DIR=/location/of/parasail/include/filesexportPARASAIL_LIB_DIR=/location/of/parasail/lib/filespipinstall-e.Note that if Parasail is installed in a non-standard location, you may have to
set theLD_LIBRARY_PATHvariable at runtime.Implementation notesThe tree construction operates in two phases. We first build the tree as a tree
of Python object nodes because it's easier to build with a dynamic data
structure. Then it linearizes the topology of the nodes into a few integer
arrays that are easy to serialize and fast to look up. The object that
represents the linearized tree can only query the database, not build the tree.
The slower tree-of-nodes implementation can build and query (albeit with more
overhead).Building wheelsWheels for this package can be built in a platform-independent way usingcibuildwheel, running under
GitHub actions. As an administrator, you can start a workflow to build wheels
by selecting the "Build wheels" action from the GitHub actions menu, and
clicking the "Run workflow" button. When the workflow completes, wheels for
Linux and macOS will be available as a zipped artifact.It is possible to run cibuildwheels locally, but only to build wheels for
Linux. In a clean Python environment, runpip install cibuildwheelto install
the tool, followed by e.g.CIBW_BUILD_VERBOSITY=1\CIBW_BUILD=cp38-manylinux_x86_64\CIBW_BEFORE_BUILD="./ci/build-parasail.sh"\python-mcibuildwheel--output-dirwheelhouse--platformlinuxto build Python 3.8 wheels for Linux. By varying the build tag, wheels for
other Python versions can be built.LicenseThis package is licensed under theBSD license.ReferencesVantage point trees were introduced inUhlmann, Jeffrey (1991). "Satisfying General Proximity/Similarity Queries
with Metric Trees". Information Processing Letters. 40 (4):
175–179. doi:10.1016/0020-0190(91)90074-r.Yianilos (1993). Data structures and algorithms for nearest neighbor search
in general metric spaces (PDF). Fourth annual ACM-SIAM symposium on Discrete
algorithms. Society for Industrial and Applied Mathematics Philadelphia, PA,
USA. pp. 311–321. pny93.The Parasail library is described inDaily, Jeff. (2016). Parasail: SIMD C library for global, semi-global, and
local pairwise sequence alignments. BMC Bioinformatics, 17(1),
1-11. doi:10.1186/s12859-016-0930-z