Package: xengsort
xengsort: Fast lightweight accurate xenograft sortingThis tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the xenograft sorting problem.For a description of the method an evaluation on several datasets, please see ourarticle in "Algorithms for Molecular Biology", or the initialWABI 2020 publication.
(There was also apreprint on bioRxiv.)
BibTeX entrys for citation can be found at the end of this README file.In case of problems,please read the tutorial and/or usage guide in this file carefully,check the troubleshooting section below if your problem is solved there,and file an issue in the issue tracker if this does not help or you have discovered an error in the documentation.See CHANGELOG.md for recent changes.
Thank you!Tutorial: Classification of human-captured mouse exomesWe here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
This tutorial uses the workflow management systemSnakemaketo run the example.
This is, however, not necessary to usexengsort.
Please refer to the Usage Guide below to learn how to runxengsorton your own data manually.Our softwarexengsortis provided as a Python package.
For efficiency, it uses just-in-time compilation provided by thenumbapackage.
We here use the package managercondato manage Python packages and to use a separate environment forxengsort.Install the conda package manager (miniconda)Go tohttps://docs.conda.io/en/latest/miniconda.htmland download the Miniconda installer:
Choose Python 3.10 (or higher), your operating system, and preferably the 64 bit version.
Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
You can let the installer do it, or do it manually by editing your.bashrcor similar file under Linux or MacOS, or by editing the environment variables under Windows.
To verify that the installation works, open a new terminal and executeconda --version # ideally 22.9.xx or higher
python --version # ideally 3.10.xx or higherInstallation via condaxengsortis available via bioconda. To create a new environment includingxengsort, runconda create --name xengsort -c bioconda xengsortIf you want to install xengsort into an existing environment, activate it withconda activate MYEVIRONMENTand executeconda install -c bioconda xengsortManual installation via gitlabObtain or update xengsortOur software can also be obtained by cloning this public git repository:git clone https://gitlab.com/genomeinformatics/xengsort.gitIf you need to update xengsort later, you can do so by just executinggit pullwithin the cloned directory tree.Create and activate a conda environmentTo run our software, acondaenvironment with the required libraries needs to be created.
A list of needed libraries is provided in theenvironment.ymlfile in the cloned repository;
it can be used to create a new environment:cd xengsort  # the directory of the cloned repository
conda env createwhich will create an environment namedxengsortwith the required dependencies,
using the providedenvironment.ymlfile in the same directory.A more explicit alternative is to runconda create --name xengsort -c conda-forge -c bioconda --file requirements.txtwhich will do the same thing using the packages mentioned inrequirements.txt.Note:While xengsort works on Windows/Mac without problems, some of the required packages may currently not exist for Windows/Mac.Setting up the environment may take some time, as conda searches and downloads packages.
After all dependencies are resolved, you activate the environment and install the package from the repository into this environment.
Make sure that you are in the root directory of the cloned repository (where thisREADME.mdfile or theCHANGELOG.mdfile is) and runconda activate xengsort  # activate environment
pip install -e .  # install xengsort package using pipRun the Snakemake example workflowYou can in principle removesnakemake-minimalandsra-toolkitfrom the list of packages.
However, you will not be able to automatically run the provided example workflow or download sequence datasets from SRA.We provide an example Snakemake workflow (for publicly available mouse exomes), which downloads all needed reference FASTA files and exome FASTQ files, generates a hash table as an index and classifies the reads.To run this workflow,you will need space for the downloaded datasets and result files, so snakemake should be executed in a separate working directory with lots of space for the datset.you will additionally need thesnakemake-minimalpackage.
Make sure you are in the working directory, that your conda environment calledxengsortis active, and then additionally install Snakemake:condainstall-cbiocondasnakemake-minimalensure that xengsort'sSnakefileis present in the working directory.
It can be symbolically linked, such as:cd/path/to/workdir
ln-s/path/to/xengsort/SnakefileSnakefileNow Snakemake can be run as follows:snakemake-n# dry-run: What will happen?snakemake-j16--use-conda# execute with 16 threadsThe-n(or--dry-run) option first performs a dry run and prints out what will be done.
In the real invocation, the-j 16option uses up to 16 CPU cores to execute 16 separate jobs in parallel or one job that uses 16 threads, or any combination.You can examine the (commented) Snakefile to see what happens in each step:Reference sequences (human and mouse genome and transcriptome) will be downloaded from Ensembl in FASTA format.The k-mer index (for k=25) will be built from the FASTA files (xengsort index).An example dataset (a human-captured mouse exome) will be downloaded from SRA using sra-toolkit and saved in FASTQ format (paired-end reads).The paired-end reads in the dataset will be classified according to species (xengsort classify)This will run for quite some time (especially downloads and index creation), best leave it overnight.
The results will appear in theresults/directory.To run your own human/mouse xenograft analysis, you can continue to use the same index.
All reference files, including the index in ZARR format (.zarr) are in theref/directory.
You can follow the structure of the Snakefile to create your own custom workflow.
Additional information on how to usexengsortis given in the usage guide below.Usage Guidexengsort is a multi-command tool with several subcommands (like git), in particularxengsort indexbuilds an index (a bucketed 3-way Cuckoo hash table)xengsort classifyclassifies a sequences sample (FASTQ files) using an existing indexIt is a good idea to runxengsort index --helpto see all available options.
Using--helpworks on any subcommand.How to build an indexTo build an index for xengsort, several parameters must be provided, which are described in the following.First, a file name and a path for the index must be chosen.
The index is stored in two files. We will usemyindexto store the index in the current folder.Second, two reference genomes (host and graft) must be provided (in FASTA files).
We assume that they are given ashost.fa.gzandgraft.fa.gz.
The files can be provided as an uncompressed file or compressed usinggzip,bzip2orxz.
The corresponding options are-Hor--hostand-Gor--graft.
Each option can take several arguments as files.xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000  [OPTIONS]We must specify the size of the hash table:-nor--nobjects: number of k-mers that will be stored in the hash table. This depends on the used refernece genomes and must be estimated beforehand! As a precise estimate of the number of different k-mers can be difficult, you can err on the safe side and provide a generously large estimate, examine the final (low) load factor and then rebuild the index with a smaller-nparameter to aechieve the desired load. There are also some tools that quickly estimate the number of distinct k-mers in large files, such asntCardorKmerEstimate. As a guide: Human and mouse genome and transcriptome together comprise around 4.5 billion 25-mers, as shown in the examples above.This option must be specified; there is no default!We may further specify additional properties of the hash table:-por--pagesizeindicates how many elements can be stored in one bucket (or page). This is 4 by default.--fillbetween 0.0 and 1.0 describes the desired fill rate or load factor of the hash table.
Together with-n, the number of slots in the table is calculated asceil(n/fill). In our experiments we used 0.88. (The number of buckets is then the smallest odd integer that is at leastceil(ceil(n/fill)/p).)--alignedor--unaligned: indicates whether each bucket should consume a number of bits that is a power of 2. Using--alignedensures that each bucket stays within the same cache line, but may waste space (padding bits), yielding faster speed but possibliy (much!) larger space requirements. With--unaligned, no bits are used for padding and buckets may cross cache line boundaries. This is slightly slower, but may save a little or a lot of space (depending on the bucket size in bits). The default is--unaligned, because the speed decrease is small and the memory savings can be significant.--hashfunctionsdefines the parameters for the hash functions used to store the key-value pairs. If the parameter is unspecified, different random functions are chosen each time. The hash functions can be specified using a colon separated list:--hashfunctions linear945:linear9123641:linear349341847. It is recommended to have them chosen randomly unless you need strictly reproducible behavior, in which case the example given here is recommended.The final important parameter is about paralellization:-Tdefines how many threads are used to calculate weakk-mers.The following hash table options should not be modified from their default values and are for internal development only:-Por--parameterscan be used to combine all parameters stated above into one string (see the--helptext)How to classifyTo classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
Then run thexengsort classifycommand with a previously built index, such asxengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode countfor single-end reads, orxengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode countfor paired-end reads.Out tool offers three different classification modes. The algorithm can be specified using the--modeparameter.count: Classifies the read based on the number of $k$-mers that belong to host or graft.coverage: Classifies the read based on the proportions covered by $k$-mers of each class.quick: Classifies the read only based on the third and third last $k$-mer.The parameter--prefixor equivalently--outis required and defines the prefix for all output files; this can be a combination of path and file prefix, such as/path/to/sorted/samplename.  Typically, there will be 5 output files for each of the first and the second read pair:{prefix}.host.1.fq.gz: host reads{prefix}.graft.1.fq.gz: graft reads{prefix}.both.1.fq.gz: reads that could originate from both{prefix}.neither.1.fq.gz: reads that originate from neither host nor graft{prefix}.ambiguous.1.fq.gz: (few) ambiguous reads that cannot be classified,and similarly with.2.fq.gz. For single-end reads, there is only.fq.gz(no numbers).The compression type can be specified using the--compressionparameter.
Currently we supportgz(default),bzip,xzandnone(uncompressed).Further parameters and options are:-Tdefines how many threads are used for classification (4 to 8 is recommended).--filter: With this flag, only the graft reads are output.--count: With this flag, only the number of reads in each category is counted, but the reads are not sorted.Please report aGitLab issuein case of problems.Happy sorting!TroubleshootingSnakemake throws errors when attempting to download the example dataset from SRA.This may happen for several reasons:The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.Another source of trouble may be that for some reason and older version ofsra-toolsgets called. We use thefasterq-dumptool to download the data, which did not exist in older verisons. To use a clean environment with a recent version ofsra-tools, you can run snakemake assnakemake -j 16 --use-conda, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (xengsort --version). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and runpip install -e ., as explained above in the tutorial.Indexing stops after a few seconds with a failure.The most likely cause of this is that you have not specified the size of the hash table (-nparameter).
(From version 1.0.1, the-nparameter is required, and the error cannot occur anymore. Please update to the latest version!)
For the human and mouse genomes this is approximately-n 4_500_000_000(4.5 billion).
It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
If uncertain, use a generous overestimate (e.g., add the sizes of the two genomes) and look in the output for the line that starts withchoice nonzero.
This is the exact number of k-mers stored.
You can use this value to re-index everything in a second iteration.Thexengsort classifystep throws an error aboutNoneTypevs.str.This can happen if you do not specify the--outor--prefixparameter.
(From version 1.0.2, this parameter is required, and the error cannot occur anymore. Please update to the latest version!)CitationIf you use xengsort, please cite the article in "Algorithms for Molecular Biology".
The BibTeX entry is provided here:@Article{pmid33810805,
   Author="Jens Zentgraf and Sven Rahmann",
   Title="Fast lightweight accurate xenograft sorting",
   Journal="Algorithms Mol Biol",
   Year="2021",
   Volume="16",
   Number="1",
   Pages="2",
   Month="Apr"
}In addition, you may also cite the WABI 2020 proceedigs paper:@InProceedings{ZentgrafRahmann2020xengsort,
  author =	{Jens Zentgraf and Sven Rahmann},
  title =	{Fast Lightweight Accurate Xenograft Sorting},
  booktitle =	{20th International Workshop on Algorithms in Bioinformatics (WABI 2020)},
  pages =	{4:1--4:16},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-161-0},
  ISSN =	{1868-8969},
  year =	{2020},
  volume =	{172},
  editor =	{Carl Kingsford and Nadia Pisanti},
  publisher =	{Schloss Dagstuhl--Leibniz-Zentrum f{\"u}r Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{https://drops.dagstuhl.de/opus/volltexte/2020/12793},
  URN =		{urn:nbn:de:0030-drops-127933},
  doi =		{10.4230/LIPIcs.WABI.2020.4},
  annote =	{Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
}