Package: genomics-data-index
Genomics data indexThis project is to design a system which can index large amounts of genomics data and enable rapid querying of this
data.Indexingbreaks genomes up into individual features (nucleotide mutations,k-mers, orgenes/MLST)
and stores the index in a directory which can easily be shared with other people. Indexes can be generated
direct from sequence data or loaded from existing intermediate files (e.g., VCF files, MLST results).# Analyze sequence data (reads/assemblies, compressed/uncompressed)gdianalysis--reference-filegenome.gbk.gz*.fasta.gz*.fastq.gz# (Alternatively) Index features in previously computed files (VCF files, or MLST results)gdiloadvcf--reference-filereference.gbk.gzvcf-files.txt
gdiloadmlst-tseemannmlst.tsv# Load from https://github.com/tseemann/mlstgdiloadmlst-sistrsistr-profiles.csv# Load from https://github.com/phac-nml/sistr_cmdQueryingprovides both aPython APIandCommand-line interfaceto select sets of samples using this index
or attached external data (e.g., phylogenetic trees or DataFrames of metadata).Python API:# Select samples with a D614G mutation on gene Sr=s.hasa('hgvs:MN996528.1:S:D614G')# Select samples with Allele 100 for Locus (gene) adk in MLST scheme ecolir=s.hasa('ecoli:adk:100')Summaries of the features(mutations, kmers, MLST) can be exported from a set of samples alongsidenucleotide alignments,distance matricesortreesconstructed from subsets of features.r.summary_features()MutationCount10 G>T120 C>T330 A>G5Visualizationof trees and sets of selected samples can be constructed using the provided Python API and the
visualization tools provided by theETE Toolkit.r.tree_styler()\.highlight(set1)\.highlight(set2)\#....render()You can see more examples of this software in action in the providedTutorials.Table of contents1. Overview1.1. Indexing1.1.1. Naming features1.2. Querying1.2.1. Python API2. Background3. Installation3.1. Conda3.2. PyPI/pip3.3. From GitHub4. Usage4.1. Indexing4.2. Querying4.3 Main usage statement5. Tutorial6. Acknowledgements1. OverviewThe software is divided into two main components:(1) Indexingand(2) Querying.1.1. IndexingTheindexingcomponent provides a mechanism to break genomes up into individual features and store these features
in a database. The types of features supported include:Nucleotide mutations,K-mers, andGenes/MLST.1.1.1. Naming featuresIndexing assigns names to the individual features, represented as strings inspired by theSequence Position Deletion Insertion (SPDI)model.Nucleotide mutations:sequence:position:deletion:insertion(e.g.,ref:100:A:T)Genes/MLST:scheme:locus:allele(e.g.,ecoli:adk:100)Kmers:Not implemented yetAlternatively, forNucleotide mutationsnames can be given usinghgvs(as output bySnpEff).Nucleotide mutations:hgvs:sequence:gene:p.protein_change(e.g.,hgvs:ref:geneX:p.P20H).1.2. QueryingThequeryingcomponent provides a Python API or command-line interface for executing queries on the genomics index. The primary type of query is aSamples querywhich returns sets of samples based on different criteria. These criteria
are grouped into differentMethods. Each method operates on a particular type ofDatawhich could include features stored in thegenomics indexas well as trees or external metadata.1.2.1. Python APIAn example query on an existing set of samplesswould be:r=s.isa('B.1.1.7',isa_column='lineage')\.isin(['SampleA'],distance=1,units='substitutions')\.hasa('MN996528.1:26568:C:A')This would be read as:Select all samples inswhich are aB.1.1.7 lineageas defined in some attached DataFrame (isa())ANDwhich are within1 substitutionofSampleAas defined on a phylogenetic tree (isin())ANDwhich have aMN996528.1:26568:C:Amutation (hasa()).Note: I have left out some details in this query. Full examples for querying are available atTutorial 1: Salmonella dataset.2. BackgroundA paper on this project is in progress. A detailed description is found in myThesis.Additionally, a poster on this project can be found atimmem2022.3. Installation3.1. CondaCondais a package and environment management software which makes it very easy to install and maintain dependencies of software without requiring administrator/root access. Packages from conda are provided using differentchannelsand thebiocondachannel contains a very large collection of bioinformatics software which can be automatically installed. To make use of conda you will have to first download andinstall conda. Once installed you can use the commandcondato install software/manage conda environments.To install this software, we will first, create a conda environment with the necessary dependencies as follows (a full conda package is not available yethttps://github.com/apetkau/genomics-data-index/issues/51).condacreate-cconda-forge-cbioconda-cdefaults--namegdipython=3.8pyqtbedtoolsiqtree'bcftools>=1.13''htslib>=1.13'# Activate environment. Needed to install additional Python dependencies below.condaactivategdiNow, you can install with:pipinstallgenomics-data-indexIf everything is working you should be able to run:gdi--versionYou should seegdi, version 0.1.0printed out.Additional dependenciesFor snpeff to work you will need to install the packagemkisofson Ubuntu (e.g.,sudo apt install mkisofs).
I do not know the exact package name on other systems.3.2. PyPI/pipTo install just the Python component of this project fromPyPIyou can run the following:pipinstallgenomics-data-indexNote that you will have to install some additional dependencies separately. Please see theconda-env.yamlenvironment file for details.3.3. From GitHubTo install the project from the source on GitHub please first clone the git repository:gitclonehttps://github.com/apetkau/genomics-data-index.gitcdgenomics-data-indexNow install all the dependencies usingcondaandbiocondawith:condaenvcreate-fconda-env.yaml
condaactivategdiOnce these are installed you can setup the Python package with:pipinstall.4. UsageThe main command is calledgdi. A quick overview of the usage is as follows:4.1. Indexing# Create new index in `index/`# cd to `index/` to make next commands easier to rungdiinitindexcdindex# Creates an index of mutations (VCF files) and kmer sketches (sourmash)gdianalysis--use-conda--include-kmer--kmer-size31--reference-filegenome.gbk.gz*.fastq.gz# (Optional) build tree from mutations (against reference genome `genome`) for phylogenetic queryinggdirebuildtree--align-typefullgenomeThe produced index will be in the directoryindex/.4.2. Querying# List indexed samplesgdilistsamples# Query for genomes with mutationgdiquerymutation'genome:10:A:T'4.3. Main usage statementUsage: gdi [OPTIONS] COMMAND [ARGS]...

Options:
  --project-dir TEXT              A project directory containing the data and
                                  connection information.

  --ncores INTEGER RANGE          Number of cores for any parallel processing
                                  [default: 8]

  --log-level [DEBUG|INFO|WARNING|ERROR|CRITICAL]
                                  Sets the log level  [default: INFO]
  --version                       Show the version and exit.
  --config FILE                   Read configuration from FILE.
  --help                          Show this message and exit.

Commands:
  analysis
  build
  db
  export
  init
  input
  list
  load
  query
  rebuild5. TutorialTutorials and a demonstration of the software are available below (code inseparate repository). You can select the[launch | binder]badge to launch each of these tutorials in an interactiveJupyterenvironment within the cloud environment usingBinder.Tutorial 1: Querying (Salmonella)-In case GitHub link is not rendering tryhereTutorial 2: Indexing assemblies (SARS-CoV-2)-In case GitHub link is not rendering tryhereTutorial 3: Querying overview-In case GitHub link is not rendering tryhereAlternatively, you can run these tutorials on your local machine. In order to run these tutorials you will first have to install thegenomics-data-indexsoftware (see theInstallationsection for details). In addition, you will have to installJupyter Lab. If you have already installed thegenomics-data-indexsoftware with conda you can install Jupyter Lab as follows:condaactivategdi
condainstalljupyterlabTo run Jupyter you can run the following:# QT_QPA_PLATFORM The below is useful to avoid having to set the DISPLAY env variable for Qt# You can ignore setting this environment variable if you are running on a machine with an X server installed and configuredQT_QPA_PLATFORM="offscreen"jupyterlabPlease see the instructions forJupyter Labfor details.6. AcknowledgementsI would like to acknowledge thePublic Health Agency of Canada, theUniversity of Manitoba,
and theVADA Programfor providing me with the opportunity, resources and training for working on this project.Some icons used in this documentation are provided byFont Awesomeand licensed under aCreative Commons Attribution 4.0license.