Package: octofludb
octofludbManage the flu-crew swine surveillance databaseInstallationoctofludbis most easily installed through PyPi:pip install octofludbYou may need to replacepipwithpip3on your system.octofludbis also dependent on the GraphDB database, which is a triplestore
database to which all data is pushed. Go tohttps://graphdb.ontotext.comand
download the GraphDB Free version.HelpYou can get a list of subcommands withoctofludb -h. More detailed
documentation is available for each subcommand, e.g.,octofludb query -h.Subcommandsoctofldubhas builtin documentation for each subcommand that describes the
all arguments and general usage. Below I will cover each of the subcommands at
a higher level with examples and background info as needed.Subcommand:init- initialize an empty octofludb databaseTo start a new database, first you have to call the daemon:$ graphdb -dThen initialize the octofludb database:$ octofludb initIf an octofludb repo is already present in GraphDB, this command does nothing.
It will NOT destroy an existing database, so is always safe to run.If you want to destroy the database and rebuild, then you can usepgraphdb.
This is a Python wrapper around GraphDB that is used internally throughoutoctofludb. It can be installed with pip:pip3 install pgraphdb.pgraphdb rm_repo octofludbSubcommand:pull- update dataUse of this command requires some understanding of theoctofludbconfiguration.Theoctofludbhome directory is~/.octofludb. This directory will is
created when you calloctofludb initand the default configuration fileconfig.yamlis copied into it. Theconfig.yamlfile is well documented and
whatever it says is more likely to be true that what I write here, so please
look it over. In short, it specifies where the data can be foundoctofludbneeds can be found.A particularly important field isoctoflu_reference, which allows you to
specify a local reference fasta file that will be used by octoFLU to classify
all strains.The commandoctofludb pull, without any additional flags, will do the
following:Go to the$OCTOFLUDB_HOME/builddirectory (create it if it doesn't exist)Load the localconfig.yamlfileLoad theschema.ttlandgeography.ttlTurtle files. These are both
stored in theoctofludbpython package in theoctofludb/datafolder.
Theschema.ttlfile includes logical relationships such as subProperty
relationship between H1 and HA that allows the database to infer that a
subject is an HA if it is an H1. Thegeography.ttlfile includes,
hierarchical relationships between countries and regions (Iowa is in the USA
and the USA is in North America) and the relationships between country names
(United Arab Emirates) and ISO country codes (ARE).Retrieve and process all Influenza A Virus data from GenBank. This step will
take a long time. Most of a day. There are two ways to speed this up. First,
if you are building a new repo, andoctofludb's GenBank processing code
hasn't changed, you can go to your~/.octofludb/builddirectory and runoctofludb upload .gb*ttl. This will upload all the past GenBank turtle
files. Of course, you can also copy over someone else's files. It is safe to
re-upload the same files many times -- no duplicate triples will be created.
Second, if you are already mostly up-to-date and just need to add the latest
few months of GenBank data, you can add a--nmonths=4argument to just
pull the data submitted in the last 4 months. All data is pulled bysubmissiondate, NOTcollectiondate, so incrementally pulling the last
few months every month will be fine.(OPTIONAL) if the--include-gisaidflag is included, and if paths to
gisaid sequence and metadata files are in theconfig.yamlfile, then
gisaid data will be parsed and uploaded.All unclassified swine sequences are classified with octoFLU using the
reference file listed inconfig.yamlor if this field is missing, the
default octoFLU reference.The subtype of each strain is determined. GenBank has a subtype annotation
under theserotypefield,octoFLUalso determines the subtype of the HA
and NA, and for gisaid agisaid_subtypefield is extracted.octofludbsynthesizes all this info to create one subtype. If there are conflicts, the
octoFLU subtype is given priority followed by the GenBank subtype.NOTE:due to a performance bug in the python rdflib library, this step can take a
long time to serialize to a turtle file, you'll have have to wait for it.H1 and H3 antigenic motifs are determined from the HA sequences using theflutilemodule.Subcommand:query- submit a SPARQL queryOnce you've uploaded your data, you will want to access it. This is done withoctofludb query. Data is pulled using SPARQL queries. SPARQL is a query
language for accessing data in a triplestore database, just like SQL is a query
language for relational databases. There are no join statements in SPARQL. A
triplestore is, in principle, nothing more than a long list of subject,
predicate, object triples. A SPARQL query defines a pattern over these triples
and returns every set of values for which the pattern is true.Here is a simple example:PREFIXf:<https://flu-crew.org/term/>SELECT?strain?host?genbank_id?country_code?dateWHERE{?sidf:strain_name?strain.?sidf:host?host.?sidf:date?date.?sidf:country?country.?countryf:code?country_code.}LIMIT10The first line defines our unique flu-crew namespace, this ensures there are no
name conflicts if we link this database to other SPARQL databases.TheSELECTline determines which columns are in the returned table.TheWHEREblock is a set of patterns that defines what is returned.All subjects and relations in a triplestore are UUIDs. Objects (such as?strainor?country) may be values or UUIDs. So?sidis some UUID such
that it has one or moref:strain_nameedges, one or moref:hostedges, one
or moref:dateedges, and one or moref:codeedges. Countries are stored
both by name and by 3-letter ISO code.?countryis a UUID that links to the
literal code and name strings. The final two lines of theWHEREclause could
alternatively be written as?sid f:country/f:code ?country_code.The final lineLIMIT 10limits the return data to just 10 entries. I often
place limits on queries while I am building them.While tables are often needed, in most of our pipelines we would rather have
FASTA files with annotated headers. This can be accomplished by adding the--fastaflag to theoctofludb querycommand and placing sequence data as
the final entry in the SELECT statement. For example:PREFIXf:<https://flu-crew.org/term/>SELECT?strain?host?genbank_id?country_code?date?seqWHERE{?sidf:strain_name?strain.?sidf:host?host.?sidf:date?date.?sidf:country?country.?countryf:code?country_code.?sidf:has_segment?gid.?gidf:segment_name"HA".?gidf:dnaseq?seq.}LIMIT10I usually pass sequences throughsmofto clean them up:$ octofludb query --fasta myquery.rq | smof clean -t n -drux > myseqs.fnaFor an example of complete with filters, aggregate, and optional data, see the*.rqquery files in theoctofludb/datafolder of theoctofludbgit repo.To explore the relations in the database, you can run queries such as the following:PREFIXf:<https://flu-crew.org/term/>SELECT?gid?p?oWHERE{?sidf:has_segment?gid.?sidf:host"swine".?gidf:segment_name"HA".?gidf:clade?clade.?gid?p?o.}LIMIT100This gives you a list of links from a swine segment to other data.Subcommand:update- submit a SPARQL deletion statementLikeoctofludb query,octofludb updatealso takes a SPARQL file. They are
called in different ways sincequeryis a readonly call andupdatecan
delete values from the database. I have only usedupdatefor deletion, though
it can also insert data.Use this command with care -- deletion can't be undone. When possible, use theoctofludb deletesubcommand to delete specific selections of data (such as a
clades or all motifs).As an example, say you want to reclassify the all strains in the 1C lineage.
You can do this by updating the octoFLU reference file and running:octofludb pull --no-subtype --no-motifs --no-schema --no-motifs --nmonths=0This will just run the classification step.However, it changes nothing, becauseoctofludbfinds that all the 1C's are
already annotated tries to save you time by not passing them to octoFLU.You could create your own table of data linking genbank ids to clades withoctofludb prep table. But then you would end up withtwoclades for each 1C
strain. The triplestore doesn'tknowthat this relationship should be unique
(there is a way toto give it this information, but let's save that for later).Instead, we need to specifically delete the 1C clades triples. This can be done
withoctofludb updateand the following SPARQL file:DELETE{?sidf:gl_clade?clade.}WHERE{?sidf:gl_clade?clade.FILTER(REGEX(?clade,"1C")).}You can check that octofludb did indeed delete the 1C clades with the query:SELECT?sid?clade.}WHERE{?sidf:gl_clade?clade.FILTER(REGEX(?clade,"1C")).}Which should return nothing.Subcommand:delete- deleteoctofludb deletecan be used as an alternative to the lower-leveloctofludb updateSPARQL calls described above. It deletes specific kinds of data, such
as all constellations, all subtypes, all us-clades, and global clades, or all
motifs.Subcommand:construct- submit a SPARQL construction statementHonestly, I never use this subcommand and am not sure if it works. Maybe I
should remove it?Subcommand:prep- munge data into Turtle filesIf you want to work with data beyond that whichoctofludb pullloads, then
you will needprep. This subcommand parses data out of various formats and
writes it into Turtle files that can be uploaded to the database.The different file formatsprepsupports are available as a list of
sub-subcommands. The three most important of these arefasta,table, andunpublished. The other sub-subcommands are slated for deprecation.Internally,octofludbtries to fit a FASTA file to a table where columns are
derived from the headers and the last column is the sequences. The default
delimiter for a FASTA header is '|'. So the following FASTA file:>A/swine/R123456/Iowa/2020|H3N2|2020-06-21
GATACAGATACAGATACAGATACAGATACAGATACA
>A/horse/T4343/Canada/2020|H1N2|2020-07-04
CATCATCATCATCATCATCATCATCATCATCATCATWould be viewed internally as the table:strain_name               subtype  date       dnaseq
A/swine/R123456/Iowa/2020 H3N2     2020-06-21 GATACA...
A/horse/T4343/Canada/2020 H1N2     2020-07-04 CATCAT...octofludbwill attempt to guess the type of each column. The each element in
the first field match the parser for thestrain_nametype, the second field
matches the parser for subtype, the third for date, the fourth for dna
sequence. But hereoctofludbhits a snag. How should the DNA sequence be
connected to the strain name? Strains don't have DNA sequences. Only the
segments do. This query shows the problem:PREFIXf:<https://flu-crew.org/term/>SELECT?strain?seqWHERE{?sidf:strain_name?strain.?sidf:has_segment?gid.?gidf:dnaseq?seq.}A strain inoctofludbis linked to many segments UUIDs. Each segment UUID is
linked to a DNA sequence. The UUIDs for segments are usually derived from the
genbank or epiflu ids. For example,MW872131is translated to the UUID<https://flu-crew.org/id/mw872131>. Because there is a deterministic way of
mapping a GenBank id to a UUID, any new data that comes in that is associated
with a known GenBank ID can be automatically linked. But in the above FASTA, no
ID is specified for the segment.octofludbcircumvents this issue by using the md5 hash of the DNA sequence as
the base of the UUID. This will merge together segments that have duplicate
sequences. Sooctofludb prep fastawill produce the turtle file:@prefixf:<https://flu-crew.org/term/>.@prefixfid:<https://flu-crew.org/id/>.@prefixusa:<https://flu-crew.org/geo/country/usa/>.@prefixxsd:<http://www.w3.org/2001/XMLSchema#>.<https://flu-crew.org/id/a%2Fhorse%2Ft4343%2Fcanada%2F2020>f:date"2020-07-04"^^xsd:date;f:has_segmentfid:e7e8df44044f1ee2b4fd9e60b33efe9a;f:strain_name"A/horse/T4343/Canada/2020";f:subtype"H1N2".<https://flu-crew.org/id/a%2Fswine%2Fr123456%2Fiowa%2F2020>f:date"2020-06-21"^^xsd:date;f:has_segmentfid:1ca74a6402f97cf38907d194bf6b837a;f:stateusa:IA;f:strain_name"A/swine/R123456/Iowa/2020";f:subtype"H3N2".fid:1ca74a6402f97cf38907d194bf6b837af:chksum"1ca74a6402f97cf38907d194bf6b837a";f:dnaseq"GATACAGATACAGATACAGATACAGATACAGATACA".fid:e7e8df44044f1ee2b4fd9e60b33efe9af:chksum"e7e8df44044f1ee2b4fd9e60b33efe9a";f:dnaseq"CATCATCATCATCATCATCATCATCATCATCATCAT".Whenever you useoctofludb prep, you should always double check the resulting
Turtle files before uploading them to the database to ensure the right
information is being pushed.Subcommand:upload- upload one or more Turtle filesNot much to say here, theuploadcommand uploads Turtle files to the
database. Any triples that are already present in the database are ignored.Subcommand:classify- classify strains with octoFLUoctofludb classifytakes a fasta file as an argument and produces a table
showing the clades. It is a wrapper around octoFLU that uses the locally
specified reference file. The path to this reference file may be specified in
the~/.octofludb/config.yamlfile. If no reference file is specified, the
default octoFLU reference is used.Subcommand:report- make specialized reportsAny report generating logic that is important enough to crystallize into
octofludb itself, that is comprised of both a query and filtering/cleaning in
Python, should be added here.There are currently two useful reports.The first isoctofludb report masterlist. This produces the file that is the input to octoflushow and the quarterly reports.The second isoctofludb report monthly. This produces the input to the monthly WGS selection pipeline.There are currently two other commands:offluandquarter.offluis a
stub that will eventually produce the public and gisaid inputs to the VCM-offlu
pipeline.quarteris currently just a wrapper aroundmasterlist, since both
octoflushow and the quarterly report use the same input.Subcommand:fetch- tag and fetch sets of identifiersoctofludb fetchcontains a selection of tools for working with specific sets of identifiers.For exampleoctofludb fetch tag id-list.txtWill read all the ids in id-list and associate them all with a unique tag. The
ids may be strain names, genbank ids, epiflu ids, epiflu isolate ids, or
barcodes.octofludbunderstands all these ids and will figure out the right
thing for you.Once your ids are tagged, you can retrieve them with the sub-subcommands:isolate,strain,segment, orsequence. Each of these commands pulls the
tagged data at a different level.The tags may also be used in SPARQL queries.Here is an longer example. Suppose you have a mysterious list of 10 genbank ids in a file namedmystery.txt:EPI653195
CY246223
JX983103
MF801489
JN652492
CY099224
MF145504
HQ896660
KM503168
JF316643First you want to clear any previous tags:octofludb fetch clearThen tag your mystery ids:octofludb fetch tag `mystery.txt`Now you view default sections of your data. Try all the following:octofludb fetch segment
octofludb fetch strain 
octofludb fetch isolate
octofludb fetch sequenceYou may also specify the tagged terms in a SPARQL query:PREFIXf:<https://flu-crew.org/term/>SELECT?strain?host?genbank_id?country_code?date?seqWHERE{?sidf:strain_name?strain.?sidf:host?host.?sidf:date?date.?sidf:country?country.?countryf:code?country_code.?sidf:has_segment?gid.?gidf:segment_name"HA".?gidf:dnaseq?seq.# limit the query to tagged segments?tag_idf:query_tag?tag.?gidf:genbank_id?tag.}Problem strain examplesHere is a (very incomplete) list of strange strain names I have to deal with:A/USA/LAN_(P10)_NA/2018- parenthesesA/R(duck/Hokkaido/9/99-tern/South Africa/1961)- worse parenthesesA/swine/Denmark/18-13002-73_PB1/2018- includes segment name (there are lots of these)A/swine/Cotes d'Armor/110466/2010- single quotes