From: toms@fcsparc6.ncifcrf.gov (Tom Schneider)
Newsgroups: bionet.molbio.genbank
Subject: Re: Apologies for Disappearing Division
Date: 20 Dec 92 22:25:00 GMT
Organization: Frederick Cancer Research and Development Center

In article <18DEC199212080223@seqvax.caltech.edu>
mathog@seqvax.caltech.edu (David Mathog) writes:

|Not only is the current design of the flat file distribution arbitrary (not
|illogical, just arbitrary), but there seems to be no good reason to 
|maintain it into a future replete with multiple fully sequenced organisms. 

There are at several reasons to maintain an ASCII distribution format
1.  Changes will break software.
2.  The current form is the defacto standard.
3.  ASCII file format transmits easily between computers and over nets.
4.  Incomplete sequences will continue to roll in for a long time.

Other forms internal to computers are too machine specific and don't
transport.  The common language for describing sequences and the genetic
objects on them has to be PORTABLE.  Until there is a better (gosh DARN it)
WRITTEN standard, it will be difficult to get away from this one.

|Many of you may have noticed that as the entries get bigger and bigger the
|existing tools become less and less useful.

Delila works just fine on 6000 entries, I haven't tried more than that yet.  It
becomes more useful with time.  That's because Delila was designed with the
idea of manipulating more than one sequence fragment at a time.  It would seem
that many programs written still only handle one sequence at a time and force
the user to deal with each sequence interactively.  When you get above a few
sequences this is tedious, and leads to mistakes.

| For instance, what does one do
|with a BLAST hit on the Yeast Chromosome III entry? There are lots of ways
|to get the entry from one of the Genbank servers (NCBI, IUBIO...) to your
|directory, but all of the ones that I'm familiar with insist on sending the
|whole thing, even though you generally only want a small piece of it.
|Obviously, we're going to have to find a better way long before we start
|getting BLAST hits on the "human X chromosome, complete sequence" entry! 

Well I'm surprised that BLAST (which I've not used) doesn't give the
coordinates of the hit!  If it did then there wouldn't be a problem.

Rather than sending the sequence, the search program could send the
instructions for obtaining the sequence!  This is what our "search" program
does (see ncifcrf.gov/pub/delila/search.p.Z), and this was the way Gary Stormo
trained the perceptron to find ribosome binding sites (ie, original use of
neural nets to find binding sites: NAR 10: 2997-3011, 1982).

Advantages:  Instructions are compact.  Instructions can be edited by hand.
Instructions can be manipulated by programs.  Well defined instructions will
last forever (see the example below).  Thus one works on a "meta level",
without touching the sequences directly.  That avoids editing errors.


|As a start, I'd really like to see NCBI RETRIEVE implement the following
|(never mind the syntax). 
|
|  SEND <entry number| [ FROM <START> TO <END> ]

So you will get back an entry WITH ALL THE NUMBERS CHANGED!!  Don't you hate
it when you have 6 different printouts and they all have different numbering
schemes?  Wouldn't it be nice if there was a single numbering scheme
even if you were working with a fragment?  There are two solutions.
The first was implemented in Delila, the second is described in the
Philosophy of Genbank paper.  Basically, one has to invent a coordinate
system and attach it to the extracted data ("book" in Delila terminology).
By the principal that books should have the same format as the original
library, there should also be a coordinate system in the big library.
(It's 1 to n by default now which is a cheap solution.)

Your send command really should have two parts:

1.  Specification - What sequence in the database am I interested in?
2.  Extraction command - How do I want the extracted fragment(s)?

Specification is best not done with "entry numbers" as these are
   hard to remember
   liable to change
It would be better to specify by:
   organism     E. coli;
   chromosome   main;
   strain       K12;
Then one can specify the genetic region a bit closer:
   gene         lacZ;

The extraction then is relative to these data objects:
   get from gene beginning to gene end;
or
   get from gene beginning - 200 to gene beginning + 50;

Obviously, once one has specified the region of interest, one could
extract several pieces.

The results that come back should look just like a GenBank entry,
so that ANY TOOL THAT WORKS ON THE ENTIRE GenBank SET WOULD ALSO
WORK ON A SUBSET!  (Or one could have the results come back as
instructions, as I mentioned above.)
   


|For the sequence this is a trivial operation.

Unfortunately it's not.  Consider that you have circular objects,
and
that you will want to extract fragments whose numbering will DECREASE.

Those problems are solved in the current version of Delila.
But it's worse than that:  suppose that you want a sequence
with certain known mutations:

   gene lacI;
   get from gene beginning -200 to gene beginning with lacIQ;

(I'm pretty sure the promoter Q mutation is within this fragment.)

This requires a more sophisticated coordinate system.



| Chopping out the appropriate
|parts of the reference section *should* be possible too, if maybe a bit 
|more challenging.

If the database were carefully defined and properly stored, pulling out the
references would be a trivial bit of code.  (Every reference would be
guaranteed to point to a range of the associated sequence.  Once instructions
have been reduced from the relative coordinates given above to absolute
coordinates, it is easy for the program to determine the intersecting features
and references.)


|Probably in the next few years we will have to formally set up "central"
|servers for a lot of the searching/retrieving that is currently done
|locally.   While the growth of sequence information is exponential, our
|disk drive budgets are not :-(.  

Right.  So that's why Delila was designed to extract subsets of a database
(called a "library" - which fits NLM quite nicely, doesn't it??).  Delila
is the "librarian".  So one gives "her" instructions, and (naturally)
she returns one a "book", which contains one or more sequences.

An example of instructions for human splice junctions is in
ncifcrf.gov/pub/delila/spliceA.in.Z.  (They are already out of date because
DARN IT GenBank changed the locus names on us, and there are no really solidly
named objects.  Don't blame me, I was squacking about this 10 years ago!!  It
would be nice to rewrite the code to use the ACCESSION number, but that's
slippery also.)


|It would be really useful to have (at least) a SQL like access to these
|database servers. That way we could specify exactly the set to be searched
|or retrieved. I'll plead guilty to having not read the Delila paper, but
|from what I've seen in these news groups it looks like Delila could be
|interpreted into SQL, and that the existing Genbank schema would only need
|to be minimally modified with the addition of a couple of views or extra
|tables. 

I'm guilty of not knowing enough SQL to know if one could build
an advanced Delila on it!

|In addition, lets see if we can agree on a SINGLE syntax for accessing 
|these databases.  Maybe that way we can get by with a single client (per
|platform) to get at Genbank, NBRF, EMBL etc. 

Wrong approach.  First we have to get the databases in the right form
before we can consider the languages.  Then one could have any number
of languages.  The primary things we need are:

1.  Consistent genetic names throughout the database.  Names that are
international standards, that aren't sloppy, slithery and slimy like
the LOCUS and which can be remembered unlike the ACCESSION.
2.  Merging of overlaps from the same strain.  (Come on!  Gime a break.  Who
the heck cares which person sequenced what part of E. coli when you are trying
to solve a BIOLOGICAL problem!!  Keep the underlying history if you insist, but
make the display of the biologically useful merged form easier than seeing the
history.  Correct it when mistakes are found!!!)
3.  Recording of strain information along with organism and chromosome.

Do you see that the consideration of the advanced language (Delila II) tells us
how to design the database?  Do you see that if the database is a disaster area
(ie, un defined and variable at a whim - witness the loss of the organelle
boondoggle!) that any attempt at designing a language will bloody your nose?

So the really fundamental thing we need FIRST is:

         ***************************
         * A Definition of GenBank *
         ***************************

  Tom "Cassandra" Schneider
  National Cancer Institute
  Laboratory of Mathematical Biology
  Frederick, Maryland  21702-1201
  toms@ncifcrf.gov