biopython/biopython — pull request #356 by eparker05 (2014-08-27 21:08:27+00:00)
Title: Adding lazy loading parsers to SeqIO

#### Problem:
The current suite of sequence parsers require loading each record
fully into memory. SeqIO.index and SeqIO.index_db were developed
to address the problem of large files, but they do not address the
problems presented by large records. Human chromosome 1 is an example
of a large record that can be difficult to access on slower or low-
memory machines. Furthermore, the overhead of parsing large records
is often unnecessary when only a portion of the record is required.

#### Solution:
By using a detailed file index, data can be loaded on demand rather
than on instantiation of the record.

#### Implementation:
Lazy loading parsers have been implemented for Fasta, Genbank, EMBL,
and Uniprot-XML file formats. These parsers can be accessed by using
the `lazy` keyword argument in `SeqIO.index_db`, `SeqIO.parse`, and
`SeqIO.read`. The `lazy` keyword argument defaults to `False` meaning
that lazy loading must be accessed explicitly.

As an alternative to using the boolean `True` to activate lazy loading,
`SeqIO.parse` and `SeqIO.read` can be fed the name of an index file
created with the lazy loading parser. This index file will be used to
speed instantiation of the records. If the index file is blank or
if it doesn't exist, the parser will attempt to write a new index.

>>> chromosome1 = SeqIO.read("chromosome1.gb", "gb", lazy="myindex.db")
>>> chromosome1
GenbankSeqRecProxy(seq=NOT_READ, id='NC_000001.11', name='NC_000001',
description='Homo sapiens chromosome 1, GRCh38 Primary Assembly.',
dbxrefs=['BioProject:PRJNA168 Assembly:GCF_000001405.26'])

#### Performance:
The lazy parsers are almost as fast as the current full-record
parsers when considering the parsing of all record data. The lazy
parsers are much faster when the lazy loading features are used.

Performance comparisons between the new lazy parsers and the current
parsers can be found in the following blog posts.
http://blog.evanaparker.com/2014/08/pre-pull-request-performance-tests-and.html
http://blog.evanaparker.com/2014/08/fasta-performance-comparison.html