qiime/qiime — pull request #1370 by wasade (2014-01-31 17:10:09+00:00)
Title: Revamped split libraries

THIS IS NOT READY FOR MERGE

This pull request contains the beginnings for restructuring the split libraries workflow. This code depends on #1278, and does not (yet) include the corresponding pyqi ``Command``. As this currently stands, it should effectively be able to replace the functionality in ``split_libraries_fastq.py`` for both fasta and fastq data.

The basic structure is to logically group the operations, currently executed in the following order:

* ``wf_init``
* ``wf_quality``
* ``wf_demultiplex``
* ``wf_primer``
* ``wf_sequence``

The ``Workflow`` object is designed such that if any one of those groups fail, the subsequent groups are not executed enabling a short circuit and lazy approach: only actually execute the necessary functions. In addition, requirements can be decorated on to the methods such that a workflow group (or subsequent method that is called) will only be executed if the requirement is met.

There is some unenforced naming conventions in use here (and this is probably not entirely in sync with PEP8 as this code started prior to the push to PEP8). The biggest convention is that a method that is a part of a workflow is prefixed as such. For instance, ``_demultiplex_golay12`` is a method that is called by ``wf_demultiplex``.

There are a few things to note:

* ``rev_comp`` has not been added to the iterators but is next on the list
* ``rev_comp_mapping_barcodes`` will be handled in the ``Command``
* ``rev_comp_barcode`` as this option is expected work both in the situation where the barcode is embedded in the sequence as well as when the barcode is in an external file. This situation is not yet handled but will likely be via an additional method call under ``wk_demultiplex``
* ``barcode_read_fps`` will be handled by the ``Command``
* ``store_qual_scores`` will be handled by the ``Command``
* ``sample_ids`` will be handled by the ``Command``
* ``store_demultiplexed_fastq`` will be handled by the ``Command``
* ``retain_unassigned_reads`` will be handled by the ``Command``
* ``phred_offset`` will be handled by the ``Command`` and the iterators

Finally, there is a reasonable amount of replicated code current in the tests in order to reset state on the ``Workflow`` object. I'd like to put more formal methods in place in the ``Workflow`` object itself but this all gets touchy given that this pull request depends on an existing pull request.

@walterst I'm particularly interested in your comments here. Do you think that this structure and approach is going to be maintainable in the future? And in addition, do you think that this is a good basis on which the remaining functionality in split_libraries.py can be merged in?