biocore/bipy — pull request #42 by jrrideout (2014-01-03 22:34:53+00:00)
Title: Add initial DistanceMatrix implementation

This is an initial implementation of the ```DistanceMatrix``` core class.

Quick summary:

- lightweight class encapsulating a ```numpy.ndarray``` of distances and a tuple of sample IDs
- has some simple convenience methods for working with a distance matrix (e.g., retrieving a row vector of distances based on sample ID)
- methods to load and save a distance matrix to file (delimited text, matches the current format used in QIIME)
- retrieval of distances in either condensed or redundant forms (useful when passing raw distances to other library code, e.g., stuff in numpy/scipy)

The class natively stores the matrix in redundant format instead of using a condensed vector. I chose to go this route because the conversion to condensed form is pretty cheap (all done within scipy) and most of our other maths/stats code expects a redundant distance matrix. It didn't seem worth it to increase code complexity at this point, though it'd be pretty easy to add native support in the future if the need arises.

Parsing/formatting is slow because of the delimited text format. Another iteration could include support for a more efficient text-based format or (better) an HDF5-based format. For example, a 16000x16000 matrix parses in ~84 seconds and serializes in ~4 minutes on my machine (this will be roughly the size of the EMP distance matrix when we have it). The memory footprint is ~2.2GB. So not awesome, but definitely workable.

Fixes #40.