qiime/qiime — pull request #889 by jrrideout (2013-04-24 19:55:09+00:00)
Title: Preliminary implementation of new DistanceMatrix class.

**This pull request is not ready for merge!** I'd like to get feedback on the current implementation before continuing.

This PR introduces a new module ```qiime.core```, which contains a ```DistanceMatrix``` class. There is also a corresponding ```tests/test_core.py```. The idea here is that ```qiime.core``` will contain other "core" QIIME data structures in the future (see the [stable API milestone](https://github.com/qiime/qiime/issues?milestone=14&state=open)).

I've also updated ```qiime.group```, ```qiime.stats```, and ```qiime.principal_coordinates``` (and their unit tests) in order to illustrate how this new class is used. I will take care of updating the rest of the code once we're happy with the new class.

Here's the basic idea behind ```DistanceMatrix```:

- It looks and acts like a numpy ```ndarray```, but has restrictions for what makes a valid distance matrix. It also has labels (sample IDs) that accompany the raw data.

**Rationale:** we've previously been using the ```(sample_ids, dm_data)``` tuple throughout QIIME, so the new class needs to behave as closely to a numpy array as possible.

- A ```DistanceMatrix``` object *has an* ```ndarray``` (I'm using composition instead of inheritance). It delegates all ```ndarray```-specific operations to its array member.

**Rationale:** I spent considerable time fiddling around with inheriting from ```ndarray```, and even had a couple of different (working) implementations. The class ended up being very messy due to numpy's inconsistent handling of ufuncs, top-level numpy functions, regular ```ndarray``` methods, and python "special" methods (e.g. operators). The ```DistanceMatrix``` implementation was so closely tied to the internals of ```ndarray``` that it would suck to maintain.

We also would have had to introduce the concept of an *invalid* ```DistanceMatrix``` instance (e.g. one that isn't square) in order to perform any shape-changing operations on a distance matrix (for example, taking slices of a distance matrix). This seems messy and overly-complicated to me. The other option would be to always return an ```ndarray``` from any shape-changing ops, but this will yield inconsistent behavior depending on what the shape is, etc.. It is also unclear what should happen to the sample IDs when, for example, adding two distance matrices.

Overall, I think the current delegation approach works out nicely. The ```DistanceMatrix``` interface is a superset of the interface provided by ```ndarray``` (which is given to us for free by using ```__getattr__```), you can never have an invalid ```DistanceMatrix``` instance, and class behavior is consistent (all numpy-specific operations return an ```ndarray```, making this class a drop-in for functions that expect a numpy array). From researching online, a lot of people seem to recommend using composition instead of inheriting from ```ndarray``` because of the tight coupling and messiness that I've become all-too-familiar with.

I'm really interested in getting feedback on the current implementation- your feedback is greatly appreciated!

Related to #829.