biom-format/biom-format — pull request #227 by jrrideout (2013-10-22 16:55:30+00:00)
Title: Add a scipy-based sparse matrix backend

**PLEASE DO NOT MERGE THIS PULL REQUEST.** The code is ready to go, but I'd like to have Jenkins test the changes. There will be test failures because Jenkins/qiime-deploy does not yet have scipy as a dependency; the outstanding qiime-deploy pull requests need to be reviewed/merged first.

Adds a new sparse matrix backend that is based on scipy's sparse matrix library. This adds scipy as an optional biom-format dependency. CSMat remains the default sparse matrix backend.

In general, I've seen minor to huge performance benefits (in both runtime and memory consumption) when testing out this backend with common table operations in QIIME. We gain the most benefits with larger sparse tables (e.g., thousands of samples and/or observations). There are some cases where CSMat outperforms this backend, but this only seems to happen with smaller tables.

I've also verified these changes against QIIME's full suite of unit tests and script usage tests.