biom-format/biom-format — pull request #63 by wasade (2012-12-03 21:34:06+00:00)
Title: Lightweight formatter

A rewrite of Table.getBiomFormatJsonString. This implementation, when tested on a 10000x10000 sparse table with 10 million nonzero entries reduced the memory required from ~6GB to ~1.5GB, and reduced runtime from ~60 minutes to ~2 minutes. I freely admit that some of the optimizations are not elegant... but they work. The += string ops appear to be more memory efficient than doing a join on a list, possibly due to the memory expense of the list itself. I did not observe a runtime difference when using joins versus += ops.

The direct IO support reduces memory even further on the above table by about 10% overall; 22% if you subtract runtime footprint of the table itself and focus just on memory required for formatting.