biocore/biom-format — pull request #553 by mwhall (2014-10-08 16:09:49+00:00)
Title: Fixes for 551 and 552

For issue 551, the `_extract_data_from_tsv()` function was modified to use an iterative readline strategy, rather than use `readlines()`. It also returns sparse data, rather than dense data. This saves a ton of RAM. The function can still handle input as a filestream object, or lists. As a result of this change, a test case in test_table.py had to be modified. It expected dense output, but has been changed to expect a list of `[row, col, val]`.

For issue 552, `json.load(fp)` was taking the entire file into RAM and not letting that memory go. This is an issue with the json package. To work around this, a simpler test was devised. Check for the first non-whitespace character in the file, and if it is `{`, then we can be very reasonably certain it is JSON and not TSV. This prevents both the memory block, and an undesirable spike in RAM usage even if the memory block were fixed.