Floby/node-tokenizer — pull request #9 by joshjung (2014-11-25 17:23:31+00:00)
Title: Several bug fixes + performance improvements

Fix for #8.

Adding unit test for special scenario that was failing for me. Also adding options.split to improve performance on predictable streams of data with boundaries across which tokens will not ever pass (e.g. CSV file)