Yelp/mrjob — pull request #950 by hendrik-cliqz (2014-10-24 13:43:04+00:00)
Title: improve scanning of long lines by avoiding expensive rescans of the buffer

Hi,

we are dealing with data which sometimes has very long lines of data (JSON).

If the length of line exceeds the chunk size (default 1024) the buffer_iterator_to_line_iterator runs with quadratic complexity because it re-scans the full buffer for every chunk.

This patch avoids this behaviour by remembering which part of the data has already been scanned.

I can now work with data which has >10 MB long lines.