zxvdr/scribble — pull request #3 by hoettle (2014-06-28 13:28:25+00:00)
Title: Add chunked reading for files

One problem case for scribble is that it reads a file completely into memory before sending it to Scribe and moving to the next one. If the file is very large (e.g. 2GB+) then that can cause memory allocation issues, as well as delays during which a file may be rotated out of existence.

This PR changes that by changing Scribble to read a chunk of data from each file before proceeding to the next, which has shown an improvement in throughput and a reduction in memory usage.

The default chunk size is 10MB per file per pass.