sixohsix/twitter — pull request #199 by adonoho (2014-02-07 00:31:26+00:00)
Title: Reduce memory usage by writing directly into byte array buffer.

Gentlefolk,

The whole point of using a `bytearray` as opposed to concatenating reads of `bytes` was to reduce memory usage. This pull request now takes that strategy to its logical conclusion by writing the balance of the chunk directly into the `bytearray`. This saves the creation of a temporary `bytes` array, up to 8KiB in size.

It does this by creating a `memoryview` of the `bytearray`. While this is still an allocation, the `memoryview`s are much smaller and are, presumably, reclaimed faster.

This patch has been running for over 24 hours and has processed over 4 MTw under Python v3.3.3 on OS X 10.8.5. It has also been run for about 10 minutes on Python v2.7.6 on a similar machine. `memoryview` does not appear to have been backported to Python v2.6.*. Hence, this pull request is incompatible with that platform.

Anon,
Andrew

P.S. The changes in this pull request are larger than strictly necessary for adding this functionality. I chose to improve the naming of my variables and move some of them closer to where they are used.