aws/aws-cli — pull request #358 by jamesls (2013-09-17 03:44:53+00:00)
Title: Cap the task queue size to a fixed amount

If the worker threads can't keep up with the rate that
tasks are being placed on the queue, the queue can grow to a large
size and consume a lot of memory. Now, the queue is capped to 1000
which means that `put()` calls will block until a worker thread
pulls an item off the task queue.

Confirmed that this kept memory usage to fixed amount. Syncing
250k 512 byte files kept the total memory usage around 100Mb the
entire time.

This isn't really a change in functionality (just memory efficiency) but I
did add a test for the maxsize changes I added in the utils module.

Existing unit/integration tests are passing for me.

This, along with https://github.com/boto/botocore/pull/131 fixes the
memory leaks reported in #326.