wal-e/wal-e — pull request #21 by fdr (2013-06-20 01:05:30+00:00)
Title: Adaptive upload throttling base on tar member cardinality

This diff does not align very well.

In order to prevent memory spikes when concurrently handling multiple
tar file uploads (with their affiliated tar-member metadata memory
bloat), limit uploads on two dimensions: number of concurrent uploads,
and number of outstanding members.

Take this opportunity to clean up and tailor-make the concurrency
control for uploads, which is now decoupled from the mechanism of the
uploader (as evidenced by the submitted tests).

Special care is exhibited such that exceptions are propagated to the
job-enqueuing caller so that failed uploads do not get incorrectly
marked as complete.