ImportError: No module named Bucket

A --dry-run on delete fails with
        DETAIL: Traceback (most recent call last):
          File "/usr/local/lib/python2.7/dist-packages/wal_e-0.7.dev-py2.7.egg/wal_e/cmd.py", line 365, in main
            import boto.s3.Bucket
        ImportError: No module named Bucket

This fixes the import
Add Christian Pedersen to CONTRIBUTORS

Signed-off-by: Daniel Farina <daniel@heroku.com>
Skip 'wal-e version' test if wal-e not in PATH

A better fix would be to exercise the code without wal-e being in
PATH, but skipping is better than noise.
Retry subprocess spawning if ENOMEM

Pathologically, on systems with large numbers of database files the
volume/partition-planning step in creating a base backup can occupy
quite a bit of memory, on the order of a few hundred megabytes.  This
is because the entire tree of files is copied into program-local
memory up-front.

This memory footprint is not fantastic and methods can be used to
reduce it, but it is badly exacerbated on systems with overcommit off:
that's because fork() will momentarily double this oversized commit
charge before running exec() to execute small programs like 'psql'.
If there is momentarily not enough memory to execute fork(), prior to
this patch wal-e would crash and all the progress made in taking the
backup will be lost.

Included in this patch are a few extra hooks to control the retry
behavior that are only used in testing.  This seems a little
overwrought, but since OOM is notoriously difficult to test for, a lot
of ceremony is expended in trying to ensure the test suite is testing
properly in modeling it via monkey patching.
Tighten up exception handling with non-preferred bucket names

Previously, almost any exception raised could raise a misleading
WARNING/HINT message that also contained a voluminous stack trace.  It
confused me into thinking this was a crash at first (for a short
base-backup upload).

This version checks the bucket name prior to submitting it to S3, and
does not handle cases as exhaustively, but does handle the one
complaint that has occurred in practice, which is upper case bucket
names.  In this precise case the warning is displayed and is
definitely topical, and the stack trace hidden.
Copy in and use a recent version of subprocess module

The subprocess source advertises to working all the way back to Python
2.2, so it should be a strict superset of WAL-E supported Python
versions (2.6+).

Although bugs in subprocess have been worked around before in WAL-E,
the final straw leading to this nuclear option is is issue #16327[0],
which causes file descriptor leaks within subprocess in event of fork
failure, which after the fork()-OOM-retry patch become much more
severe.

In that latter case, a vicious cycle forms wherein failed fork()s leak
resources (leaked file handles), causing the parent process to grow,
which increases the memory spike required for fork-exec, making future
forks more likely to fail.  WAL-E can die in that case from the file
descriptor leak.

[0]: http://bugs.python.org/issue16327
Enfeeble failing subprocess doctests

These seem to have specific dates and require specific error messages
from basic POSIX utilities.  This patch is split out to explicitly
document the hacks applied to subprocess.
Fix set notation use in subprocess.py

It doesn't seem like this should be committed to Python.org in a file
that claims to be 2.2 compatible.  Probably worth a bug report.
Move file system traversal code into the tar partitioner

Besides making more sense than being coupled to an s3 operator, the
impetus is to help find out where memory is being expended during this
phase of wal-e's operation on systems with many files.  On some
systems it can easily be over a gigabyte, and with overcommit off this
can make it extraordinarily difficult to take backups (because of
fork-exec, wherein the fork must commit all that memory).

On the same system, though, a byte-count of the output of 'find' is
only about 50-100 meg, so there's clearly a times-ten blowup
somewhere, and the guess is it has to do with the eagerly tarfile
data-structures.  This breaking out of operators will hopefully help
with finding out.
Add __slots__ to StreamPadFileObj

Comment from the source:

Try to save space via __slots__ optimization: many of these can be
created on systems with many small files that are packed into a tar
partition, and memory blows up when instantiating the tarfile instance
full of these.
Incrementalize the generation of tar partitions

Now instead of happening up front, the segmentation occurs lazily.
This is because holding so many TarInfo values can have an enormous
memory footprint.
Wait for completion of an upload before segmenting more partitions

Explained a comment in the source:

	Wait for the pool to contain additional capacity before
	apply_async-ing even more uploads.

	Reasoning: 'parts' is lazy and generating each 'tpart' from it
	is memory and cpu intensive, so avoid doing it all at once
	when the parallelism is
Free TarInfo instances more aggressively
Apply tar member cardinality limit to segments

To avoid excessive memory usage in database clusters that have many,
many very tiny files that can be jammed into one partition.
Refactor partition constants into readable symbols

Now that there are two of them this seems more useful.  The magical
number for PARTITION_MAX_MEMBERS has been altered slightly in its
absolute value (it's smaller) and derivation (to make it round more
neatly with a Postgres page-aligned quantity, as to say more about how
many small-ish files would fit in one such partition).
Fix off-by-one in tar segmentation that results in too many members
Adaptive upload throttling base on tar member cardinality

In order to prevent memory spikes when concurrently handling multiple
tar file uploads (with their affiliated tar-member metadata memory
bloat), limit uploads on two dimensions: number of concurrent uploads,
and number of outstanding members.

Take this opportunity to clean up and tailor-make the concurrency
control for uploads, which is now decoupled from the mechanism of the
uploader (as evidenced by the submitted tests).

Special care is exhibited such that exceptions are propagated to the
job-enqueuing caller so that failed uploads do not get incorrectly
marked as complete.
When GetBucketLocation fails, use S3-classic endpoint

From the comments:

	A 403 can be caused by IAM keys that do not permit
	GetBucketLocation.  To not change behavior for environments
	that do not have GetBucketLocation allowed, fall back to the
	default endpoint, preserving behavior for those using
	S3-Classic.
Remove dead constructor in test
Remove unused import in test