Include VERSION in package as a resource

With this, wal-e can (perhaps?) finally be packaged as a cheeseshop
package, and can learn the long-missing 'version' action.

Signed-off-by: Daniel Farina <daniel@heroku.com>
Stamp v0.6.1

Signed-off-by: Daniel Farina <daniel@heroku.com>
setup.py: downcase WAL-E to wal-e

Continue the general conversion of machine-intended references to
WAL-E to be lower case, much like the executable, package name,
repository, and so on.

It is still handy to have capitalized name (and WAL is an acronym
anyway) because it makes references to WAL-E stand out in human-read
text without quoting or special formatting.

Signed-off-by: Daniel Farina <daniel@heroku.com>
setup.py: prepare metadata for cheeseshop upload

Also refactor the way short files are read into cheeseshop metadata.

Signed-off-by: Daniel Farina <daniel@heroku.com>
Add build and packaging tools for PyPI

Editorialized (mostly, just squashed) by Daniel Farina.

Signed-off-by: Daniel Farina <daniel@heroku.com>
Add contributor: Kirill Klenov

Signed-off-by: Daniel Farina <daniel@heroku.com>
Fix license placeholder expansion

Signed-off-by: Daniel Farina <daniel@heroku.com>
Stamp v0.6.2

Signed-off-by: Daniel Farina <daniel@fdr.io>
Allow use of S3 buckets in regions other than us-east-1.

Work around region issue in S3Backup.new_connection and
boto.storage_uri.  When attempting to connect to a bucket in a
non-standard (non us-east-1) region, boto complains

  S3ResponseError: 301 Moved Permanently
  The bucket you are attempting to access must be addressed using the
  specified endpoint. Please send all future requests to this endpoint.

Naming collision fix contributed by Toby Collier (@tobyw4n)

Add a unit test and integration test.  This adds a testing dependency on
unittest2 which is a Python 2.6 backport of the unittest module
improvements in Python 2.7.

Having AWS credentials in your environment along with the
WALE_S3_INTEGRATION_TESTS (any arbitrary value) will trigger inclusion
of the integration tests.

References:

  https://github.com/boto/boto/issues/443
  https://github.com/heroku/WAL-E/issues/24

Signed-off-by: Bo Shi <bs@alum.mit.edu>
Signed-off-by: Daniel Farina <drfarina@acm.org>
tests: excise unittest2 dependency

Given that pytest is already pretty much here to stay without
significant work, write these tests to use pytest's functionality to
be able to avoid unittest/unittest2.

Unfortunately this seems to require a change of behavior in how the
cause of the skip will be reported.

Signed-off-by: Daniel Farina <daniel@heroku.com>

Conflicts:
	tests/test_s3_worker.py
Fix upper case bucket names regression (from #3).

Evidently, boto.storage_uri called without extra connection arguments
constructed a connection object compatible with uppercase bucket names
(which I discovered today is only allowed in us-east-1; one gets an
error attempting to create a bucket with an uppercase name in all other
regions).  Issue #3 added a 'host' connection argument that appears to
have changed the configuration.  Explicitly request the appropriate
calling format.

Signed-off-by: Daniel Farina <daniel@heroku.com>
Retry subprocess spawning if ENOMEM

Pathologically, on systems with large numbers of database files the
volume/partition-planning step in creating a base backup can occupy
quite a bit of memory, on the order of a few hundred megabytes.  This
is because the entire tree of files is copied into program-local
memory up-front.

This memory footprint is not fantastic and methods can be used to
reduce it, but it is badly exacerbated on systems with overcommit off:
that's because fork() will momentarily double this oversized commit
charge before running exec() to execute small programs like 'psql'.
If there is momentarily not enough memory to execute fork(), prior to
this patch wal-e would crash and all the progress made in taking the
backup will be lost.

Included in this patch are a few extra hooks to control the retry
behavior that are only used in testing.  This seems a little
overwrought, but since OOM is notoriously difficult to test for, a lot
of ceremony is expended in trying to ensure the test suite is testing
properly in modeling it via monkey patching.
Revert version of boto required for v0.6 series

This increment took place to handle the multi-delete procedures seen
in v0.7.  Boto 2.0.0 should work fine.
s3_worker: don't capture all exceptions

Problematic exceptions that can be caught by this except: include
KeyboardInterrupt.  It seems like this wouldn't matter much, but in
general 'except:' without re-raising is probably not good form.

Signed-off-by: Daniel Farina <daniel@heroku.com>
Skip 'wal-e version' test if wal-e not in PATH

A better fix would be to exercise the code without wal-e being in
PATH, but skipping is better than noise.
Tighten up exception handling with non-preferred bucket names

Previously, almost any exception raised could raise a misleading
WARNING/HINT message that also contained a voluminous stack trace.  It
confused me into thinking this was a crash at first (for a short
base-backup upload).

This version checks the bucket name prior to submitting it to S3, and
does not handle cases as exhaustively, but does handle the one
complaint that has occurred in practice, which is upper case bucket
names.  In this precise case the warning is displayed and is
definitely topical, and the stack trace hidden.

Conflicts:
	tests/test_s3_worker.py
Copy in and use a recent version of subprocess module

The subprocess source advertises to working all the way back to Python
2.2, so it should be a strict superset of WAL-E supported Python
versions (2.6+).

Although bugs in subprocess have been worked around before in WAL-E,
the final straw leading to this nuclear option is is issue #16327[0],
which causes file descriptor leaks within subprocess in event of fork
failure, which after the fork()-OOM-retry patch become much more
severe.

In that latter case, a vicious cycle forms wherein failed fork()s leak
resources (leaked file handles), causing the parent process to grow,
which increases the memory spike required for fork-exec, making future
forks more likely to fail.  WAL-E can die in that case from the file
descriptor leak.

[0]: http://bugs.python.org/issue16327
Enfeeble failing subprocess doctests

These seem to have specific dates and require specific error messages
from basic POSIX utilities.  This patch is split out to explicitly
document the hacks applied to subprocess.
Fix set notation use in subprocess.py

It doesn't seem like this should be committed to Python.org in a file
that claims to be 2.2 compatible.  Probably worth a bug report.
Move file system traversal code into the tar partitioner

Besides making more sense than being coupled to an s3 operator, the
impetus is to help find out where memory is being expended during this
phase of wal-e's operation on systems with many files.  On some
systems it can easily be over a gigabyte, and with overcommit off this
can make it extraordinarily difficult to take backups (because of
fork-exec, wherein the fork must commit all that memory).

On the same system, though, a byte-count of the output of 'find' is
only about 50-100 meg, so there's clearly a times-ten blowup
somewhere, and the guess is it has to do with the eagerly tarfile
data-structures.  This breaking out of operators will hopefully help
with finding out.