deis/deis — pull request #1803 by carmstrong (2014-09-10 05:47:10+00:00)
Title: [WIP] feat(database): use deis-store and WAL-e to ship WAL logs

Note that this is a work-in-progress and **is not yet ready for review**. I PRed this because @aledbf asked for code samples of how to test deis-store, and I thought it would be useful to see the direction I'm going with the database component.

Note the various `TODO` notes in the code - there are still some logic changes to make, and this branch is unlikely to work as-is. However, it provides an example of how to interact with the store component -- specifically, the `/app/bin/create_bucket` script.

Using this branch will require running Docker 1.2.0 (so CoreOS 423.0.0+) as well as the running `deis-store` components from #1754.

**TODOs**
- [ ] Modify WAL-e to support a custom S3 endpoint and port, as well as the `OrdinaryCallingFormat`, if necessary (we may be able to trick it into detecting this with a strange enough bucket name - see `/app/bin/boot`). See https://github.com/wal-e/wal-e/pull/131 and https://github.com/wal-e/wal-e/issues/135. **This is the main blocker for this PR!**
- [ ] `/app/bin/boot`: Rewrite logic which checks if the database is already initialized -- we can't test for the existence of `/var/lib/postgresql/9.3/main` any longer, as we need to run confd before this check happens (mainly so the deis-store-gateway is in `/etc/hosts`), and confd will template files in this directory. So, we need a better method to determine if the database has already been initialized.
- [ ] `/app/bin/boot`: Figure out how to determine if a backup exists -- `envdir /etc/wal-e.d/env /usr/local/bin/wal-e backup-list` will likely always return true, so we need to check its output intelligently (maybe number of lines of output?)
- [ ] `/app/bin/boot`: Perform an initial backup if none exist
- [ ] No longer install `deis-database-data` container, but use its volume if it already exists -- necessary for the upgrade path to this version (may not be done in this PR)

TESTING: build and restart database
```console
$ make -C database stop build start
```

closes #1385