deis/deis — pull request #1910 by carmstrong (2014-09-25 02:51:19+00:00)
Title: [WIP] feat(store): add deis-store component

This PR implements the deis-store backing service described in #1384. It is a Dockerified implementation of [Ceph](http://ceph.com).

`deis-store` consists of three components:
**deis-store-monitor**
This container configures and starts a [Ceph monitor](https://ceph.com/docs/v0.85/man/8/ceph-mon/), the bookkeepers of the cluster. We block until there are three monitors running before we proceed with allowing the daemons to come up.

**deis-store-daemon**
This container configures and starts a [Ceph object store daemon](https://ceph.com/docs/v0.85/man/8/ceph-osd/). There are three of these containers at a minimum in a cluster, as data stored in Ceph is configured to be duplicated twice. We allow degraded writes even if just one OSD is up.

**deis-store-gateway**
This container configures and starts [radosgw](https://ceph.com/docs/v0.85/radosgw/), an S3- and Swift-compatible object store. This container is accessible by hostname within the cluster at `http://deis-store-gateway:8888`, as well as externally to the cluster at `http://store.domain.tld`. Note that only containers which inherit from `deis-store-base` or are separately modified to have confd manage `/etc/hosts` will be able to resolve this internal hostname.

This PR also adapts the database and registry (WIP) to use `deis-store` for storing WAL logs/backups and Docker image filesystem layers, respectively.

TESTING:
Testing deis-store will involve a custom-built `deisctl` as implemented in deis/deisctl#96. `deisctl install platform` should then provision and start deis-store.

TODOS:
- [ ] `database`: Rewrite logic which checks if the database is already initialized -- we can't test for the existence of /var/lib/postgresql/9.3/main any longer, as we need to run confd before this check happens (mainly so the deis-store-gateway is in /etc/hosts), and confd will template files in this directory. So, we need a better method to determine if the database has already been initialized.
- [ ] `database`: configure periodic backups
- [ ] `database`: Change database prefix when performing restore (see https://github.com/wal-e/wal-e/issues/140#issuecomment-55788137)
- [ ] `registry`: use the S3 API credentials stored in etcd to store filesystem layers in `deis-store`
- [ ] No longer install `deis-database-data` or `deis-registry-data` containers, but use the volume if it already exists -- necessary for the upgrade path to this version (may not be done in this PR)
- [ ] Test failure and disaster scenarios (stop a container and then start it later, etc.)
- [ ] Write integration/component tests
- [ ] Update docs anywhere we list the Deis components
- [ ] Configure automated builds in Docker Hub

closes #1384, closes #1385, closes #1906
replaces #1754, replaces #1803
refs deis/deisctl#96

[skip ci]