deis/deis — pull request #1754 by carmstrong (2014-09-03 08:42:57+00:00)
Title: feat(store): add deis-store component

This is the deis-store component described in https://github.com/deis/deis/issues/1384.
It uses Ceph as a backing service, which gives us the flexibility to
use mounted devices or (eventually) a blob store gateway.

Note that to test this, you will need Docker 1.2.0. You can use CoreOS 423.0.0 from #1731.

TESTING: build, install, and start the stores
```console
$ make -C store build
$ make install-stores start-stores
```

If you make changes and want to restart stores:
```console
$ make -C store stop build
$ make start-stores
```

Once the services are running, you can enter one of the services and `ceph -s` to see the cluster health.

**To-Dos**
- [ ] Figure out why my `pg_num` and `pgp_num` settings are being ignored - it looks like the default of 64 is used instead of 128. This isn't catastrophic, but 128 is optimal. See http://ceph.com/docs/master/rados/operations/placement-groups/ for details on placement groups and choosing an optimal value.
- [ ] Test failure and disaster scenarios (stop a container and then start it later, etc.)

**Notes**
- The deis-store component relies on the hostnames of the machines to communicate. Since we are using `--net host` here, the container inherits the hostname of the CoreOS host. **We need to ensure all provision scripts use unique hostnames!** Off the top of my head, I believe either Rackspace or Digital Ocean creates all hosts as deis-1. This will cause problems.

replaces #1729