deis/deis — pull request #1729 by carmstrong (2014-08-28 21:35:23+00:00)
Title: WIP: deis-store component (Ceph-based backing store)

This is the deis-store component described in #1384. It uses [Ceph](http://ceph.com) as a backing service, which gives us the flexibility to use mounted devices or a blob store gateway.

This PR is provided as a work-in-progress, enabling users to follow its progress. Currently, development is blocked until a new CoreOS alpha is released which provides:
- Docker 1.2.0 - necessary so we can edit `/etc/hosts` within containers (necessary because Ceph uses hostnames, not IPs) - https://github.com/coreos/coreos-overlay/pull/814
- Kernel RDB module - necessary so we can mount Ceph-backed devices -https://github.com/coreos/coreos-overlay/pull/823

Both of these have been merged, but haven't made it to an alpha release yet. The actual provisioning of the Ceph services is blocked until Docker 1.2.0. Using the volume is blocked until the kernel RDB module is built.

**Current features**
- Automatically spins up `DEIS_NUM_INSTANCES` store components, so there's a store on every host
- Hard-scheduling of data containers to specific machines (in-order, so `deis-store-1` is on `deis-1` for convenience) -- note that the static scheduling logic is very similar to other data containers, but has some important nuanced differences. @gabrtv, this is important for when we move this into `deisctl`.
- Sets a hostname for store containers (matches the unit name, like `deis-store-1`)
- SSHd so the ceph user on other store machines can login
- confd-managed authorized_keys and ssh_config to enable passwordless, seamless SSH on a non-standard port
- Locking logic to determine if the Ceph cluster has already been installed, and if so, if a machine is permitted to continue setting up the Cluster (do they have the lock?)

**Remaining work**
- [ ] Use confd to configure `/etc/hosts`
- [ ] `ceph-deploy new` the cluster
- [ ] Edit `ceph.conf` to customize settings
- [ ] `ceph-install` the cluster
- [ ] Install monitoring agents
- [ ] Install OSD agents
- [ ] Create volume spanning all hosts
- [ ] Test recovery and writing to the volume with only one host

After discussing with @gabrtv, we're going to hold off on installing the object gateway until later. Seeing how we do with the volume mount is good enough for now, and the gateway is independent of these services anyway (since it runs Apache and just talks to the cluster from another machine).

Note that the Ceph deployment steps follow my [Ceph multinode Vagrant](https://github.com/carmstrong/multinode-ceph-vagrant) workshop for VM-based Ceph provisioning.

TESTING: build, install, and start the stores
```console
$ make -C store build
$ make install-stores start-stores
```

If you make changes and want to restart stores:
```console
$ make -C store stop build
$ make start-stores
```