garlik/4store — pull request #45 by davechallis (2011-11-08 13:02:53+00:00)
Title: Store discovery and admin with 4s-boss and 4s-admin tools, plus a few minor fixes/features

### Approximate change log ###

* Added 4s-boss daemon, provides store discovery and cluster admin
* Added 4s-admin tool, admin clusters running 4s-boss daemons. Currently supports:
* list-nodes - lists status of 4s-boss daemons on cluster
* list-stores - list status of all stores on cluster
* start-stores - start some/all store backends
* stop-stores - stop some/all store backends
* delete-stores - delete one/some stores across a cluster
* 4store.conf location can now be set at compile time (./configure --with-config-file=) or run time (added -c/--config-file option to most 4s-* commands)
* Added function to get/set config file path (fs_[get|set]_config_file)
* Added man pages for 4s-admin, 4s-boss, 4store.conf
* Added tests for 4s-admin/4s-boss
* Modified existing tests to use explicit config file
* 4s-backend now maintains a locked runtime.info file while running, containing port/pid information
* Missing avahi/dns-sd now results in a message rather than warning at configure time
* Added 'tags' and 'TAGS' to .gitignore
* Several minor typo or formatting (mixed tab/spaces) fixes

### Tested on ###

* Ubuntu 11.x (single machine, 4+1 host cluster)
* RedHat Enterprise 5.x (single machine, 4+1 host cluster)
* Mac OS X Lion (single machine)

Tested with ~1000 kbs without any issues (should in theory be able to cope with 65535 kbs of up to 65535 segments each without a protocol change).

Everything should by default work identically to current version of 4store (i.e. use avahi/dns-sd if available, localhost otherwise). No additional library dependencies as far as I know.

4s-httpd, 4s-query, 4s-import etc. only result in single config file read, and single query to 4s-boss at startup, so query performance should be unmodified.

### Quick setup/testing ###

Quickest way to set up and use 4s-boss discovery:

1. Run 4s-boss on each storage node
2. On master node, add the following to 4store.conf:

[4s-boss]
discovery = sole
nodes = node1;node2;node3;node4

3. On master node, check all is well with:

4s-admin list-nodes
4s-admin list-stores

Some additional info at https://github.com/davechallis/4store/blob/master/src/admin/README.md, and in the man pages (the README.md is a little out of date, need to remove some things like the fallback option).

### Security considerations ###

4s-boss accepts any packets sent to it, so access to 4s-boss port (6733 by default) needs to be limited to trusted hosts. Password auth would be a nice future addition though.

### Outstanding features ###

Current minor todo list is something like:

* Remove fallback option to 4s-boss
* Add create-store functionality
* Create/install a default 4store.conf file
* Add auto start/stop option to 4store.conf, to allow 4s-boss to start/stop backends at startup/shutdown (would make init script creation a lot easier)
* Add mirrored segment info to store information struct
* Add --no-colour option or similar to 4s-admin