lizardfs/lizardfs — pull request #195 by Zorlin (2014-11-29 23:24:09+00:00)
Title: Add lizardfs-cluster init script to manage cluster state for HA

This represents our work (with the help of the LizardFS team and others) on making a proper, official HA solution for LizardFS.

We propose using Corosync combined with this init script to manage the promotion and demotion of masters and shadow masters.

Steps to test it out:
* Install corosync and pacemaker: sudo apt-get install corosync pacemaker
* Generate a key: sudo corosync-keygen
* Copy that key to every host as /etc/corosync/authkey
* Open /etc/corosync/corosync.conf and changed mcastaddr IP to 226.94.1.1 and bind IP to 10.1.21.0 (or whatever subnet you want to run your cluster on).
* Install LizardFS in its various roles - master on at least two hosts and chunk on at least one (chunk is optional for testing).
* Change /etc/default/corosync to "yes" on all hosts
* Start corosync: sudo service corosync start
* Check the state of the cluster: sudo crm_mon -1 - it should show all nodes as part of the cluster.
* Disable STONITH: sudo crm configure property stonith-enabled=false
* Create the primitive for the failover IP (use your own IP if you want a different one): sudo crm configure primitive failover-ip ocf:_heartbeat_:IPaddr2 params ip=10.1.21.200 cidr_netmask=24 op monitor interval=0.15s
** Please note that you should change monitor interval as appropriate. We suggest values of 0.15s for a low latency cluster, 0.3s for a medium latency cluster and 1s for high latency or variable clusters. If you are using this in a production environment, 0.3s is probably the best choice.
* Change resource stickiness to prevent unnecessary movement: sudo crm configure rsc_defaults resource-stickiness=100
* If you have less than 3 hosts, disable quorum handling: sudo crm configure property no-quorum-policy=ignore
* Edit your mfsmaster.cfg so that it explicitly says "PERSONALITY = master" (not commented), then copy it to "mfsmaster.shadow.cfg" and "mfsmaster.master.cfg" in /etc/mfs - editing appropriately (ie, change the shadow.cfg to say shadow for the personality).
* Finally, add a primitive that represents the cluster: sudo crm configure primitive lizardfs-cluster lsb:lizardfs-cluster
* And make the failover-ip follow that: sudo crm configure colocation col_lizardfs-cluster_failover-ip inf: failover-ip lizardfs-cluster

This solution works solidly at this point with any number of nodes. You can also run a logger (recommended) or shadow master on your chunkservers and exclude them from being able to run the master service. Adding chunkservers into the mix is actually recommended as they will help with quorum voting to assist in preventing split-brain events.

There was a previous pull request open for this - this new one was made to clarify and update the init script and remove irrelevant changes.