master, metalogger: Fix how `sessions' downloading is handled

Downloading sessions file can happen at any time (asynchronously)
which means it can also happen virtually at the same time
as potential error in applying changelog.

Up till now we assumed that at end of sessions download we are
in 'download state' which was not quite true if changelog apply
error occurred.

Additionally we shall prevent sessions file to be downloaded
at all before we safely enter `synchronized state'.

Change-Id: I55d76735f81d84d60689e64447fc99195e9a2d43
common: Make serialization framework support empty packets

Change-Id: Ie591f1fe3c2a1cf79de1fe99f31da5edd1c557d4
assist troubleshooting asciidoc related FTBFS

 Increase verbosity so conversion errors would appear in build log.

 "--keep-artefacts" leaves intermedient .xml files behind (may be handy in
 case of validation problems).
build: preserve environment build FLAGS

 to avoid dropping hardening build flags when built on Debian.
data: Add useful sample of removal syntax in mfshdd.cfg
mount: Do minor spelling corrections
doc: Fix sample "mfstopology.cfg" file

Apparently master doesn't like empty lines in "mfstopology.cfg" -- for each
empty line it logs

    mfstopology: incomplete definition in line: N

This commit replaces empty lines with comments in sample conf file
to reduce noise.

Closes: #199
doc: Add "mfsmetadump.8" and "mfsrestoremaster.8" man pages
ha-cluster: Add lizardfs-cluster-manager script

This commit adds a script for managing clusters of LizardFS metadata
servers. There are only stubs of all the commands which will be provided
by this script.

Change-Id: Ie0e1e8a800b674d9f3f2da3337e2fccafa17f943
ha-cluster: Update help in lizardfs-cluster-manager

Change-Id: Ie02d8b7c1b55ed5d74e8c3c66494ad3a1e602553
metarestore: Fix uncaught exceptions when applying changelogs

When metarestore is applying changelog files, uncaught exception can
occur when getting version of the last change. This would cause metarestore
to terminate in some cases eg. when one of the changelog files is malformed.

This commit fixes the bug (introduced in e701e69) by catching the
exception and setting the version of the last change to 0.

Change-Id: I28fe9f5d87d42ba34b4b94e61e15d87b4f19a730
master: Fix flushing changes to shadow masters on exit

There was a possibility that when stopping the master server not all
changes were successfully sent to connected metaloggers and shadow
masters. Consider the following scenario:

* master server creates a lot of changelogs and receives SIGTERM
* before exiting, the master server writes all the changes to sockets
* master server closes all sockets and terminates
* operating system flushes data from sockets' buffers in background

Now if a shadow master sends any message to the master server, the
operating system on the master server's side would drop all
unacknowledged data from socket buffers and reply with RST. This is how
TCP handles a situation when new data arrives to a socket, which is no
longer opened by any process, but exists because it waits for all the
data to be transmitted and acked. Shadow masters send many messages to
the master server which might result in a connection reset and losing
data, e.g. ANTOAN_NOP. This can be observed in
test_flushing_changes_to_shadow, which often fails.

This commit fixes the issue, by changing the exit phase from:

* write all pending messages to ma<->ml sockets
* close all ma<->ml sockets
* terminate

to:

* write all pending messages to ma<->ml sockets
* write LIZ_MATOML_END_SESSION message to each ma<->ml socket; metaloggers
  and shadow masters close connections when this message is received
* wait for sockets to be closed by shadow masters and metaloggers
* terminate

This makes sure that all changes are received by all the connected shadow
masters. The LIZ_MATOML_END_SESSION is handled by shadow masters and
metaloggers in the following way:

* old versions close the socket because of an "unknown message" error
* newer versions understand this message and close the socket in the
  packet's handler.

LIZ_MATOML_END_SESSION is always the last message sent by the master
server. No other messages would be created after sending this one.

Change-Id: Ie93fb9362f80e2c93155a5358aab3f24651abd10
master,probe: Add support for listing shadow masters

Change-Id: Ifbeca560b6fb5a279d3ffd30de6eaa9a675e8d04
tests: Fix failing test_shadow_reloading_metadata

After a change introduced in b75dac65879304923 this test fails because
of a timeout: master server refuses to terminate quickly if any
connected shadow master is not responding. This commit fixes the issue
by changing a master<->shadow timeout from the default value (60
seconds) to a minimum allowed value (10 seconds). After this change
stopping the master server takes 10 seconds and the test succeeds.

Change-Id: I32c63ab0dcf8f6a2ba69cfa41ab5651fd805fb82
tests: Add a test of lizardfs-probe list-metadataservers

Change-Id: Id1debef527288292a05e9a97efee756c48db351b
ha-cluster: Add OCF resource agent for LizardFS

This resource agent will manage a LizardFS master and shadow master
instances to provide high-availability failover.  It does so by
implementing an OCF resource agent which is designed to be used as a
master/slave resource.

An OCF master/slave resource initially starts in the slave state, which
in our case means a shadow master, and is later promoted to a master
role.

When initially started as a shadow master, we don't know the address of
the actual master; in fact, we may become the actual master, or the
master may be communicated to us in a later "notify" event.  So we start
the shadow master with an invalid master, and change and reload the
configuration once we find out who the actual master is.

When the resource agent is polled for its status, it updates its "master
score" to indicate which node should be preferred as the next master.
The current master gets the highest score, connected shadow masters get
the next highest, and disconnected shadow masters get extremely low but
non-zero scores.

The resource agent takes a few configuration parameters; none are
required, as they all have reasonable default values.  Most of them are
merged into the normal lizardfs config files, which are copied and
edited in order to change the shadow/master configuration options as the
states change.  The particular collection of parameters that are
currently exposed via the resource agent are determined more by what was
convenient for my testing than any principled criteria; they could be
removed and simply configured via the static configuration file, or
increased such that all static configuration options are exposed via the
resource agent configuration.

There is a test script, which is not yet integrated with the main
lizardfs test system.  It runs a few basic sanity checks, then runs
ocf-tester to try running through all of the commands and ensuring that
it meets the OCF specifications.  ocf-tester doesn't set up multiple
instances, so it can't fully test master/slave resources, so we also
include a test by hand to spin up two instances, promote one, and notify
the other.  Note that these tests never actually exercise any changes to
the metadata, so they're still not in any way complete, but they provide
a basic smoketest to ensure that everything at least claims to work.
Also note that the test script does several things that conflict with an
actual instance, like updating the global master score, so shouldn't be
run on a system with lizardfs or an actual cluster running.

For documentation on how to set up a basic master/slave failover with a
floating IP address that follows the current master, see
src/resourceagent/README.

For information on what an OCF resource agent is and how they should be
written, see:

* http://www.linux-ha.org/doc/dev-guides/ra-dev-guide.html
  The OCF Resource Agent Developer's Guide
* http://www.linux-ha.org/wiki/OCF_Resource_Agents
  The Linux HA Wiki page on OCF Resource Agents
* https://github.com/ClusterLabs/resource-agents/tree/master/heartbeat
  A collection of existing resource agents for various software; in
  particular, the mysql and pgsql resource agents were used for
  reference.
* http://git.drbd.org/gitweb.cgi?p=drbd-utils.git;a=blob;f=scripts/drbd.ocf;hb=HEAD
  Another good master/slave resource agent which was used for
  reference, distributed in a separate package than the common
  resource-agents package.

Change-Id: Ia684d5ae6b2402bec566a5bcecbe80e37973d2e4
ha-cluster: Add create-new-cluster command

This commit adds a new command to lizardfs-cluster-manager:
create-new-cluster. It can be used to configure a new cluster (configure
LizardFS, corosync, pacemaker, etc)

Change-Id: I9033fb22c9834ba82a65239ebb9a32e35ccf5e90
doc: Fix inconsistencies in ha-cluster README

Change-Id: I000b4d2767291dcbe6cfe7ed7598d3d1d8d3d3db
ha-cluster: Remove resource agent's unnecessary config variables

This commit simplifies the OCF resource agent by removing unnecessary
variables leaving only three: master_cfg, master_ip and run_dir. The
test_mfsmaster_agent.sh script is also modified to reflect these
changes. Because the test which tests multiple instances provided by
this script no longer works (but this has been caused by one of
previous changes), it is commented out.

Change-Id: I377ba145a2682daffdff78be2b37017890460fc9
admin: Rename lizardfs-probe to lizardfs-admin

Rename lizardfs-probe to lizardfs-admin, preserving backward
compatibility. The tool will soon be used for manipulating file system
configuration and that's the reason of the name change.

Change-Id: Iadb1772dac21d84e908e8cf55b198d7a9f1f6368