ctdb-recoverd: Gently abort recovery when election is underway

Sometimes the recovery daemon fails to get the recovery lock on one
node so that node is banned.  This seems to always happen during an
election.  The recovery is triggered because other nodes are found to
have recovery mode enabled.  They have recovery mode enabled because
an election has been forced.

The recovery daemon's main_loop() only does an initial check for an
election.  After that, a node can force an election and, in the
process, set itself to be the current winner.  In this situation,
verify_recmode() will always return MONITOR_RECOVERY_NEEDED so
do_recovery() is called.  If the previous recovery master hasn't
admitted defeat and released the recovery lock, then do_recovery()
will rightly fail.  However, it would be better if it failed a little
more gracefully, since this case is not that unusual.

Instead of trying to take the recovery lock, return early with an
error if there is an election in progress.  Note that the race is
still there but it is now much narrower.

There are probably more subtle ways of avoiding this issue, including
something like this in main_loop():

-	if (pnn != rec->recmaster) {
+	if (pnn != rec->recmaster || rec->election_timeout) {
 		return;
 	}

However, this check is done earlier so it leaves the race window open
a little wider.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Mon Jul 21 06:57:07 CEST 2014 on sn-devel-104
messaging4: Use tdb_append in irpc_add_name

This makes the custom locking code unnecessary here

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Add NULL check to irpc_add_name

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Move str_list_add

If this fails, we'd have to revert the tdb_append. str_list_remove is
simpler :-)

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Remove an unused NTSTATUS var

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Remove unnecessary locking

We don't do any modifying operations on the database, so locking is not
needed here

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Fix a memleak in an error path

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
messaging4: Change irpc_servers_by_name to NTSTATUS

For me, counted arrays are easier to deal with than NULL-terminated
ones. Here we also had a "server_id_is_disconnection" convention, which
was not really obvious.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>

Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Mon Jul 21 20:28:53 CEST 2014 on sn-devel-104
sys_poll_intr: fix timeout arithmetic

Callers of sys_poll_intr() assume timeout to be in milliseconds like
poll(2) expects, but implementation used nanosecond units. Also make
sure timeout doesn't become infinite by mistake during time arithmetic.

Signed-off-by: Daniel Kobras <d.kobras@science-computing.de>
Reviewed-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Stefan Metzmacher <metze@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Jul 22 00:12:24 CEST 2014 on sn-devel-104
ctdb: Avoid a talloc in ctdb_queue_send

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>
ctdb: Fix verbose_memory_names

If we have already partly written a packet, "data" and thus "pkt->data"
does not point to the start of the packet anymore. Assign "hdr" while
it still points at the start of the header.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

Autobuild-User(master): Amitay Isaacs <amitay@samba.org>
Autobuild-Date(master): Tue Jul 22 06:09:50 CEST 2014 on sn-devel-104