Revert "client: handle transient connection errors"

This reverts commit dc0c58547cd4b20a8e2cd21f3c8363f34fd03e75.

There is a simpler solution that retrying random number of times. Do not set
socket non-blocking till connect succeeds.

(This used to be ctdb commit 74acc2c568300ef42740cf11299a1b2507047f60)
client: Set the socket non-blocking only after connect succeeds

If the socket is set non-blocking before connect, then we should catch
EAGAIN errors and retry. Instead of adding a random number of retries,
better to wait for connect to succeed and then set the socket to
non-blocking.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 524ec206e6a5e8b11723f4d8d1251ed5d84063b0)
statd-callout: Make sure statd callout script always runs as root

In RHEL 6+, rpc.statd runs as "rpcuser" instead of root as on RHEL 5. This
prevents CTDB tool commands talking to daemon since "rpcuser" cannot access
CTDB socket.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Pair-Programmed-With: Martin Schwenke <martin@meltin.net>

(This used to be ctdb commit fe8c4880b371492a38554868d4ca10918c54e412)
recoverd/takeover: Use IP->node mapping info from nodes hosting that IP

When collating IP information for IP layout, only trust the nodes that are
hosting an IP, to have correct information about that IP.  Ignore what all the
other nodes think.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 1c7adbccc69ac276d2b957ad16c3802fdb8868ca)
tools/ltdbtool: Fix handling of -e option

Also, include description of -e option in usage.

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 35264e42ade4676468cf7713fa339c784e932953)
tests: Fix typo in variable name

Signed-off-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 043e18a8324ccb2c8ddd7b323ebedb5b0de1298d)
traverse: Add CTDB_CONTROL_TRAVERSE_ALL_EXT to support withemptyrecords

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit e691df43d20871468142c8fb83f7c7303c4ec307)
traverse: Ensure backward compatibility for CTDB_CONTROL_TRAVERSE_ALL

This makes sure that CTDB_CONTROL TRAVERSE_ALL is compatible with older versions
of CTDB (i.e. 1.2.39 and 1.2.40 branches).

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 5808f0778b39b79ab7a5c7f53ad27947131386ec)
build: silence building of porting_test

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit e96acf19b4d1e0f951ab92b88869a01ff06398be)
recoverd: remove bogus comment "qqq" from "add prototype new banning code"

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 9f01b8db72780acf2f88f1392bc0a796dd4c6176)
ctdbd: unimplement the unused SET_DMASTER control

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 2e92deef5221ee651028ef87138b3113f1fece91)
logging: fix comment typo

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-by: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit 4c0cbfbe8b19f2e6fe17093b52c734bec63dd8b7)
Fix a severe recovery bug that can lead to data corruption for SMB clients.

Problem:
Recovery can under certain circumstances lead to old record copies
resurrecting: Recovery selects the newest record copy purely by RSN. At
the end of the recovery, the recovery master is the dmaster for all
records in all (non-persistent) databases. And the other nodes locally
hold the complete copy of the databases. The bug is that the recovery
process does not increment the RSN on the recovery master at the end of
the recovery. Now clients acting directly on the Recovery master will
directly change a record's content on the recmaster without migration
and hence without RSN bump.  So a subsequent recovery can not tell that
the recmaster's copy is newer than the copies on the other nodes, since
their RSN is the same. Hence, if the recmaster is not node 0 (or more
precisely not the active node with the lowest node number), the recovery
will choose copies from nodes with lower number and stick to these.

Here is how to reproduce:

- assume we have a cluster with at least 2 nodes
- ensure that the recmaster is not node 0
  (maybe ensure with "onnode 0 ctdb setrecmasterrole off")
  say recmaster is node 1
- choose a new database name, say "test1.tdb"
  (make sure it is not yet attached as persistent)
- choose a key name, say "key1"
- all clustere nodes should ok and no recovery running
- now do the following on node 1:

1. dbwrap_tool test1.tdb store key1 uint32 1
2. dbwrap_tool test1.tdb fetch key1 uint32
   ==> 1
3. ctdb recover
4. dbwrap_tool test1.tdb store key1 uint32 2
5. dbwrap_tool test1.tdb fetch key1 uint32
   ==> 2
4. ctdb recover
7. dbwrap_tool test1.tdb fetch key1 uint32
   ==> 1
   ==> BUG

This is a very severe bug, since when applied to Samba's locking.tdb
database, it means that for SMB clients on clustered Samba there is
the potential for locking out oneself from previously opened files
or even worse, data corruption:

Case 1: locking out

- client on recmaster opens file
- recovery propagates open file handle (entry in locking.tdb) to
  other nodes
- client closes file
- client opens the same file
- recovery resurrects old copy of open file record in locking.tdb
  from lower node
- client closes file but fails to delete entry in locking.tdb
- client tries to open same file again but fails, since
  the old record locks it out (since the client is still connected)

Case 2: data corruption

- clien1 on recmaster opens file
- recovery propagates open file info to other nodes
- client1 closes the file and disconnects
- client2 opens the same file
- recovery resurrects old copy of locking.tdb record,
  where client2 has no entry, but client1 has.
- but client2 believes it still has a handle
- client3 opens the file and succees without
  conflicting with client2
  (the detached entry for client1 is discarded because
   the server does not exist any more).
=> both client2 and client3 believe they have exclusive
  access to the file and writing creates data corruption

Fix:

When storing a record on the dmaster, bump its RSN.

The ctdb_ltdb_store_server() is the central function for storing
a record to a local tdb from the ctdbd server context.
So this is also the place where the RSN of the record to be stored
should be incremented, when storing on the dmaster.

For the case of the record migration, this is currently done in
ctdb_become_dmaster() in ctdb_call.c, but there are other places
such as in recovery, where we should bump the RSN, but currently
don't do it.

So moving the RSN incrementation into ctdb_ltdb_store_server fixes
the recovery-record-resurrection bug.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-By: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit feb1d40b21a160737aead22e398f3c34ff3be8de)
ctdb_call: don't bump the rsn in ctdb_become_dmaster() any more

This is now done in ctdb_ltdb_store_server(), so this
extra bump can be spared.

Signed-off-by: Michael Adam <obnox@samba.org>
Reviewed-By: Amitay Isaacs <amitay@gmail.com>

(This used to be ctdb commit cad3107b12e8392f786f9a758ee38cf3a3d58538)
tests: Support waiting for "recovered" state in tests

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 2438f3a4944f7adbcae4cc1b9d5452714244afe7)
tests: Add a test for subsequent recoveries corrupting databases

Signed-off-by: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit b1577a11d548479ff1a05702d106af9465921ad4)
tests: add a comment to recovery db corruption test

The comment explains that we use "ctdb stop" and "ctdb continue"
but we should use "ctdb setcrecmasterrole off".

Signed-off-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 06ac62f890299021220214327f1b611c3cf00145)
util: New functions ctdb_set_child_info() and ctdb_is_child_process()

Must be called by all child processes.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 59b019a97aad9a731f9080ea5be14d0dbdfe03d6)
util: ctdb_fork() should call ctdb_set_child_info()

For now we pass NULL as the child name.  Later we'll give ctdb_fork()
and friends an extra argument and pass that through.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit ba8866d40125bab06391a17d48ff06a4a9f9da89)
ctdbd: Add --pidfile option

Default is not to create a pid file.

Signed-off-by: Martin Schwenke <martin@meltin.net>
Pair-programmed-with: Amitay Isaacs <amitay@gmail.com>
Reviewed-by: Michael Adam <obnox@samba.org>

(This used to be ctdb commit 996e74d3db0c50f91b320af8ab7c43ea6b1136af)