let nfs be disabled by NONFS env var

Change-Id: I03e9d53b557d333fb11a6b4a618aba95c1920c55

Conflicts:

	xlators/mgmt/glusterd/src/glusterd-utils.c
cli: let commands specify the exit value in batch mode

Old behavior: when cli is ran in batch mode (sequence of commands are
fed to it in stdin), if a command returns an error (ie. -1), the cli
exits upon it with 255 (-1 on 8 bit).

New behavior: consider any non-zero return from cli commands as error
and use the negative of that return value as exit value, thus giving
control to cli commands over the exit value, while (as of the existing
command set) adhering to the convention of exiting with 1 on error.

Spotted upon stumbling upon mount/umount commands which did want
to exit with 1 on error but that was not possible as of old behavior.

Change-Id: I6f41191cdc718c3e676cfae1e404152f4cb715c5
BUG: 765214
Signed-off-by: Csaba Henk <csaba@redhat.com>
geo-rep / gsyncd: further cleanup refinements

- Regarding issue of leftover ssh control dirs:

  If master side worker is stuck in connection establishment
  phase, have the monitor kill it softly (ie. first by SIGTERM,
  to let it cleanup). This is trickier than sounds on first hearing,
  because if worker is stuck in waiting for a RePCe answer
  (in threading.Condition().wait()), then SIGTERM is ignored
  (more precisely, Python holds it back for the wait and resends it to
  itself when wait is over).

  So instead of signalling the worker only, we send TERM to the
  whole process group -- that brings down the ssh connection, which
  wakes up the waiting worker, which then can cleanup. Only problem
  is that monitor is also in the process group and it should not coomit
  a suicide. That is taken care by setting up a one-time SIGTERM
  handler in the monitor.

- Regarding slave gsyncd stuck in chdir:

  Slave gsyncd is usually well behaved: if master does not send
  keepalives, it takes care to exit. However, if a hang occurs
  in early phase, when slave is to change to the gluster mountpoint,
  no timeout is set up for that (and unlike on master side, neither
  is there an external actor like the monitor to do that).

  So, to manage this scenario, we do the chdir in a (supposedly)
  short lived thread, and in the main thread we wait for the termination
  of this thread. If that does not happen within the time limit, main
  thread calls for cleanup and exit. (This logic explicitely takes the
  appropriate action in the cases when chdir succeeds or when hangs;
  but what about the remaining case, when chdir fails? Well in that case
  the chdir thread's exception handler will put the process to
  cleanup and exit route.)

Change-Id: I6ad6faa9c7b1c37084d171d1e1a756abaff9eba8
BUG: 786291
Signed-off-by: Csaba Henk <csaba@redhat.com>
geo-rep / gsyncd: sanitize error log of external commands

If a command invoked by gsyncd fails, gsyncd makes a log
of what comes out on its stderr. So far the log indeterministically
broke lines at random places. Now put some effort into reconstructing
original lines and having a faithful log.

BUG: 764678
Change-Id: I16fcc75d3e0f624c10c71d9b37c937ca677087cc
Signed-off-by: Csaba Henk <csaba@redhat.com>
geo-rep / gsyncd: rework internal RPC event model

So far the concurrency model adopted by the RePCe RPC backed was,
in theory, M:N -- that is, requests coming from whatever number
of client side contexts were handled by a thread pool of unrelated
size on the slave side.

This had the following issues:

- in practice, M == 1, ie. there was just a single thread that
  synchronously makes RePCe requests (ie. there is no use of a
  multithreaded server)
- due to the Python threading implementation¹, the thread pool
  of the server was an overhead, not a gain
- also, server was not thread-safe, as with heavy load + large
  responses, the response messages could end up interleaved
- We had one tunable for gsyncd concurrency, "sync-jobs"; this
  determined both the size of the rsync pool on master side and
  size of RePCe server thread pool on slave side. Indeed, these
  two resources are not related, so contolling them by the same
  value is an artificial constraint (that came to existence due
  to the lack of a better idea).

Now we move to a 1:1 model, and the actual number of contexts is
2 (can be forced to be 1).

Master creates two gsyncd slaves, a "foreground" and a "background"
one. Basic crawl procedure makes a linear progress that sends
synchronous requests -- these are sent to the foreground slave. However,
there are a lot of requests for which we either don't need to wait at
all (eg. set_xtime) or at least not at the point of making the request
(eg. setattr -- the continuation of the crawl does not depend
on setattr response, it's enough to ensure that a setattr failure
vetoes set_xtime). Now these are sended to the background server,
asynchronously.

- refactored RePCe to have single threaded server and to make
  async messages eaiser
- refactored master to send async requests wherever feasible
- refactored resources (in particular, SSH) to support "clones"
- refactored gsyncd.c wrapper to allow rsync invocations that
  have multiple gsyncd siblings
- new tunable: "max-servers" for controlling number of slave
  processes; if it's 1, "foreground" and "background" slaves
  will be the same. Now "sync-jobs" controls only the size of the
  rsync worker pool, and it defaults to the number of cores
  (cf. sysconf(3)).

These changes are compatible over the wire with release-3.2
gsynd; however earlier 3.3 betas (that feature the gsyncd.c
wrapper) will support only max-slaves=1 mode of operation.

¹ http://www.dabeaz.com/python/GIL.pdf

Change-Id: Id5a9927e8ec2179d5b33663b3f05bb0482d54242
Signed-off-by: Csaba Henk <csaba@redhat.com>