synctask: support for (assymetric) counted barriers

This patch introduces a new set of primitives:

  - synctask_barrier_init (stub)
  - synctask_barrier_waitfor (stub, count)
  - synctask_barrier_wake (stub)

Unlike pthread_barrier_t, this barrier has an explicit notion of
"waiter" and "waker". The "waiter" waits for @count number of
"wakers" to call synctask_barrier_wake() before returning. The
wait performed by the waiter via synctask_barrier_waitfor() is
co-operative in nature and yields the thread for scheduling other
synctasks in the mean time.

Intended use case:

  Eliminate excessive serialization in glusterd and allow for
concurrent RPC transactions.

  Code which are currently in this format:

---old---

  list_for_each_entry (peerinfo, peers, op_peers_list) {
          ...
          GD_SYNCOP (peerinfo->rpc, stub, rpc_cbk, ...);
  }

  ...

  int rpc_cbk (rpc, stub, ...)
  {
          ...
          __wake (stub);
  }

---old---

  Can be restructred into the format:

---new---

  synctask_barrier_init (stub);
  {
          list_for_each_entry (peerinfo, peers, op_peers_list) {
                  ...
                  rpc_submit (peerinfo->rpc, stub, rpc_cbk, ...);
                  count++;
           }
   }
   synctask_barrier_wait (stub, count);

   ...

   int rpc_cbk (rpc, stub, ...)
   {
           ...
           synctask_barrier_wake (stub);
   }

---new---

  In the above structure, from the synctask's point of view, the region
between synctask_barrier_init() and synctask_barrier_wait() are spawning
off asynchronous "threads" (or RPC) and keep count of how many such
threads have been spawned. Each of those threads are expected to make
one call to synctask_barrier_wake(). The call to synctask_barrier_wait()
makes the synctask thread co-operatively wait/sleep till @count such threads
call their wake function.

  This way, the synctask thread retains the "synchronous" flow in the code,
yet at the same time allows for asynchronous "threads" to acheive parallelism
over RPC.

Change-Id: Ie037f99b2d306b71e63e3a56353daec06fb0bf41
BUG: 913662
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4558
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Krishnan Parthasarathi <kparthas@redhat.com>
tests/fileio.rc: library for file descriptor based IO in tests

There are situations in test scripts where we want to keep open file
descriptors while performing other commands. Bash has abilities
to manage file descriptors by numbers, but the syntax is a little
brain damaging.

This library provides wrappers around it to abstract away bash's
syntax and also provides a helper function to pick a free file
descriptor on the fly.

The APIs are pretty self explanatory.

Change-Id: I82f1d1957646dd6c468d9e85c90ec30c978c7ad6
BUG: 764966
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4565
Reviewed-by: Amar Tumballi <amarts@redhat.com>
tests: move common funtion definitions to include.rc

Change-Id: I3526557d0a675692e94edb71b7d92c9202e6afa4
BUG: 765473
Signed-off-by: Raghavendra G <raghavendra@gluster.com>
Reviewed-on: http://review.gluster.org/4559
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: Added validation function for quota-timeout.

Change-Id: I7f82f4217a41e0fe41272e6ef82925e1fe97fcd5
BUG: 765230
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4557
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: Don't queue transactions during open-fd fix

Before Anonymous fds are available, afr had to queue up
transactions if the file is not opened on one of its
subvolumes. This happens until the attempt to open the
file either succeeds or fails. These attempts happen
until the file is successfully opened on the subvolume.
Now client xlator uses anonymous fds to perform the fops
if the fd used for the fop is not 'opened'.
Fops will be successful even when the file is not opened
so there is no need to queue up the transactions anymore in afr.
Open is attempted on the subvolume where it is not
opened independent of the fop.

Change-Id: Id1a4b4ebe6f89f9efe8f6a8247918b91247d0819
BUG: 913051
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4568
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
libglusterfs: Fixed valid_ipv4_address and valid_host_name

In valid_ipv4_address(), strtok_r was ignoring consecutive dots,
and was marking any address with consecutive dots as valid. Now
such addresses will be marked as invalid.

In valid_host_name(), any name ending with any special characters
was marked as valid. Now such hostnames will be marked as invalid.

Change-Id: If9b625d9040a14bde8c176fe36e337cb483e284a
BUG: 822830
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4579
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/distribute: Preserve file size during rebalance migration

If holes are encountered, then we do not write these to the dst,
which sometimes causes file size to be lesser than src. Data is not
corrupted, as when non-zero reads are received, we do write that data.

Calling a truncrate to give file size to prevent it from being
truncated to less than src in case the file end has holes.

Thanks to Brian Foster for providing the test case

Change-Id: I3cdd143b63ec8d797273d76189dff8b05eb9e551
BUG: 915554
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4574
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
volgen: Use bind-address option for bricks when option set on glusterd

Brick processes listen on all the interfaces on a given port.
When multiple glusterds run on one machine, glusterd assumes
that it 'owns' the ports on that machine. This can lead to the
different glusterd instances to step on each other's ports.
This fix ensures that brick processes listen only on the its
host IP when glusterd has bind-address option set.

Change-Id: I4c1b05643c64d3098bf56e977e768e611ffce0f5
BUG: 913662
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/4580
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
glusterd: Increasing throughput of synctask based mgmt ops.

Change-Id: Ibd963f78707b157fc4c9729aa87206cfd5ecfe81
BUG: 913662
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/4570
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
tests/cluster.rc: support for virtual multi-server glusterd tests

Since http://review.gluster.org/4556 glusterd is capable of running
many instances of itself on a single system. This patch exploits
that feature and enhances the regression test framework to expose
handy primitives so that test cases may be written to test glusterd
in a cluster.

Usage:

1. Include "$(dirname)/../cluster.rc" to get access to the extensions

2. Call launch_cluster $N where $N is the count of virtual servers

Calling launch_cluster, starts $N glusterds which bind to $N different
IPs and dynamically defines these primitives:

 - Variables $H1 .. $Hn assigned to hostnames of each "server".

 - Variables $CLI_1 .. $CLI_n assigned as commands to run CLI commands
   on the corresponding N'th server.

 - Variables $B1 .. $Bn assigned to the backend directories on each
   "server".

 - Function kill_glusterd, which accepts a parameter - index number of
   glusterd to be killed.

 - Variables $glusterd_1 .. $glusterd_n assigned to the command lines
   to restart the corresponding glusterd, if it was previously killed.

The current set of primitives and functions were implemented with the goal
of satisfying ./tests/bugs/bug-913555.t. The API will be made richer as
we add more cluster test cases

Change-Id: Ieb13ed9f4a72ac0321db0ca0844c7b294145bb32
BUG: 913555
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4566
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
debug/error-gen: Add support for reconfiguring options.

Change-Id: Ia10dc29e8608b02037b08e32a72766b6d43a98ba
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4576
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
mgmt/glusterd: Expose error-gen options through volume set.

Change-Id: I7c696d99b43544923fb96d177229cdbac32c09fe
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4577
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
nfs/server: Fix multiple crashes in acl handling code.

Change-Id: I9b39a485c8b98d9eabe6153487f4dfbd26f8af13
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4578
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
mgmt/glusterd: Move found-brick logs to DEBUG

Change-Id: I1c311c21d7bdcad4956d3428bda39131c331cd7a
BUG: 812356
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4585
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/distribute: Add filter to support file patterns to be migrated

'gluster volume rebalance' command will be enhanced to support passing of these
options/pattern.

<pattern> is comma separated list as show below. The Precedence is from right
to left.

e.g- "*avi,*pdf:10MB,*:1KB"
     The precedence is as follows:
     migrate all files with size equal or greater than 1KB "*:1KB"
     migrate all pdf files with size equal or greater than 10MB "*pdf:10MB"
     migrate all avi files "*avi"

With this option, it is possible to choose which files to migrate.

Change-Id: I6d6d6a015bcbacf1debae2f278a2d92306fb055d
BUG: 896456
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4366
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/dht: print hash function munging logs in DEBUG mode

Change-Id: Ia2e6bce80710d103da9d78afdb389ea162b00686
BUG: 912564
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4590
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
glusterd: Added validation function for stripe-block-size.

Change-Id: I050d01b01eac46550aa435da7d96a972e0393d35
BUG: 770655
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4561
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Do not call xdr_string() with a NULL error message

It is illegal to call xdr_string() with a NULL string. Linux
just retruns false, NetBSD gets a SIGSEGV when xdr_string()
calls strlen(NULL)

BUG: 916439
Change-Id: Ia958470ada6e8e55a86d439922ec942d038f5f13
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/4589
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/distribute: Prevent spurious multiple defrag crawls

In dht_notify, we used to create a thread to start defrag
crawls after we had heard from all child subvols.
This was in-correct, as a later event, could also trigger the
crawl again(due to the fact that all subvols had responded).

The fix is to make sure, the thread is started only once after
all subvols have responded the first time

Change-Id: Ifc2978b9dc866af2395b79911eca50ab38ff9457
BUG: 916449
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4587
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
glusterd: Fix some options in vme table

Some of the options had invalid '.flags' members. In the original table
these table these were supposed to be the op-versions, but, the entries
for the below options were missing the flags field the op-version was
entered in that place.

Change-Id: I408f5a972743eb37d9a58a809e8be8cb385bced8
BUG: 903478
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/4593
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>