tests/fileio.rc: library for file descriptor based IO in tests

There are situations in test scripts where we want to keep open file
descriptors while performing other commands. Bash has abilities
to manage file descriptors by numbers, but the syntax is a little
brain damaging.

This library provides wrappers around it to abstract away bash's
syntax and also provides a helper function to pick a free file
descriptor on the fly.

The APIs are pretty self explanatory.

Change-Id: I82f1d1957646dd6c468d9e85c90ec30c978c7ad6
BUG: 764966
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4565
Reviewed-by: Amar Tumballi <amarts@redhat.com>
tests: move common funtion definitions to include.rc

Change-Id: I3526557d0a675692e94edb71b7d92c9202e6afa4
BUG: 765473
Signed-off-by: Raghavendra G <raghavendra@gluster.com>
Reviewed-on: http://review.gluster.org/4559
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: Added validation function for quota-timeout.

Change-Id: I7f82f4217a41e0fe41272e6ef82925e1fe97fcd5
BUG: 765230
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4557
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: Don't queue transactions during open-fd fix

Before Anonymous fds are available, afr had to queue up
transactions if the file is not opened on one of its
subvolumes. This happens until the attempt to open the
file either succeeds or fails. These attempts happen
until the file is successfully opened on the subvolume.
Now client xlator uses anonymous fds to perform the fops
if the fd used for the fop is not 'opened'.
Fops will be successful even when the file is not opened
so there is no need to queue up the transactions anymore in afr.
Open is attempted on the subvolume where it is not
opened independent of the fop.

Change-Id: Id1a4b4ebe6f89f9efe8f6a8247918b91247d0819
BUG: 913051
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4568
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
libglusterfs: Fixed valid_ipv4_address and valid_host_name

In valid_ipv4_address(), strtok_r was ignoring consecutive dots,
and was marking any address with consecutive dots as valid. Now
such addresses will be marked as invalid.

In valid_host_name(), any name ending with any special characters
was marked as valid. Now such hostnames will be marked as invalid.

Change-Id: If9b625d9040a14bde8c176fe36e337cb483e284a
BUG: 822830
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4579
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/distribute: Preserve file size during rebalance migration

If holes are encountered, then we do not write these to the dst,
which sometimes causes file size to be lesser than src. Data is not
corrupted, as when non-zero reads are received, we do write that data.

Calling a truncrate to give file size to prevent it from being
truncated to less than src in case the file end has holes.

Thanks to Brian Foster for providing the test case

Change-Id: I3cdd143b63ec8d797273d76189dff8b05eb9e551
BUG: 915554
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4574
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
volgen: Use bind-address option for bricks when option set on glusterd

Brick processes listen on all the interfaces on a given port.
When multiple glusterds run on one machine, glusterd assumes
that it 'owns' the ports on that machine. This can lead to the
different glusterd instances to step on each other's ports.
This fix ensures that brick processes listen only on the its
host IP when glusterd has bind-address option set.

Change-Id: I4c1b05643c64d3098bf56e977e768e611ffce0f5
BUG: 913662
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/4580
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
glusterd: Increasing throughput of synctask based mgmt ops.

Change-Id: Ibd963f78707b157fc4c9729aa87206cfd5ecfe81
BUG: 913662
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/4570
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
tests/cluster.rc: support for virtual multi-server glusterd tests

Since http://review.gluster.org/4556 glusterd is capable of running
many instances of itself on a single system. This patch exploits
that feature and enhances the regression test framework to expose
handy primitives so that test cases may be written to test glusterd
in a cluster.

Usage:

1. Include "$(dirname)/../cluster.rc" to get access to the extensions

2. Call launch_cluster $N where $N is the count of virtual servers

Calling launch_cluster, starts $N glusterds which bind to $N different
IPs and dynamically defines these primitives:

 - Variables $H1 .. $Hn assigned to hostnames of each "server".

 - Variables $CLI_1 .. $CLI_n assigned as commands to run CLI commands
   on the corresponding N'th server.

 - Variables $B1 .. $Bn assigned to the backend directories on each
   "server".

 - Function kill_glusterd, which accepts a parameter - index number of
   glusterd to be killed.

 - Variables $glusterd_1 .. $glusterd_n assigned to the command lines
   to restart the corresponding glusterd, if it was previously killed.

The current set of primitives and functions were implemented with the goal
of satisfying ./tests/bugs/bug-913555.t. The API will be made richer as
we add more cluster test cases

Change-Id: Ieb13ed9f4a72ac0321db0ca0844c7b294145bb32
BUG: 913555
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4566
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
debug/error-gen: Add support for reconfiguring options.

Change-Id: Ia10dc29e8608b02037b08e32a72766b6d43a98ba
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4576
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
mgmt/glusterd: Expose error-gen options through volume set.

Change-Id: I7c696d99b43544923fb96d177229cdbac32c09fe
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4577
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
nfs/server: Fix multiple crashes in acl handling code.

Change-Id: I9b39a485c8b98d9eabe6153487f4dfbd26f8af13
BUG: 915280
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/4578
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
mgmt/glusterd: Move found-brick logs to DEBUG

Change-Id: I1c311c21d7bdcad4956d3428bda39131c331cd7a
BUG: 812356
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4585
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/distribute: Add filter to support file patterns to be migrated

'gluster volume rebalance' command will be enhanced to support passing of these
options/pattern.

<pattern> is comma separated list as show below. The Precedence is from right
to left.

e.g- "*avi,*pdf:10MB,*:1KB"
     The precedence is as follows:
     migrate all files with size equal or greater than 1KB "*:1KB"
     migrate all pdf files with size equal or greater than 10MB "*pdf:10MB"
     migrate all avi files "*avi"

With this option, it is possible to choose which files to migrate.

Change-Id: I6d6d6a015bcbacf1debae2f278a2d92306fb055d
BUG: 896456
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4366
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/dht: print hash function munging logs in DEBUG mode

Change-Id: Ia2e6bce80710d103da9d78afdb389ea162b00686
BUG: 912564
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4590
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
glusterd: Added validation function for stripe-block-size.

Change-Id: I050d01b01eac46550aa435da7d96a972e0393d35
BUG: 770655
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/4561
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
iobuf: Added a function iobref_iobuf_unref

Original-author: Venky Shankar <vshankar@redhat.com>

Change-Id: Ibf861db6c1b084b798d210962344487a1919aad2
Signed-off-by: Prashanth Pai <nullpai@gmail.com>
features/compress: Compression/DeCompression translator

Changes made to original code :
* Changed Makefile.am to reflect changes from commit 58e6296f
* Code to be merged into glusterd-volgen.c moved to
  glusterd-volume-set.c file introduced by commit 99fe09f4
* Added .default_value and .description to xlator options[]
  and removed them from glusterd-volume-set.c
* Moved all cdc xlator specific functions to cdc-helper.c
  retaining only fop implementations in cdc.c
* Changed op_version form 0 to 2 in glusterd-volume-set.c

The following work has been done by Venky Shankar.
Original-author : Venky Shankar <vshankar@redhat.com>

On the server this translator compresses (deflate) data before
it's pushed out to the network to the client. On the client
this compressed data is decompressed (inflate) to it's original
form. What is saved is the amount of data that is sent over
the wire.

(De)Compression is done using Zlib library (inflate/deflate
routines respectively). As of now, data is transferred in
gzip format (header + <compressed data> + trailer). The
header can be taken away to make data more lightweight.
Trailer is needed as it contains the CRC and length of the
original (uncompressed) data used for validating inflated
data.

HOW TO USE
----------
As of now, you'd need to edit the volfile to use this feature;
there is no way to do so from CLI. On the server, load it above
the server translator so that data is compressed just before it
is send out to the network. On the client, load it below the
client translator. But since there can be multiple instances of
the client translator (distribute, replicate etc..), you'd need
to have that many instances of this translator *pointing* to
it's respective client translator.

Change-Id: Ib7a66b6f1f70fe002b7c513588cdf75c69370805
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Signed-off-by: Prashanth Pai <nullpai@gmail.com>
features/compress: Added writev

Writev compresses data on client and decompresses on server.
Similarly readv compresses data on server and decompresses
on client.

The possible values for "mode" option of the cdc xlator
were compress/decompress earlier. This has been changed
after adding writev. Currently the mode option can be
client/server.

Removed gzip header while compressing which was an overhead.
Compressed data is identified by xdata key/value

Other minor changes :
* Check for min file size and buffer size while loading xlator
* Remove cleanup_out label from fop

Change-Id: If4eb4f8af2d63b4bc059e9daaecfda9e1ad8ea56
Signed-off-by: Prashanth Pai <nullpai@gmail.com>
features/compress: Set compression xlator options from CLI

Ex : gluster volume set <vol_name> compress on
     gluster volume set <vol_name> compress.buffer-size 32768

Change-Id: Id30459fd76f7b5155d7b063c36e9caeee4923339
Signed-off-by: Prashanth Pai <nullpai@gmail.com>