cluster/afr: Allow data/entry self heal for metadata split-brain

Problem:
Currently whenever there is metadata split-brain, a variable
sh->op_failed is set to 1 to denote that self heal got failed.
But if we proceed for data self heal, even code-path of data
self heal also relies on the sh->op_failed variable. So if will
check for sh->op_failed variable and will eventually fails to
do data self heal. So needed a mechanism to allow data self heal
even if metadata is in split brain.

Fix:
Some data structure revamp is done in
http://review.gluster.com/#/c/5106/ fix and this patch is
based on the above fix. Now we can store which particular self-heal
got failed i.e GFID_OR_MISSING_ENTRY_SELF_HEAL, METADATA, DATA,
ENTRY. And we can do two types of self heal failure check.
1. Individual type check: We can check which among all four
   (Metadata, Data, Gfid or missing entry, entry self heal)
   got failed.

2. In afr_self_heal_completion_cbk, we need to make check
   based on the fact that if any specific self heal got failed treat
   the complete self heal as failure so that it will populate
   corresponding circular buffer of event history accordingly.

Change-Id: Icb91e513bcc752386fc8a78812405cfabe5cac2d
BUG: 977797
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/5253
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
store: Fix resource leaks in gf_store_iter_* functions

Also, removed (redundant) member fd from gf_store_iter_t

Change-Id: I40f0469997f77fa2f578a5495ca4ce285f1a59f2
BUG: 904065
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/5243
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
glusterd: More checks before starting rebalance/remove-brick

Check if a previous remove-brick operation has been committed before
starting a new rebalance/remove-brick task.

Change-Id: I553e5ba64a6a352ca91032ab1a17997051a4494e
BUG: 963541
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/5019
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: post-op should complete before starting flush

Problem:
At the moment afr-flush makes sure that a delayed post-op
is woken up but it does not wait for it to complete the
post-op before flush unwinds.
These are the steps that are happening:
1) flush fop comes on an fd which wakes up a delayed post-op
and continues with the flush fop.
2) post-op sends fsync on the wire.
3) flush completes and unwinds to fuse.
4) graph switch happens on the fuse mount disconnecting the
old graph's client connections to bricks.
5) xattrop after fsync fails with ENOTCONN because the
connections from old graph are taken down now.

Fix:
Wait for post-op to complete before starting to flush.
We could make flush act similar to fsync (i.e.) wind
flush as is but wait for post-op to complete before unwinding
flush, but it is better to send flush as the final fop. So
wind of flush will start after post-op is complete. Had to
change fsync to accommodate this change.

Change-Id: I93aa642647751969511718b0e137afbd067b388a
BUG: 980548
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5274
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/afr: Provide an option to disable afr durability

Change-Id: I40eec20ca6b3f857245a2438883822e251077ee9
BUG: 979365
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5269
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/afr: Refactor inodelk to handle multiple domains

- afr_local_copy should not be memduping locked nodes, that would
  mean that lock is taken in self-heal on those nodes even before
  it actually takes the lock. So removed memdup code. Even entry
  lock related copying (lockee info) is also not necessary for
  self-heal functionality, so removing that as well. Since it is
  not local_copy anymore changed its name.

- My editor changed tabs to spaces.

Change-Id: I8dfb92cb8338e9a967c06907a8e29a8404782d61
BUG: 967717
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5099
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/afr: Let two data-self-heals compete in new domain

Problem:
At the moment data-self-heal acquires locks in following
pattern. It takes full file lock then gets xattrs on files on both
replicas. Decides sources/sinks based on the xattrs. Now it acquires
lock from 0-128k then unlocks the full file lock. Syncs 0-128k range
from source to sink now acquires lock 128k+1 till 256k then unlocks
0-128k, syncs 128k+1 till 256k block... so on finally it takes full file
lock again then unlocks the final small range block.
It decrements pending counts and then unlocks the full file lock.

     This pattern of locks is chosen to avoid more than 1 self-heal
to be in progress. BUT if another self-heal tries to take a full
file lock while a self-heal is already in progress it will be put in
blocked queue, further inodelks from writes by the application will
also be put in blocked queue because of the way locks xlator grants
inodelks. So until the self-heal is complete writes are blocked.

Here is the code:
xlators/features/locks/src/inodelk.c - line 225
if (__blocked_lock_conflict (dom, lock) && !(__owner_has_lock (dom, lock))) {
         ret = -EAGAIN;
         if (can_block == 0)
                 goto out;

         gettimeofday (&lock->blkd_time, NULL);
         list_add_tail (&lock->blocked_locks, &dom->blocked_inodelks);
}

This leads to hangs in applications.

Fix:
Since we want to prevent two parallel self-heals. We let them compete
in a separate "domain". Lets call the domain on which the locks have
been taken on in previous approach as "data-domain".

In the new approach When a self-heal is triggered,
it acquires a full lock in the new domain "self-heal-domain".
    After this it performs data-self-heal using the locks in
    "data-domain" as before.
unlock the full file lock in "self-heal-domain"

With this approach, application's writevs don't have to wait
in pending queue when more than 1 self-heal is triggered.

Change-Id: Id79aef3dfa888945977fb9758374ac41c320d0d5
BUG: 967717
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5100
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli: Fix in letter case in volume heal output

Change-Id: I25d13444c2cbff9b26642e91677ad1e09e77aa1e
BUG: 978936
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/5259
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
fuse: pass the mountflags to older (< 2.6.21) kernels

The change for Bug 853895 fixed mounting volumes read-only for recent
kernels. Older kernels fail the first mount() syscall, and the second
mount() did not add the 'mountflags'.

Full analysis and a description for reproducing is in the bugreport. The
test included in http://review.gluster.org/4163 would have caught this
problem when the tests are executed on RHEL-5 or similar systems.

Change-Id: I440591344a6a5af7b2018e37a2a1fda9de8b5ab2
Bug: 980770
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/5278
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
posix: add a simple health-checker

Goal of this health-checker is to detect fatal issues of the underlying
storage that is used for exporting a brick. The current implementation
requires the filesystem to detect the storage error, after which it will
notify the parent xlators and exit the glusterfsd (brick) process to
prevent further troubles.

The interval the health-check runs can be configured per volume with the
storage.health-check-interval option. The default interval is 30
seconds.

It is not trivial to write an automated test-case with the current
prove-framework. These are the manual steps that can be done to verify
the functionality:

- setup a Logical Volume (/dev/bz970960/xfs) and format is as XFS for
  brick usage

- create a volume with the one brick

    # gluster volume create failing_xfs glufs1:/bricks/failing_xfs/data
    # gluster volume start failing_xfs

- mount the volume and verify the functionality

- make the storage fail (use device-mapper, or pull disks)

    # dmsetup table
    ..
    bz970960-xfs: 0 196608 linear 7:0 2048

    # echo 0  196608 error > dmsetup-error-target
    # dmsetup load bz970960-xfs dmsetup-error-target
    # dmsetup resume bz970960-xfs

    # dmsetup table
    ...
    bz970960-xfs: 0 196608 error

- notice the errors caught by syslog:

    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): metadata I/O error: block 0x0 ("xfs_buf_iodone_callbacks") error 5 buf count 512
    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): I/O Error Detected. Shutting down filesystem
    Jun 24 11:31:49 vm130-32 kernel: XFS (dm-2): Please umount the filesystem and rectify the problem(s)
    Jun 24 11:31:49 vm130-32 kernel: VFS:Filesystem freeze failed
    Jun 24 11:31:50 vm130-32 GlusterFS[1969]: [2013-06-24 10:31:50.500674] M [posix-helpers.c:1114:posix_health_check_thread_proc] 0-failing_xfs-posix: health-check failed, going down
    Jun 24 11:32:09 vm130-32 kernel: XFS (dm-2): xfs_log_force: error 5 returned.
    Jun 24 11:32:20 vm130-32 GlusterFS[1969]: [2013-06-24 10:32:20.508690] M [posix-helpers.c:1119:posix_health_check_thread_proc] 0-failing_xfs-posix: still alive! -> SIGTERM

- these errors are in the log of the brick as well:

    [2013-06-24 10:31:50.500607] W [posix-helpers.c:1102:posix_health_check_thread_proc] 0-failing_xfs-posix: stat() on /bricks/failing_xfs/data returned: Input/output error
    [2013-06-24 10:31:50.500674] M [posix-helpers.c:1114:posix_health_check_thread_proc] 0-failing_xfs-posix: health-check failed, going down
    [2013-06-24 10:32:20.508690] M [posix-helpers.c:1119:posix_health_check_thread_proc] 0-failing_xfs-posix: still alive! -> SIGTERM

- the glusterfsd process has exited correctly:

    # gluster volume status
    Status of volume: failing_xfs
    Gluster process						Port	Online	Pid
    ------------------------------------------------------------------------------
    Brick glufs1:/bricks/failing_xfs/data			N/A	N	N/A
    NFS Server on localhost					2049	Y	1897

Change-Id: Ic247fbefb97f7e861307a5998a9a7a3ecc80aa07
BUG: 971774
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/5176
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli: Fix remove brick cli out for wrong volume name

Problem:  gluster volume remove-brick command, was not printing the error in
          case of volume-name specified is wrong.

Fix:      Fix will print error message to indicate that provided volume name is
          invalid. Although patch for bug 961669
          http://review.gluster.org/#/c/4975/ does print cli-output now, but
          still xml is unable to use the response values

Change-Id: I2ee1df86c1e756fb8e93b4d6bbdd102b4f368f87
BUG: 961307
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/4972
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd/common-utils: move hostname helper functions to common-utils

Change-Id: If47e209cb61ea0eb74ee2d6ef9e9342b2d6ee13a
BUG: 980838
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/5261
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
nufa: allow subvols with fanout > 1

Previously, nufa wouldn't work on volume topologies such as
distribute-replicate or distribute-stripe.

Change-Id: Ia89ed4412a00601022c1fc94f046056ce4820fe8
BUG: 980838
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/5262
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
api: Bump version to 5.

version 5 adds support for fallocate & discard.

Change-Id: Ifdc226953a25e4a4eb63e55d2a7c7ec77c397215
BUG: 953694
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/5276
Reviewed-by: Bharata B Rao <bharata.rao@gmail.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
glusterd: Bump GD_MAX_OP_VERSION

Change-Id: I7a212435ce7e98fe01aad2c0d1f698de8ea84235
BUG: 981278
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/5287
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>