cluster/afr: detect in-progress creation in lookup and return ENOENT

if any subvol returned ENOENT while parent entrylk lock was held,
yield and return ENOENT for the entire lookup.

This is how the issue happens:

Multiple clients A, B and C are attempting 'mkdir -p /mnt/a/b/c'

1 Client A is in the middle of mkdir(/a). It has acquired lock.
  It has performed mkdir(/a) on one subvol, and second one is still
  in progress
2 Client B performs a lookup, sees directory /a on one,
  ENOENT on the other, succeeds lookup.
3 Client B performs lookup on /a/b on both subvols, both return ENOENT
  (one subvol because /a/b does not exist, another because /a
  itself does not exist)
4 Client B proceeds to mkdir /a/b. It obtains entrylk on inode=/a with
  basename=b on one subvol, but fails on other subvol as /a is yet to
  be created by Client A.
5 Client A finishes mkdir of /a on other subvol
6 Client C also attempts to create /a/b, lookup returns ENOENT on
  both subvols.
7 Client C tries to obtain entrylk on on inode=/a with basename=b,
  obtains on one subvol (where B had failed), and waits for B to unlock
  on other subvol.
8 Client B finishes mkdir() on one subvol with GFID-1 and completes
  transaction and unlocks
9 Client C gets the lock on the second subvol, At this stage second
  subvol already has /a/b created from Client B, but Client C does not
  check that in the middle of mkdir transaction
10 Client C attempts mkdir /a/b on both subvols. It succeeds on
   ONLY ONE (where Client B could not get lock because of
   missing parent /a dir) with GFID-2, and gets EEXIST from ONE subvol.
This way we have /a/b in GFID mismatch. One subvol got GFID-1 because
Client B performed transaction on only one subvol (because entrylk()
could not be obtained on second subvol because of missing parent dir --
caused by premature/speculative succeeding of lookup() on /a when locks
are detected). Other subvol gets GFID-2 from Client C because while
it was waiting for entrylk() on both subvols, Client B was in the
middle of creating mkdir() on only one subvol, and Client C does not
"expect" this when it is between lock() and pre-op()/op() phase of the
transaction.

Original-author: Anand Avati <avati@redhat.com>
Change-Id: Idca475dbbc2a51e09da6fa0f9e1e37148caef208
BUG: 860210
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4625
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Tests: Change rebalance status verification

According to the comment at the following URL
https://bugzilla.redhat.com/show_bug.cgi?id=916226#c2
"success:" can come even before rebalance is completed.
Changed it to check for "completed" instead.

Change-Id: Ibe9d3b75493240f30261ac2a1280f32ef32886da
BUG: 916226
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4614
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: add more specific log messages

Change-Id: I57fbdd83f3098e64886c3dd690a1ae04fc37442d
BUG: 928648
Signed-off-by: Bala.FA <barumuga@redhat.com>
Reviewed-on: http://review.gluster.org/4739
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cli: add more meaningful error messages

Change-Id: I6e88e6763fa537f4705427b4673d86e6443c2c98
BUG: 928648
Signed-off-by: Bala.FA <barumuga@redhat.com>
Reviewed-on: http://review.gluster.org/4747
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
rpc-lib: Move defaulting to socket message to debug level

Problem:
For every gluster cli operation from command line
rpc init process is required. During init process we
print "no transport type set, defaulting to socket"
message at WARNING level, which is not necessary.

Solution:
Moved the log level to DEBUG.

Change-Id: I73f4644264368b0f6c11a77ef66595018784ce79
BUG: 928204
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/4727
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
build: require /usr/bin/fusermount when not carrying our own version

The fuse.so from glusterfs-fuse will try to call /usr/bin/fusermount.
This obviously fails when the fuse package is not installed and
fusermount is not available.

In order to prevent this problem, the glusterfs-fuse package should
require /usr/bin/fusermount so that it gets automatically pulled in when
glusterfs-fuse is installed with yum.

BUG: 947830
Change-Id: I20fe836a1aaf751dbc04d9ec4ba5ea50573c71c5
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/4768
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/distribute: Start rebalance with option readdir-optimize on

With readdir-optimize set to on, we instruct the posix layer to ignore
directory entries from not first subvolume. DHT discards directories
returned from non first subvolume. By making posix itself ignore it,
we are making directory crawls faster

Change-Id: Ia1faf2dedec0c615c0632c3c063e846f5742ede6
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/4613
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
core: add dispatch table for init/fini

This adds a layer of indirection so that derivative translators such as
NUFA and switch can refer to the parent's init/fini (in both cases DHT's)
without having to create stub functions.

Change-Id: I1af1fea70a9ddd2aa20485af7ae65f9660f19dd6
BUG: 924490
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.org/4709
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
dht: make nufa/switch call dht's init/fini

These functions keep changing as new functionality is added, so copying
and pasting the code is not a good solution.  This way ensures that all
fields get initialized properly no matter how much new stuff we throw in.

Change-Id: I9e9b043d2d305d31e80cf5689465555b70312756
BUG: 924488
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.org/4710
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
protocol/client: Print valid loc identifiers

Change-Id: I45f91105862a2484b8906a7a63b98ab4aaf80d05
BUG: 924643
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4683
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
build: enable fusermount by default

The fusermount available in gluster is customized to ensure
mounting with SELinux happens properly, i.e - to have a separate
thread for fuse_thread_proc which can process getxattr requests
and in parallel perform sys_mount() in a different thread, thereby
avoiding a deadlock.

However our build and packaging defaults to not including our
fusermount. This patch reverses the defaults.

Change-Id: I793af4c2f56aeac46efae3db30e7c64ee7c18850
BUG: 811217
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4773
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
pump: Set self-heal readdir size in pump

Problem:
In Pump entry self-heal happens for each directory during the
first opendir using conservative merge. But in entry-self-heal
readdir is issued with '0' size. So entry self-heal is not
creating any files. After pump thinks entry self-heal is complete
it proceeds to heal each of the file in the directory it just
healed. Fortunately most of the times it chooses source-brick
in pump as read-child for readdir. This happens because readchild is the
subvolume on which lookup succeeds first. In pump lookup succeeds
faster in local process than on the destination brick process most
of the times. For all the entries pump finds in readdir it does a
lookup. During this lookup the entry on the destination brick is
created and healed. This is the reason why replace-brick
succeeds whenever read-child for the directory is chosen as the
source-brick.  Which is most of the times. When read-child is chosen
as the destination brick, readdir returns no entries so replace-brick
completes without syncing the whole data.

Fix:
Set readdir-size in pump so that entry self-heal happens with
64k size. This ensures that entry self-heal triggered from
opendir actually creates the files on the destination brick.

Change-Id: I65ea45d3c2735a9578f3aa34eff771b6563241ca
BUG: 909800
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4712
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
dict: Put "goto out" in dict_unserialize to avoid process crash

Problem:
In the dictionary serialization function, if the
[(buf + vallen) > (orig_buf + size)], then memdup is getting failed.

Fix:
Put "goto out" whenever this condition is met.

Change-Id: I662628a936596dbb47825aad47d7dbab2879eb07
BUG: 947824
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/4767
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/afr: Made afr_sh_purge_entry_common message log more clear.

FIX:
In missing entry self heal, once the source directories are determined
after the lookup and if file is not present on any of the brick which
contains the souce directory, the entry is removed from the directory.
So log message should give information of "Purging of entry".

Change-Id: I4d3deb602e0812dc1c9c8ba0a466716d81dede7e
BUG: 947312
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/4753
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
posix: fix dangerous "sharing" of fd in readdir between two requests

posix_fill_readdir() is a multi-step function which performs many
readdir() calls, and expects the directory cursor to have not
"seeked away" elsewhere between two successive iterations. Usually
this is not a problem as each opendir() from an application has its
own backend fd, and there is nobody else to "seek away" the directory
cursor. However in case of NFS's use of anonymous fd, the same fd_t
is shared between all NFS readdir requests, and two readdir loops can
be executing in parallel on the same dir dragging away the cursor in
a chaotic manner.

The fix in this patch is to lock on the fd around the loop. Another
approach could be to reimplement posix_fill_readdir() with a single
getdents() call, but that's for another day.

Change-Id: Ia42e9c7fbcde43af4c0d08c20cc0f7419b98bd3f
BUG: 948086
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4774
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: Treat all dir fop failure as success in changelog

For example:
If a new entry creation fop fails with EEXIST or a delete entry fop
fails with ENOENT, on all the subvols the fop is wound, then no
change took place to the directory. So we can treat that case as no
change happened to the directory.

Change-Id: I3b3a7931954da2166a9cba19ff9f76f37739d751
BUG: 860210
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4626
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
tests: Remove grep process entries from pidgrep

Problem:
We were picking process with lowest pid from ps|grep
result. However, lowest pid need not be oldest process
as recycling of PIDs can take place.

Solution:
Removed grep process entries from ps entries using 
grep -v grep.

Change-Id: I2b9687a05a34cf6358f773183770d69a3fb9eb10
BUG: 858488
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/4765
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
gsync: Display additional information in status command

Added code to display extra information when status command
is executed.
 Information shown now are
1 Number of files synced
2 crawl time
3 total sync time
4 bytes synced

bytes synced is taken from rsync output .
--stats option of rsync gives extra infor
mation about the sync.In stats output there
is a field called Total transferred file
size which states the ammount of bytes synced .
This information is parsed from stdout output
using regular expressions.Bytes synced information
can be used to calculate throughput.

Change-Id: Id9bba9fff45ee7049bb8257c6fd918e5237e05b1
BUG: 947774
Signed-off-by: sarvotham s pai <spai@redhat.com>
Reviewed-on: http://review.gluster.org/4749
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>