cluster/afr: Pre-op should be undone for non-piggyback post-op

Problem:
When fop fails post-op is always performed
over the network irrespective of whether pre-op is piggybacked
or not. Decrementing Pre-op-done count even for the piggybacked
ones is wrong.
I have added an assert for pre_op_done to be non-zero and when
dd of=a if=/dev/urandom bs=5M count=1000 is executed and a brick
is taken down, the mount is crashing.

Fix:
Decrement pre-op-done count only when the post-op is not
piggybacked.

Change-Id: Ie837251a43bfb437f0fada191302eeee60be1601
BUG: 863939
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4310
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cli: output xml in pretty format

Gluster cli now prints XML outputs in 'pretty' format. This solves the problem
of empty elements occuring as two tags instead of being collapsed into one.

Change-Id: Iab7aeadcff29c18ae388b58e446a16e937ac09ed
BUG: 849293
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/4355
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
object-storage: add UFO unit tests

Run the UFO unit tests following the main GlusterFS unit tests. The
UFO unit tests run under the Nose unit test running harness, requiring
the python-nose package to be installed. These unit tests run fast
enough that we could even consider running them on a commit hook run
by rfc.sh.

Note that they are not functional tests, as they don't require a real
Gluster file system to run.

Change-Id: I55972890c24be16fc588a6950cb6bfd539287bbe
Signed-off-by: Peter Portante <peter.portante@redhat.com>
Reviewed-on: http://review.gluster.org/4333
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
cluster/dht: update ctx-time only if we receive the new iatt

1. Used local->postparent(contains merged iatt of all succesful calls) instead
of postparent for dht ctx time update.

2. dht_inode_ctx_time_update avoided in case of opret -1.

Change-Id: Ie04a7842a41c241f911b6a3f76267b996d27fb43
BUG: 881013
Signed-off-by: Varun Shastry <vshastry@redhat.com>
Reviewed-on: http://review.gluster.org/4338
Reviewed-by: Shishir Gowda <sgowda@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
afr: replace afr_more_important_error with afr_most_important_error

afr_more_important_error() is written to return whether a new errno
should override an existing errno for high-level operations that
could span multiple sub-operations. It specifically prioritizes
ESTALE over EIO over ENOENT, and otherwise defaults to the latest
error passed having priority.

This change preserves current behavior, but rewrites the logic to
return the higher priority error of the existing and new errno. The
purpose of the change is to make the logic a bit more clear and set
the stage for future changes to make the logic flexible based on
context.

BUG: 892730
Change-Id: Id1aa48855dfb0507abc9d1ef22f2259b30472576
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-on: http://review.gluster.org/4375
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
tests: Fix tests for bug 857330

Changed some tests to use EXPECT, which I should have done originally, instead
of using TEST. With recent changes to the EXPECT primitive we get more info for
failures, which hopefully will make debugging of the intermittent failures of
these tests easier.

Change-Id: If763dcfb7e370a3f360277c3c183b950aede5685
BUG: 857330
Signed-off-by: Kaushal M <kaushal@redhat.com>
Reviewed-on: http://review.gluster.org/4390
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
debug/trace: save the recent fops received in the event-history

* Make use of event-history in debug/trace xlator to dump the recent fops,
when statedump is given. trace xlator saves the fop it received along
with the time in the event-history and upon statedump signal, dumps its
history. The size of the event-history can be given as a xlator option.

* Make changes in trace to take logging into log-file or logging to
history as an option. By default both are off.

Change-Id: I12baee5805c6efb55735cead4e2093fb94d7a6a0
BUG: 797171
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.org/4088
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
nlm: use virtual ip of server to connect to client

In cases where the servers use virtual ip's, this commit
makes sure we use them and not the physical ip.

This change also refactors code around nlm4_establish_callback
by sending granted msg only after a connection establishment,
and removing the separate thread creation.

Change-Id: I087362c547a25aa52ef7fc6653845a3863466ee6
BUG: 888283
Signed-off-by: Rajesh Amaravathi <rajesh@redhat.com>
Reviewed-on: http://review.gluster.org/4326
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
afr: conditionally prioritize EIO errors over ENOENT

The most important errno logic historically only prioritized ESTALE
over ENOENT. Commit c8c0942d added EIO prioritization over ENOENT
to ensure that split-brain was reported when it occurs in
conjunction with bricks missing the file entry. The unintended side
effect of this change is that (non split-brain) EIO errors reported
from the bricks themselves are now reported to the client when the
expectation is that afr should squash said errors in favor of
marking the file inconsistent.

The high-level problem is that EIO is overloaded with different
meanings from different contexts. This commit adds an eio parameter
to the errno priority logic to conditionally flag when EIO is of
higher priority and should be propagated to the client.

BUG: 892730
Change-Id: Ib692a8a1f1737ef190d57894f392ec53ffb33aab
Signed-off-by: Brian Foster <bfoster@redhat.com>
Reviewed-on: http://review.gluster.org/4376
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: replace obsolete /usr/local reference for remote ssh/gsyncd

See https://bugzilla.redhat.com/show_bug.cgi?id=895656
    https://bugzilla.redhat.com/show_bug.cgi?id=764679 (GLUSTER-2947)
    https://bugzilla.redhat.com/show_bug.cgi?id=764623 (GLUSTER-2891)

The comments in the bzs are a bit obtuse and/or vague. As near as I
can make out we had, for a while, a "convenience symlink" to or from
/usr/local/libexec/gsyncd, which no longer exists.

And, lacking any comments in the code, I gather this is some sort of
fallback or failsafe logic: if the first, normal attempt to invoke gsyncd
fails then an attempt is made to ssh to the box and invoke it.

In any event, there's nothing in /usr/local/... so it's unquestionably
wrong to try to invoke anything there.

BUG: 895656
Change-Id: I3b7ac7a049b91ce101b930599294830147cc60ad
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/4392
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Joe Julian <joe.julian.prime@gmail.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/afr: Fail readv on data-split-brain

Problem:
Afr prevents opens on a file in split-brian but the
fd that is already open still has the capability to perform
both reads and writes to the file.

Fix:
Fail readvs on a file with EIO.

Change-Id: I8e07f24c36fab800499b36ab374f984b743332cd
BUG: 873962
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4199
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: log changes in volume stop (and in op sm codepath)

This patch makes log changes mostly in the op state machine as also
in volume stop codepath of glusterd.

Changes made:

* Moved log level from INFO to DEBUG, of log messages on the various
  state transitions within a transaction.
  For example, messages of the following kind:
  a. "Sent op req to <n> peers"
  b. "Received LOCK from uuid: <peer-uuid>", etc.

* Changed some of the log messages to give as much information as
  available in case of failure.

* Added logs to identify on which machine lock/stage/commit failed.

* Quite a few s/THIS/this changes.

Also, with this change, log changes in all other volume ops
should (hopefully) boil down to modifying the respective logs in
handler, stage and commit (and brick ops in some cases).

Change-Id: I2b8443042b07fb41a1d12033741f7e156aa6b3da
BUG: 812356
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-on: http://review.gluster.org/4382
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaushal M <kaushal@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/afr: Disable delayed post op when eager-lock is off

Problem:
When eager-lock is disabled, inodelks for write-fops on same
fd conflict with each other. If eager-lock is disabled but
delayed post-op is enabled then each write fop's inodelk unlock
waits for post-op-delay-secs. So the conflicting write fop
acquires inodelk after post-op-delay-secs. This results in
post-op-delay-secs delay for every write fop on the fd for
sequential writes (Ex: dd).

Fix:
Disable delayed-post-op when eager-lock is off.

Change-Id: I87ea4c8d1c7bb269b9b174388ae50f37e82629b7
BUG: 895235
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4391
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
extras: Don't update RETVAL with invalid value.

Problem:
Start and restart functions update the value of RETVAL internally.
Assigning $? to RETVAL after function return was updating RETVAL with
invalid value.

Fix:
Removed assignment statement after start and restart functions return.

Stop function was always returning 0 as return value due to one extra echo
command. Removed echo line.

Change-Id: Iec1a83a83463f419dd45e12daf4e5ded80568a3a
BUG: 888442
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/4349
Reviewed-by: Anand Avati <avati@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
build: glusterfs-server should own known files under /var/lib/glusterd

In continuation of Change-Id Ib1df8da7e8abdcd3edece45bd39fe238f28838f1
there are more files that should be owned by the glusterfs-server
package.

Note that starting glusterd, doing some peer-probes and creating volumes
will create additional run-time files under /var/lib/glusterd. The
filenames depend on the volumes that are created, and that prevents
packages from knowing (and owning) them in advance. It seems common
practice to leave files untouched under /var/lib/<app>.

It would be okay to remove files generated by the package itself, but we
should not remove files generated by other packages, or files that have
been created by users manually. I do not see a way to detect files
generated by gluster* applications and remove only these. Keeping the
runtime files under /var/lib/glusterd (which do not have predictive
names) is safe and they should not require much space anyway.

BUG: 829734
Change-Id: I2b79682b014fcf62c2a73f076cf0b11b4ba87f4a
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/4361
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>