tests: Gather only files which need to built

Previous code would rather provide a filter but never look
for newly "Added" files leading to failures in subsequent
build.

This patch fixes such issue by actually verifying pedantically
for only files which need to be tested.

Change-Id: Ia716acf82ee60f8ffe5e36257f1cc866c6062718
BUG: 904005
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/6016
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
gfapi.py: support dynamic loading of versioned libraries

Currently gfapi.py only loads libraries by filename ending in ".so".  On
an installed system without development packages, the <lib>.so filenames
are not available. ctypes.util.find_library() can be used to detect the
files dynamically.

In addition to this, also fixing some minor indention errors and package
the library into the Python site-packages path. Python applications and
libraries can now access libgfapi through 'from glusterfs import gfapi'.

Change-Id: I71e38dabd3ade5dcf24813bf2fc25cda91b571c6
BUG: 1005146
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/5835
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
protocol/server: eliminate an unnecessary frame create and destroy

connection_cleanup creates a frame solely so that it can be copied.

But the process of copying creates a new frame, and there's nothing
apparently magic about the one being copied; we can eliminate an
unnecessary create and destroy.

Which in the grand scheme of things probably isn't the worst thing
we do, but it's low hanging fruit.

Change-Id: I4a23b84a53e086137b7d4167ad8c20b673d1ffe5
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/6019
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cli: skipped tag in xml output of rebalance/remove-brick status

Skipped files count is available in CLI output of rebalance status
and remove-brick status, but not available in xml output.

Example output:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volRebalance>
    <op>3</op>
    <nodeCount>1</nodeCount>
    <node>
      <nodeName>localhost</nodeName>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <skipped>0</skipped>
      <status>0</status>
      <statusStr>completed</statusStr>
    </node>
    <aggregate>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <skipped>0</skipped>
      <status>0</status>
      <statusStr>completed</statusStr>
    </aggregate>
  </volRebalance>
</cliOutput>

BUG: 1012772
Change-Id: I05191293403e66e0d681f0cd0422aa3c78a2d91d
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/6000
Reviewed-by: Kaushal M <kaushal@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cli: runtime in xml output of rebalance/remove-brick status

"runtime in secs" is available in the CLI output of
rebalance status and remove-brick status, but not available
in xml output when --xml is passed.

runtime in aggregate section will be max of all nodes runtimes.

Example output:
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volRebalance>
    <op>3</op>
    <nodeCount>1</nodeCount>
    <node>
      <nodeName>localhost</nodeName>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <skipped>0</skipped>
      <runtime>1.00</runtime>
      <status>3</status>
      <statusStr>completed</statusStr>
    </node>
    <aggregate>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <skipped>0</skipped>
      <runtime>1.00</runtime>
      <status>3</status>
      <statusStr>completed</statusStr>
    </aggregate>
  </volRebalance>
</cliOutput>

BUG: 1012773
Change-Id: I8deaba08922a53cd2d3b411e097a7b3cf591b127
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/5997
Reviewed-by: Kaushal M <kaushal@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
glusterd: Validating invalid log-level under geo-rep config options.

Change-Id: I8ff6b48ef41fd6e9ea68c42dfb9878f8a08ed627
BUG: 1010874
Signed-off-by: Ajeet Jha <ajha@redhat.com>
Reviewed-on: http://review.gluster.org/5989
Reviewed-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
geo-rep: logrotate: Logrotate handling

In existing georep logrotate was implemented by handling SIGSTOP
and SIGCONT, gsyncd was failing to start again after SIGSTOP.
New approach uses WatchedFileHandler in logging, which tracks the
log file changes or logrotate. Reopens the log file if logrotate is
triggered or if same log file is updated from other process.

As per python doc:
http://docs.python.org/2/library/logging.handlers.html:
The WatchedFileHandler class, located in the logging.handlers module,
is a FileHandler which watches the file it is logging to. If the file
changes, it is closed and reopened using the file name.

A file change can happen because of usage of programs such as newsyslog
and logrotate which perform log file rotation. This handler, intended
for use under Unix/Linux, watches the file to see if it has changed
since the last emit. (A file is deemed to have changed if its device
or inode have changed.) If the file has changed, the old file stream
is closed, and the file opened to get a new stream.

Change-Id: I30f65eb1e9778b12943d6e43b60a50344a7885c6
BUG: 1012776
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/5968
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/afr: Change Self-heal domain separator to ':'

'-' can be present in a volume. This may lead to domain
collisions in future.

Tests:
Checked in gdb that domain comes with ':' separator:

Breakpoint 1, pl_common_inodelk (frame=0x7fdabcce51a4,
this=0x8bde20, volume=0x8b50d0 "r2-replicate-0:self-heal",
inode=0x7fdab822f0e8, cmd=6, flock=0x7fdabc76eee4,
loc=0x7fdabc76ede4, fd=0x0, xdata=0x7fdabc6e0ab0) at inodelk.c:597

Change-Id: I4456ae35ac8bf21e6361c34e9ad437f744a2e84b
BUG: 993981
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/6025
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Tests: Enable fore-ground self-heal

Change-Id: Ibfca8ddb7c663d44ed447be13b2eabb7bd393bb3
BUG: 993981
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/6028
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterfsd, libgfapi: destroy the temporary graphs constructed for comparison

* The new and the oldgraphs which have been constructed whenever there is
  a volfile change (either reconfigure of the existing graph or creating
  a new graph) for comparison should be freed. Otherwise frequent graph
  changes will lead to huge memory leak

Change-Id: I4faddb1aa9393b34cd2de6732e537a60f600026a
BUG: 948178
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.org/5388
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cli: add node uuid in rebalance and remove brick status xml output

This patch adds node uuid in rebalance/remove-brick status xml output.
Output XML will look like

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cliOutput>
  <opRet>0</opRet>
  <opErrno>0</opErrno>
  <opErrstr/>
  <volRebalance>
    <op>3</op>
    <nodeCount>1</nodeCount>
    <node>
      <nodeName>localhost</nodeName>
 ==>> <id>883626f8-4d29-4d02-8c5d-c9f48c5b2445</id>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <status>3</status>
      <statusStr>completed</statusStr>
    </node>
    <aggregate>
      <files>0</files>
      <size>0</size>
      <lookups>0</lookups>
      <failures>0</failures>
      <status>3</status>
      <statusStr>completed</statusStr>
    </aggregate>
  </volRebalance>
</cliOutput>

Change-Id: I5a1d4f9043b33b9e88150647a243ddb16154e843
BUG: 1012296
Signed-off-by: Bala.FA <barumuga@redhat.com>
Reviewed-on: http://review.gluster.org/6005
Reviewed-by: Kaushal M <kaushal@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: disable client side self-healing in opendir()

The purpose of opendir() self-healing was to recreate files
which go missing after a crash/reboot (because the file was
created just before the crash).

This refactor addresses crash related healing problems in
the self-heal daemon and therefore eliminates the need for
opendir() self-healing in the client side.

Change-Id: I880e2896429b3ebf3669956040c3c14860494152
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: remove remaining traces of readdir-failover

The feature was removed long ago, but there are still traces left.
Remove all traces.

Change-Id: Icbee156b9db2111b7e6c37ab3e9a0a62e1721272
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: disable open fd self-healing on client side

Change-Id: I9cbef8005e7627f9f34d770b50932a0c32f43fab
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: remove self-healing from lookup()

In AFR lookup() we will only find out which of the copies is good
for reading and save that in the inode ctx.

Change-Id: I33bef58362fb1f0163b7202cf4ad52dc928c3f54
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: move self-heal specific functions to afr-self-heal-common.c

No code change, just re-organize.

Change-Id: Ieddda28c8919a8aac5f5ebfd5d5f6448deea92f6
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: revert (temporarily) fsync based durability fixes

Change-Id: Ifb09d6e69150d335b1c314217738b34d533aaa43
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: remove unused lock recovery code

Lock recovery now happens at protocol/client level. Remove all unused
recovery code in AFR.

Change-Id: Ic1ce01ee17dd1a094b8ca70d5d15319aa1c654f1
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: consistently use xdata sent from up in all transaction fops

This xdata will be loaded with dirty marker and post-op in future patches

Change-Id: Ie21c1b38b20eb0b663d60658c2d24e78d17d90eb
Signed-off-by: Anand Avati <avati@redhat.com>
cluster/afr: keep count of possible consistency changing events

Later, use this to recalculate freshness/staleness per inode.

Change-Id: If2fd882fdd14fcdbec4ee767840264478b67d4ad
Signed-off-by: Anand Avati <avati@redhat.com>