geo-rep: Fix ValueError - signal only works in main thread

When a worker process not confirmed within 60 seconds of start
then monitor thread was terminated instead of stopping and restarting
the worker thread.
Before terminate monitor thread tries to add a signal handler for
SIGTERM to cleanup the stuff before terminate. Signal handling will
not work inside thread, so ValueError was raised.

This patch will not terminate monitor thread, instead only kills and
restarts the worker.

Change-Id: I14df26c0cc3097af29293c81536c13b86075e28f
BUG: 1078068
Signed-off-by: Aravinda VK <avishwan@redhat.com>
Reviewed-on: http://review.gluster.org/7294
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
geo-rep/glusterd: Fix geo-rep status on introduction of volume lock

        Getting op context in 'glusterd_op_gsync_set' is no longer
        valid as it is expected that 'rsp_dict' sent from caller is
        filled. It was fine till now as no one was setting the op
        context. The introduction of volume locks sets it,
        consequently breaking geo-rep status command. Hence the
        code that gets dict from op context if present is removed.

        Also corrected some indentation issues in 'glusterd_op_gsync_set'

Signed-off-by: Kotresh H R <khiremat@redhat.com>

Change-Id: Ieacd6e6c9be3c92159f849caca2acf5aabca1e32
BUG: 1077697
Signed-off-by: Kotresh H R <khiremat@redhat.com>
Reviewed-on: http://review.gluster.org/7289
Reviewed-by: Avra Sengupta <asengupt@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
build: do not create versioned <xlator>.so files

There has been a misspelled option in the Makefile.am files. The option
is called -avoid-version, and not -avoidversion.

It is not trivial to provide a test-case for this. One way would be to
check generated RPMs with a command like this (output should be empty):

  $ rpm -qlp *.rpm | grep -E '/xlator/.+.so.0'

Change-Id: I2a6cc557eada4d098b73af5a254f8c75707543da
BUG: 1078365
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7299
Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: refactor

- Remove client side self-healing completely (opendir, openfd, lookup)
- Re-work readdir-failover to work reliably in case of NFS
- Remove unused/dead lock recovery code
- Consistently use xdata in both calls and callbacks in all FOPs
- Per-inode event generation, used to force inode ctx refresh
- Implement dirty flag support (in place of pending counts)
- Eliminate inode ctx structure, use read subvol bits + event_generation
- Implement inode ctx refreshing based on event generation
- Provide backward compatibility in transactions
- remove unused variables and functions
- make code more consistent in style and pattern
- regularize and clean up inode-write transaction code
- regularize and clean up dir-write transaction code
- regularize and clean up common FOPs
- reorganize transaction framework code
- skip setting xattrs in pending dict if nothing is pending
- re-write self-healing code using syncops
- re-write simpler self-heal-daemon

Change-Id: I1e4080c9796c8a2815c2dab4be3073f389d614a8
BUG: 1021686
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/6010
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
debug/io-stats:fix compile warning

Compiler Warning:
---------------------------------------------------------
io-stats.c: In function 'io_stats_dump':
io-stats.c:950:24: warning: comparison between 'gf1_cli_stats_op' and
'enum gf1_cli_info_op' [-Wenum-compare]
                 if (op == GF_CLI_INFO_ALL ||
io-stats.c:951:24: warning: comparison between 'gf1_cli_stats_op' and
'enum gf1_cli_info_op' [-Wenum-compare]
                     op == GF_CLI_INFO_CUMULATIVE)
                        ^
---------------------------------------------------------
Fix: Use the appropriate enum in function defintion of io_stats_dump().
Note: Using the same BZ ID as the commit that introduced this argument.

Change-Id: I24e1aaf9ab86b4f337e3daa729d561ec208f2a95
BUG: 1030580
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-on: http://review.gluster.org/7217
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Glusterd/Remove-brick: Reconfigure the nfs server volfile upon remove-brick start

Problem : For remove-brick start operation  all client volfiles are reconfigured
except nfs server volfile. Hence, even after layout is fixed by the rebalance
process, the nfs clients dont see the change and go on creating directories and
files in the decommissioned brick which leads to data loss after remove-brick
commit.

Solution : Reconfigure the nfs server volfile for remove-brick start

credit: kaushal@redhat.com
        spalai@redhat.com

Change-Id: Ib8cd8b45a9e1f888d5e00dff65cdf77c1613a2af
BUG: 1070734
Signed-off-by: Susant Palai <spalai@redhat.com>
Reviewed-on: http://review.gluster.org/7162
Reviewed-by: Kaushal M <kaushal@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
encryption/crypt: Add mem-accounting for crypt xlator

Without these changes crypt.t crashes when compiled with -DDEBUG

Change-Id: I1f7372aa30a09dbe3ae81d1dd598cf36e17fe0b7
BUG: 1030058
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7319
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterfsd: Remove dead code.

Previous cleanup of this function had removed some lines
which had left dead code. Just removing that.

Fix for coverity CID: 1167461 .

Change-Id: I2a34fc407ce0eb4c4ba759c8ce6574a00b37020a
BUG: 789278
Signed-off-by: Raghavendra Talur <rtalur@redhat.com>
Reviewed-on: http://review.gluster.org/6937
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd: persistent client xlator/ afr changelog names

-Add a unique brick-id field to glusterd_brickinfo_t
-Persist the id to the brickinfo file
-Use the brick-id as the client xlator name during vol create, add-brick and
 replace-brick operations.
-For older volumes,generate the id in-memory during glusterd restore but defer
 writing it to the brickinfo file until the next volume set operation.
-send and receive the brick-ids during peer probe.

Feature page:
www.gluster.org/community/documentation/index.php/Features/persistent-AFR-changelog-xattributes

Related patch:
http://review.gluster.org/#/c/7122

Change-Id: Ib7f1570004e33f4144476410eec2b84df4e41448
BUG: 1066778
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-on: http://review.gluster.org/7155
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Reviewed-by: Kaushal M <kaushal@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd: op-version check for brickops.

cluster op-version must be atleast 4 for add/remove brick to proceed.
This change is required for the new afr-changelog xattr changes that
will be done for glusterFS 3.6 (http://review.gluster.org/#/c/7155/).

In add-brick, the check is done only when replica count is increased
because only that will affect the AFR xattrs.

In remove-brick, the check is unconditional failing which there will be
inconsistencies in the client xlator names amongst the volfiles of
different peers.

Change-Id: If981da2f33899aed585ab70bb11c09a093c9d8e6
BUG: 1066778
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-on: http://review.gluster.org/7122
Reviewed-by: Kaushal M <kaushal@redhat.com>
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
features/glupy: Rename Glupy python module to avoid namespace conflict

* Rename gluster.py to glupy.py to avoid namespace conflict (#1018619)
* Move the main Glupy files into glusterfs-extra-xlators rpm
* Move the Glupy Translator examples into glusterfs-devel rpm
* Add Glupy entry to the MAINTAINERS file

BUG: 1018619
Change-Id: I48de598ba5ae8eec0e7e276bbcca1abb0e549cef
Signed-off-by: Justin Clift <justin@gluster.org>
Reviewed-on: http://review.gluster.org/6979
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
features/compress: Add mem accounting support for compress

Change-Id: I89a7a4cd64ef65ad3bab180d66797a62b4e1e195
BUG: 923540
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7320
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/afr: Sparse file self-heal canges

- Fix boundary condition for offset
- Honour data-self-heal-algorithm option
- Added tests for sparse file self-healing

Change-Id: I14bb1c9d04118a3df4072f962fc8f2f197391d95
BUG: 1080707
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7339
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: suppress spurious error message during startup

From glusterd log:
----------
E [glusterd-store.c:1981:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
----------

The message is emitted from glusterd_store_retrieve_volume() when it reads the
volinfo file because it doesn't do anything with the key-value pair. Suppress the
error. The key is needed by glusterd_store_retrieve_bricks() which anyway
re-reads it.

Also change the log level to WARNING since we do not error out if an unknown key
is got while parsing the volinfo file.

Change-Id: Icd7962d9e16e0f90e6a37ee053dcafe97d2cab94
BUG: 1079279
Reviewed-on: http://review.gluster.org/7314
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaushal M <kaushal@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
rpm: Include the hook scripts of add-brick, volume start, stop and set.

Change-Id: I9485da3d9a54a52cc524daff8f4f4caca3a6065d
BUG: 1080970
Signed-off-by: Poornima G <pgurusid@redhat.com>
Reviewed-on: http://review.gluster.org/7346
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
log: enhance gluster log format with message ID and standardize errno reporting

	Currently there are quite a slew of logs in Gluster that do not
	lend themselves to trivial analysis by various tools that help
	collect	and monitor logs, due to the textual nature of the logs.

	This FEAT is to make this better by giving logs message IDs so
	that the tools do not have to do complex log parsing to break
	it down to problem areas and suggest troubleshooting options.

	With this patch, a new set of logging APIs are introduced that
	take additionally a message ID and an error number, so as to
	print the message ID and the descriptive string for the error.

	New APIs:
		- gf_msg, gf_msg_debug/trace, gf_msg_nomem, gf_msg_callingfn

	These APIs follow the functionality of the previous gf_log*
	counterparts, and hence are 1:1 replacements, with the delta
	that, gf_msg, gf_msg_callingfn take additional parameters as
	specified above.

	Defining the log messages:
	Each invocation of gf_msg/gf_msg_callingfn, should provide an ID
	and an errnum (if available). Towards this, a common message id
	file is provided, which contains defines to various messages and
	their respective strings. As other messages are changed to the
	new infrastructure APIs, it is intended that this file is edited
	to add these messages as well.

	Framework enhanced:
	The logging framework is also enhanced to be able to support
	different logging backends in the future. Hence new configuration
	options for logging framework and logging formats are introduced.

	Backward compatibility:
	Currently the framework supports logging in the traditional
	format, with the inclusion of an error string based on the errnum
	passed in. Hence the shift to these new APIs would retain the log
	file names, locations, and format with the exception of an
	additional error string where applicable.

	Testing done:
	Tested the new APIs with different messages in normal code paths
	Tested with configurations set to gluster logs (syslog pending)
	Tested nomem variants, inducing the message in normal code paths
	Tested ident generation for normal code paths (other paths
		pending)
	Tested with sample gfapi program for gfapi messages
	Test code is stripped from the commit

	Pending work (not to be addressed in this patch (future)):
	- Logging framework should be configurable
	- Logging format should be configurable
	- Once all messages move to the new APIs deprecate/delete older
	  APIs to prevent misuse/abuse using the same
	- Repeated log messages should be suppressed (as a configurable
	  option)
	- Logging framework assumes that only one init is possible, but
	  there is no protection around the same (in existing code)
	- gf_log_fini is not invoked anywhere and does very little
	  cleanup (in existing code)
	- DOxygen comments to message id headers for each message

Change-Id: Ia043fda99a1c6cf7817517ef9e279bfcf35dcc24
BUG: 1075611
Signed-off-by: ShyamsundarR <srangana@redhat.com>
Reviewed-on: http://review.gluster.org/6547
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
doc: Update manpages

Change-Id: Id14c7c3229ed266cd15915a2136e3290ce2c5ed2
BUG: 1031328
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/7338
Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
tests: Stale file lookup test

Change-Id: I6edfc5b7ee42677e92d9cff6a7180692d20e9310
BUG: 1080759
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7341
Reviewed-by: Ravishankar N <ravishankar@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
DHT/Rebalance : Hard link Migration Failure

Probelm : __is_file_migratable used to return ENOTSUP
for all the cases. Hence, it will add to the failure
count. And the remove-brick status will show failure
for all the files.

Solution : Added 'ret = -2' to gf_defrag_handle_hardlink to
be deemed as success. Otherwise dht_migrate_file will try to
migrate each of the hard link, which not intended.

Change-Id: Iff74f6634fb64e4b91fc5d016e87ff1290b7a0d6
BUG: 1066798
Signed-off-by: Susant Palai <spalai@redhat.com>
Reviewed-on: http://review.gluster.org/7124
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
tests/afr: select correct read-child for data OPs.

Change-Id: If84bc489b6c45bde3bdb858da5f1600cea78c8a5
BUG: 1080759
Signed-off-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-on: http://review.gluster.org/7345
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>