glusterd: cut out a standalone socket path calculation routine

Change-Id: If5f196c9154ea59e37b83d3e4cad445fee6e9d45
BUG: 826512
Signed-off-by: Csaba Henk <csaba@redhat.com>
Reviewed-on: http://review.gluster.com/3490
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Pranith Kumar Karampuri <pranithk@gluster.com>
glusterd-hooks: added support for separate namespace for 'volume set' keys

The keys in the above mentioned namespace could be used by hook
scripts to peform tasks on 'special' keys as defined by the storage
admin. The choice of the key and its semantics of it are outside
the scope of glusterd. It is the responsibility of the storage admin
to keep the meaning of the key(s) consistent.

If a user gives a command like

 'gluster volume set <VOLNAME> user.for-this-key do-this"

scripts would get 'user.for-this-key=do-this' as argument.

Change-Id: I5509e17d99e4ddd8bf5df968dcd51ff9a80dc3ab
Signed-off-by: Amar Tumballi <amarts@redhat.com>
BUG: 825902
Reviewed-on: http://review.gluster.com/3443
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krishnan Parthasarathi <kp@gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
storage/posix: Move landfill inside .glusterfs

Change-Id: Ia2944f891dd62e72f3c79678c3a1fed389854a90
BUG: 811970
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/3158
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
storage/posix: Prevent gfid handle leaks

The case which can lead to gfid handle leaks:
Self-heal removes directory '/d' with 10 files in it, in brick b1.
This dir is renamed to <landfill>/<hashval of '<brick-path>/d'>
by posix. Before the janitor thread could remove the directory,
self-heal could remove another directory with same path '/d'.
Then again the rename to same path is done by posix as before.
The gfid-handles of the old '/d', 10 files in it are not unlinked.

To prevent such problems, rename the directory to be removed to
<landfill>/<gfid-str>.

Change-Id: Iad13708e1ebcc5222b64c058aa9a2d372e1bfa5b
BUG: 811970
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/3159
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
mgmt/glusterd: Do shd validation for replicate volumes

Staging needs to build graphs for replicate volumes in stopped
state as well.
Change-Id: I6474cd0fc43c9fa1916826d4a452f301fe7fe811
BUG: 823128
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/3489
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Csaba Henk <csaba@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
distribute: support user-specified layouts.

The new type is DHT_HASH_TYPE_DM_USER=1 (on disk in network byte order) and
we treat it the same as DHT_HASH_TYPE_DM except that we don't stomp on it
during rebalance.

Change-Id: I893571a9b89577acdea2fe868915b18d3663fd77
BUG: 807312
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.com/3004
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
replicate: add hashed read-child method.

Both the first-to-respond method and the round-robin method are susceptible
to clients repeatedly choosing the same servers across a series of opens,
creating hot spots.  Also, the code to handle a replica being down will
ignore both methods and just choose the first remaining (which is not an
issue for two-way but can be otherwise).  The hashed method more reliably
avoids such hot spots.  There are three values/modes.

0: use the old (broken) methods.

1: select a read-child based on a hash of the file's GFID, so all clients
   will choose the same subvolume for a file (ensuring maximum consistency)
   but will distribute load for a set of files.

2: select a read-child based on a hash of the file's GFID plus the client's
   PID, so different children will distribute load even for one file.

Mode 2 will probably be optimal for most cases.  Using response time when we
open the file is problematic, both because a single sample might not have
been representative even then and because load might have shifted in the
hours or days since (for long-lived files).  Trying to use more current load
information can lead to "herd following" behavior which is just as bad.
Pseudo-random distribution is likely to be the best we can reasonably do,
just as it is for DHT.

Change-Id: I798c2760411eacf32e82a85f03bb7b08a4a49461
BUG: 802513
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.com/2926
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/distribute: set the inode layout in readdirp_cbk() for files

with this, inode-linking it in readdirp_cbk will be neater.

Change-Id: Ie2cd646438f851e1755e9b6a3fc9898059bee359
Signed-off-by: Amar Tumballi <amar@gluster.com>
BUG: 816140
Reviewed-on: http://review.gluster.com/2717
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/dht: set conf->defrag to NULL after freeing the defrag structure

Also no need to free the xlator object after rebalance is over, as the process
is about to be killed.

Change-Id: I6973e43c0353b5de61c0b39e52a22c618be361f4
BUG: 826584
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.com/3495
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
mgmt/glusterd: self-heals should be on in glustershd

BUG: 825740
Change-Id: I44829fb985f9c394b1e240e8ee7f8d026593add9
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/3481
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
protocol/client: do not access the local object after being freed

Change-Id: I2d3aeb084168b9ed68a670b91e09126917f82968
BUG: 826588
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.com/3494
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Optimize for small dicts, and avoid an overrun.

As dicts get used more and more in the I/O path (especially for xattrs and
the new xdata feature), removing some of their inherent inefficiency
becomes more important.  This patch addresses some of the issues around
allocating data_pair_t structures separately.  Along the way, I found that
the way we're allocating the "members" hash table was subtly wrong, and
could lead to a memory overrun.  This is a latent bug because nobody uses
dict_get_new_full that way, but I added an assert to guard against that
possibility.  One beneficial side effect is that we now save four pointers'
worth of space per dict, offsetting the extra space used for the new
members.

Change-Id: Ie8c3e49f1e584daec4b0d2d8ce9dafbc76fb57b2
BUG: 827448
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.com/3040
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
protocol/server: do not wind opendir call if fd creation fails

If resolve fails in some fd based operation, then do not use fd to get
gfid (fd might be NULL). Use the gfid present in resolve structure.

Change-Id: I1058274a2f9b4e58a76e4e6019e7c5ce1906d365
BUG: 827376
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.com/3504
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Add server-side aux-GID resolution.

Change-Id: I09bcbfa41c7c31894ae35f24086bef2d90035ccc
BUG: 827457
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.com/3241
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
nfs/nlm: when setting nlmclnt->rpc_clnt, do not overwrite old rpc_clnt

Change-Id: I01a1c0c0c8d3402b8fe061258001eea2c0029e83
BUG: 819518
Signed-off-by: Krishna Srinivas <ksriniva@redhat.com>
Reviewed-on: http://review.gluster.com/3419
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
core: xlator option framework cleanups

- Upon init/reconf, if an option is not in the opt dict,
  and a default value is neither specified, null it out.
  With this, the xlator config data that comes
  out of init/reconf becomes deterministic in terms
  of the xlator option declarations and the incoming
  option value dictionary. (Needed for correct operation
  of volume reset.)

- We can rely bravely on the guarantee given by init/reconf
  as of which no NULL value is passed to the converter
  functions. Drop the spurious null check of not_null(),
  and rebaptize it to pass().

Change-Id: Ifa068bcc0275456c01ed00a3a315a985eb262e49
BUG: 765147
Signed-off-by: Csaba Henk <csaba@redhat.com>
Reviewed-on: http://review.gluster.com/3505
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: makefile typo fix.

instead of /var/lib/glusterd, symlink was pointing to /var/log/glusterd

Change-Id: I485ad8d6cc8535378179621dea7539328d22454c
Signed-off-by: Amar Tumballi <amarts@redhat.com>
BUG: 824231
Reviewed-on: http://review.gluster.com/3503
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: regenerate brick vol-files on upgrade

If upgrade/downgrade option is set in glusterd it terminates
after the volfiles are regenerated. No need for 'sleep 10'
hack anymore.

BUG: 825872
Change-Id: I12e666eb871aad7e7efa954b9307993952745d92
Signed-off-by: Pranith Kumar K <pranithk@gluster.com>
Reviewed-on: http://review.gluster.com/3482
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterd: fix issues with volume reset handling

- properly resolve shortened key names
- make sure user gets decent feedback

Change-Id: I94b75f34b29cb71fb1a2edf17c3f1bf841bb552a
Signed-off-by: Csaba Henk <csaba@redhat.com>
BUG: 826958
Reviewed-on: http://review.gluster.com/3500
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
protocol/client: provide a buffer for storing reply of readlink.

since a readlink response can be bigger than size of rdma-msges
that can be inlined, we need to provide a buffer where server
can do an rdma-write of response.

Change-Id: I6ab06c3a94702f810ab0c57b409aaaf35cc93057
BUG: 822337
Signed-off-by: Raghavendra G <raghavendra@gluster.com>
Reviewed-on: http://review.gluster.com/3464
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>