cli: Increased timeout for gluster volume heal commnads

Problem:
If number of files are very large, then gluster volume heal
volumename info commnads take large time. So timeout of 2
minutes seems to be insufficient.

Fix:
Increased timeout to 10 minutes

Change-Id: I5f847163e01c4afbb587b726833ad80183f1a928
BUG: 986945
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/5372
Reviewed-by: Kaushal M <kaushal@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
tests: Avoid timing delays caused by fsync

Change-Id: I2da53dd14d7352bdc878fce5a9019a955e150ce9
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5377
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
storage/posix: implement batched fsync in a single thread

Because of the extra fsync()s issued by AFR transaction, they
could potentially "clog" all the io-threads denying unrelated
operations from making progress.

This patch assigns a dedicated thread to issues fsyncs, as
an experimental feature to understand performance characteristics
with the approach.

As a basis, incoming individual fsync requests are grouped into
batches, falling in the same @batch-fsync-delay-usec window of
time. These windows can extend in practice, as processing of
the previous batch can take longer than @batch-fsync-delay-usec
while new requests are getting batched.

The feature support three modes (similar to the -S modes of fs_mark)

- syncfs: In this mode one syncfs() is issued per batch, instead
  of N fsync()s (one per file.)

- syncfs-single-fsync: In this mode one syncfs() is issued per
  batch (which, on Linux, guarantees the completion of write-out
  of dirty pages in the filesystem up to that point) and one single
  fsync() to synchronize or flush the controller/drive cache. This
  corresponds to -S 2 of fsmark.

- syncfs-reverse-fsync: In this mode, one syncfs() is issued per
  batch, and all the open files in that batch are fsync()'ed in
  the reverse order of the queue. This corresponds to -S 4 of
  fsmark.

- reverse-fsync: In this mode, no syncfs() is issued and all the
  files in the batch are fsync()'ed in the reverse order. This
  corresponds to -S 3 of fsmark.

Change-Id: Ia1e170a810c780c8d80e02cf910accc4170c4cd4
BUG: 927146
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/4746
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/dht: mark linkfile creation/deletion as internal fop

Currently dht creates/deletes linkfiles for various ops like
rename/linking and when layout changes. dht_linkfile_create
already sends a key GLUSTERFS_INTERNAL_FOP_KEY in dict to
identify this as an internal fop and not user based op.

Enhancing rename related links/unlinks to send this key too.
Marker/changelog or other xlators can now identify these as
internal fops and handle them accordingly

Change-Id: Ib1ca789e6dbce48703c55ad3f4f88f7cd2df3d06
BUG: 987428
Signed-off-by: shishir gowda <sgowda@redhat.com>
Reviewed-on: http://review.gluster.org/5335
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli: gluster volume heal commands are more elaborative

1. "gluster volume heal volume-name"
output :Launching heal operation to perform index self heal on volume volume-name has been successful

2. "gluster volume heal volume-name full"
Output :Launching heal operation to perform full self heal on volume volume-name has been successful

3. "gluster volume heal volume-name info"
Output :Gathering list of entries to be healed on volume volume-name has been successful 

4. "gluster volume heal volume-name info healed"
Output :Gathering list of healed entries on volume volume-name has been successful 

5. "gluster volume heal volume-name info split-brain"
Output :Gathering list of split brain entries on volume volume-name has been successful 

6. "gluster volume heal volume-name info heal-failed"    
Output :Gathering list of heal failed entries on volume volume-name has been successful 

Change-Id: I74c90e8129d23d513ddb7879358a9d21c94a5c0d
BUG: 978936
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/5286
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
storage/posix: Fix conditional compiling for syncfs

Change-Id: Ief22e1c0f2b5074060752d70da41ae93f1028d62
BUG: 927146
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/5381
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
features/marker: pass xdata in marker_unlink()

Change-Id: Ia310af96b25f29351f3adc4bbc97aea271df7673
BUG: 987747
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Reviewed-on: http://review.gluster.org/5379
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cli :remove-brick process output leads to ambiguity

The output of remove-brick  status as "Not started" leads to
ambiguity.We should not show the status of the Server nodes
which do not participate in the remove-brick process.

Change-Id: I85fea40deb15f3e2dd5487d881f48c9aff7221de
BUG: 986896
Signed-off-by: susant <spalai@redhat.com>
Reviewed-on: http://review.gluster.org/5383
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
core: increase the auxillary group limit to 65536

Make the allocation of groups dynamic and increase the limit
to 65536.

Change-Id: I702364ff460e3a982e44ccbcb3e337cac9c2df51
BUG: 953694
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/5111
Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
write-behind: preserve error returned as-is

Change-Id: Ib766403774c1323e0bbddafedeaa47e7fa3a59fa
Signed-off-by: Amar Tumballi <amarts@redhat.com>
BUG: 987415
Reviewed-on: http://review.gluster.org/5296
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Anand Avati <avati@redhat.com>
glusterfsd: is_mgmt_rpc_reconnect should not be `char` but gf_boolean_t

Change-Id: I35c4dc040ebb5e926c7f54d823d1c81b1228bd4f
BUG: 763046
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/5345
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
features/changelog: fixes when enabling changelog

Other enhancements being:
      * ignore fops made by rebalance
      * ignore internally triggered fops

BUG: 987734
Change-Id: I7dd164ae3c209fdb8ec43a27e67b8846f937c93b
Signed-off-by: Venky Shankar <vshankar@redhat.com>
Reviewed-on: http://review.gluster.org/5380
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
doc: Create a features folder.

Moved rdma and bd documents to doc/features. Added a new
document on rebalance.

Change-Id: I04269202adc9605754fc29876433c88480b822a3
BUG: 811311
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/5395
Reviewed-by: Anand Avati <avati@redhat.com>
Tested-by: Anand Avati <avati@redhat.com>
build: Add '--without bd' to disable 'bd' xlator rpmbuild

Change-Id: I9f1774788362d8250693002318b617cd16e1a668
BUG: 950083
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/5397
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Bala FA <barumuga@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
cluster/afr: Handle REPLICATE_TRASH_DIR from old bricks

Change-Id: Ib99f79d3fa607c818dbc62006516480f598d8add
BUG: 886998
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/4640
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
gsyncd: distribute the crawling load

* also consume changelog for change detection.
* Status fixes
* Use new libgfchangelog done API
* process (and sync) one changelog at a time

Change-Id: I24891615bb762e0741b1819ddfdef8802326cb16
BUG: 847839
Original Author: Csaba Henk <csaba@redhat.com>
Original Author: Aravinda VK <avishwan@redhat.com>
Original Author: Venky Shankar <vshankar@redhat.com>
Original Author: Amar Tumballi <amarts@redhat.com>
Original Author: Avra Sengupta <asengupt@redhat.com>
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/5131
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
glusterd/cli changes for distributed geo-rep

Commands:
gluster system:: execute gsec_create
gluster volume geo-rep <master> <slave-url> create [push-pem] [force]
gluster volume geo-rep <master> <slave-url> start [force]
gluster volume geo-rep <master> <slave-url> stop [force]
gluster volume geo-rep <master> <slave-url> delete
gluster volume geo-rep <master> <slave-url> config
gluster volume geo-rep <master> <slave-url> status

The geo-replication is distributed. The session will be created, and
gsyncd will be spawned on all relevant nodes, instead of only one
node.

geo-rep: Collecting status detail related data

Added persistent store for saving information about
TotalFilesSynced, TotalSyncTime, TotalBytesSynced

Changes in the status information in socket:
Existing(Ex):
FilesSynced=2;BytesSynced=2507;Uptime=00:26:01;

New(Ex):
FilesSynced=2;BytesSynced=2507;Uptime=00:26:01;SyncTime=0.69978;
TotalSyncTime=2.890044;TotalFilesSynced=6;TotalBytesSynced=143640;

Persistent details stored in
/var/lib/glusterd/geo-replication/${mastervol}/${eSlave}-detail.status

Change-Id: I1db7fc13ffca2e415c05200b0109b1254067f111
BUG: 847839
Original Author: Avra Sengupta <asengupt@redhat.com>
Original Author: Venky Shankar <vshankar@redhat.com>
Original Author: Aravinda VK <avishwan@redhat.com>
Original Author: Amar Tumballi <amarts@redhat.com>
Original Author: Csaba Henk <csaba@redhat.com>
Signed-off-by: Avra Sengupta <asengupt@redhat.com>
Reviewed-on: http://review.gluster.org/5132
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Vijay Bellur <vbellur@redhat.com>
fuse-bridge: Cure some unhealthy format strings

Mostly sign errors:

Before:

0-glusterfs-fuse: 74: CREATE() /FILE => 0x1e9de9c (ino=-7793743383831318294)
0-glusterfs-fuse: 76: SETATTR() /FILE => -7793743383831318294
...

After:

0-glusterfs-fuse: 109: CREATE() /FILE => 0x267fe9c (ino=12597207469970374167)
0-glusterfs-fuse: 111: SETATTR() /FILE => 12597207469970374167
...

Change-Id: I422d2397fe071b0ac155b8c7e8a40a3365d6d58c
Signed-off-by: Lubomir Rintel <lubo.rintel@gooddata.com>
Reviewed-on: http://review.gluster.org/5402
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
mgmt/glusterd: let each brick write the valgrind o/p to different file

Till now all the brick processes were writing the valgrind information
to the same log file.

Change-Id: I0251c943935e2901b729c71f21d0677edb9f6867
BUG: 922877
Signed-off-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-on: http://review.gluster.org/5394
Reviewed-by: Amar Tumballi <amarts@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
tests: fix test script to turn on write-behind

Change-Id: I8a3ddc8183355236ff7725229441e27bbf8188e3
BUG: 927146
Signed-off-by: Anand Avati <avati@redhat.com>
Reviewed-on: http://review.gluster.org/5406
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>