client/rpc: Add ping and msgs stats

This can be seen as below,

># cat $META/graphs/active/vol-client-0/private |grep ping_msgs_sent
ping_msgs_sent = 2

># cat $META/graphs/active/vol-client-0/private |grep "^msgs_sent"
msgs_sent = 13

where $META is /<fuse-mountpt>/.meta

Change-Id: I2107ec2b045bac701377760635e18758adb943a3
Signed-off-by: Krishnan Parthasarathi <kparthas@redhat.com>
Reviewed-on: http://review.gluster.org/8285
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Atin Mukherjee <amukherj@redhat.com>
Reviewed-by: Anand Avati <avati@redhat.com>
Symlink mtime changes when rebalancing

Added mtime preservation for special files during rebalance.

Change-Id: If04921d4d66853fde8b4d8a3ab748790864f8f42
BUG: 1122443
Signed-off-by: Anders Blomdell <anders.blomdell@control.lth.se>
Reviewed-on: http://review.gluster.org/8383
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-by: Anand Avati <avati@redhat.com>
ec: Add -msse2 option to avoid fedora/rawhide compilation errors

This solves a compilation error on some compiler versions.
To avoid compilation issues on non-Intel architectures, patch
http://review.gluster.org/8366/ has already been merged. It
disables ec on those architectures.

These modifications are temporary patches until a proper
solution for bug #1125166 is ready.

Change-Id: I74d0b79e84601cc4f86ad08ce0f8102b99a79a68
BUG: 1125168
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/8395
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/dht: Modified logic of linkto file deletion on non-hashed

Currently whenever dht_lookup_everywhere gets called, if in
dht_lookup_everywhere_cbk, a linkto file is found on non-hashed
subvolume, file is unlinked. But there are cases when this file
is under migration. Under such condition, we should avoid deletion
of file.

When  some other rebalance process changes the layout of parent
such that dst_file (w.r.t. migration) falls on non-hashed node,
then may be lookup could have found it as linkto file but just
before unlink, file  is under migration or already migrated
In such cased unlink can be avoided.

Race:
-------
If we have two bricks (brick-1 and brick-2) with initial file "a"
under BaseDir which is hashed as well as cached on (brick-1).

Assume "a"  hashing gives 44.

                              Brick-1              Brick-2

Initial Setup:               BaseDir/a             BaseDir
                             [1-50]                [51-100]

Now add new-brick Brick-3.

1. Rebalance-1 on node Node-1 (Brick-1 node) will reset
the BaseDir Layout.

2. After that it will perform
a)  Create linkto file on  new-hashed (brick-2)
b)  Perform file migration.

1.Rebalance-1 Fixes the base-layout:
                 Brick-1             Brick-2           Brick-3
                 ---------         ----------         ------------
                 BaseDir/a            BaseDir           BaseDir
                  [1-33]              [34-66]           [67-100]

2. Only a) is     BaseDir/a          BaseDir/a(linkto)   BaseDir
   performed                         Create linktofile

Now rebalance 2 on node-2 jumped in and it will perform
step 1 and 2-a.

After (rebal-2, step-1), it changes the layout of the BaseDir.
                    BaseDir/a     BaseDir/a(link)    BaseDir
                    [67-100]           [1-33]        [34-66]

For  (rebale-2, step-2), It will perform lookup at Brick-3 as w.r.t new
layout 44 falls for brick-3. But lookup will fail.
So  dht_lookup_everywhere gets called.

NOTE: On brick-2 by rebalance-1, a linkto file was created.

Currently that linkto files gets deleted by rebalance-2 lookup as it
is considered as stale linkto file.  But  with patch if rebalance is
already in progress or rebalance is over,  linkto file will not be
unlinked. If rebalance is in progress fd will be  open and if rebalance
is over then linkto file wont be set.

Change-Id: I3fee0d28de3c76197325536a9e30099d2413f079
BUG: 1116150
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/8345
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
doc: mem-pool documentation

Change-Id: I9bc454c92a078de5e3a30372a50c6f2cbdcd6c21
BUG: 1118309
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/8343
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
storage/posix: fix issue in posix_fsync

Fix the issue that posix_fsync does not correctly return and save
 error code in op_errno when call to sys_fdatasync fails.

Change-Id: Id0b62cfa009dbb52c8a0992abd5c46330fa0a8c0
BUG: 1125814
Signed-off-by: Zhang Huan <zhhuan@gmail.com>
Reviewed-on: http://review.gluster.org/8398
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Reviewed-by: Santosh Pradhan <spradhan@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
geo-rep/libgfchangelog: Create working dir during changelog_register if not present.

Earlier, xysnc's register was being called first,
which was creating working directory before calling
changelog_register. Now it is history crawl first.
Hence working directory would not have been created.
Create it in gf_changelog_register itself if it is
not already created.

Change-Id: Ief3f2b87deaf5da16c135b64be1be42e0a7647f3
BUG: 1125843
Signed-off-by: Kotresh H R <khiremat@redhat.com>
Reviewed-on: http://review.gluster.org/8399
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Aravinda VK <avishwan@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Regression test portability: system-specific cleanup

Add NetBSD system-specific cleanup code
Define killall as pkill when it is missing.

BUG: 764655
Change-Id: I3a1cfa3029fd60ede334a670fc115c211700063d
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/8290
Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
Tested-by: Harshavardhana <harsha@harshavardhana.net>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
nfs-ganesha  dumps core when Pynfs tests OPDG1O and OPDG11 are run.

Assgining -1 to ret if fd_create fails. The tests somehow
enter an error case 'if' block (this will be a separate bug fix),
before glfd is created. Need an addiitional check to see
if glfd exists before entering the 'else' block in 'out'.

glfs_h_creat was missing a similar check, adding that in
this bug fix.)

Change-Id: I3b69540c53e8169538848b9e7f23ffc8ef0ac189
BUG: 1121062
Signed-off-by: Meghana Madhusudhan <mmadhusu@redhat.com>
Reviewed-on: http://review.gluster.org/8333
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Reviewed-by: soumya k <skoduri@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
tests: Avoid test bug-884597.t to loop infinitely

If some unexpected error happens while this test is trying to
create a linkto file, an infinite loop is generated and the test
never ends.

This patch limits the number of retries it can do to find that
file.

Change-Id: Ie556c509116109ead99c78d671b2669b4428faac
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/8318
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
ec: Test volume mount point in a better way

An 'ls -a1' on an empty volume seems to return 3 entries instead
of the expected 2 ('.' and '..') in the build servers. I changed
the test to a simple 'stat', which is enough and more reliable.

Change-Id: I12d0f47394ad378b40fc9b86507cdb3543f99970
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/8313
Reviewed-by: Jeff Darcy <jdarcy@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Raghavendra Bhat <raghavendra@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
cluster/ec: Fix incorrect management of NFS requests

Some operations, specially those comming from NFS, do not use a
regular fd and use an anonymous fd (i.e. a previous open call has
not been sent). Any context information created during open or
create will not be present on these fd's, so we simply return NULL
for contexts of those fd.

Also it seems that NFS can send write requests with a very big
buffer (higher that the default value of 128 KB). Some changes
have been made to correctly handle these large buffers.

Change-Id: I281476bd0d2cbaad231822248d6a616fcf5d4003
BUG: 1122417
Signed-off-by: Xavier Hernandez <xhernandez@datalab.es>
Reviewed-on: http://review.gluster.org/8367
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
glusterd/snapshot:  Inherit the mount options of a original brick
when creating snapshots

When creating a snapshot a LVM is created at the backend and is mounted
under /var/run/gluster/snaps/... However, this mount does not inherit
the mount options for the original brick acting as the parent for the
snap.

If the snap is restored, this could lead to performance degredations,
functional limitations, or in extreme scenarios even potential data
loss.

Change-Id: I67d70fd83430d83dacc5380c6c928e27fb9c9e1b
BUG: 1125180
Signed-off-by: Vijaikumar M <vmallika@redhat.com>
Reviewed-on: http://review.gluster.org/8394
Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
Tested-by: Krishnan Parthasarathi <kparthas@redhat.com>
glusterd: Bump max op version

Change-Id: Ic66a0a751233ebbcb65d13f7e3265a046fae9a0b
BUG: 1125431
Signed-off-by: Atin Mukherjee <amukherj@redhat.com>
Reviewed-on: http://review.gluster.org/8397
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Kaushal M <kaushal@redhat.com>
Do not call rpc_transport_unref() on NULL trans

rpc_clnt_disable() sets rpc->conn->trans to NULL, hence we should not
call rpc_transport_unref() afterwards. I moved it before the
rpc_clnt_disable() call, but I am not sure it should be called at all,
perhaps it should just go away.

BUG: 764655
Change-Id: I488d0207494e3a3fad52e64e67b2e740b236b864
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Reviewed-on: http://review.gluster.org/8393
Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Tested-by: Raghavendra G <rgowdapp@redhat.com>
geo-rep/libgfchangelog: Support of symlinks while creation of working dir.

In gf_changelog_register, enable symlink support while creating
working directory if its not already created.

Change-Id: I4b81762d04f5276a65087d2ff94f2a36dc504b58
BUG: 1125843
Signed-off-by: Kotresh H R <khiremat@redhat.com>
Reviewed-on: http://review.gluster.org/8409
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Venky Shankar <vshankar@redhat.com>
feature/geo-rep: Keep marker.tstamp's mtime unchangeable during snapshot.

Problem:

Geo-replicatoin does a full xsync crawl after snapshot
restoration of slave and master. It does not do history crawl.

Analysis:

Marker creates 'marker.tstamp' file when geo-rep is started
for the first time. The virtual extended attribute
'trusted.glusterfs.volume-mark' is maintained and whenever
it is queried on gluster mount point, marker fills it on
the fly and returns the combination of uuid, ctime of
marker.tstamp and others. So ctime of marker.tstamp, in other
sense 'volume-mark'  marks the geo-rep start time when the
session is freshly created.

From the above, after the first filesystem crawl(xsync) is
done during first geo-rep start, stime should always be less
than 'volume-mark'. So whenever stime is less than volume-mark,
it does full filesystem crawl (xsync).

Root Cause:

When snapshot is restored, marker.tstamp file is freshly
created losing the timestamps, it was originally created with.

Solution:

1. Change is made to depend on mtime instead of ctime.
2. mtime and atime of marker.tstamp is restored back when
   snapshot is created and restored.

Change-Id: I4891b112f4aedc50cfae402832c50c5145807d7a
BUG: 1125918
Signed-off-by: Kotresh H R <khiremat@redhat.com>
Reviewed-on: http://review.gluster.org/8401
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Regression test portability: arequal-checksum

Building arequal-checksum on non Linux systems requires a few adjustments:
- use __builtin_alloca() on all platforms
- on systems without argp, get it from contrib/standalone-argp
  (this requires adding cflags support to the build_tester function)
- FTW_ACTIONRETVAL option to nftw(3) is GNU libc specific, work around
  if it does not exist.
- md5sum is Linux-specific. Use md5 -n on NetBSD and md5 -q on FreeBSD/Darwin
- Use 'cc' as synonymous for all compilers, it can behave as gcc/clang depending
  on which is default
- cleanup tabs/whitespaces

BUG: 764655
Change-Id: I9090c17da596fbf00fc1fbd7593163ce8cd3b84c
Signed-off-by: Emmanuel Dreyfus <manu@netbsd.org>
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/8283
Tested-by: Gluster Build System <jenkins@build.gluster.com>
socket: add boundary checks for iobuf_get2 over rpc_hdr_bytes

A malformed packet can cause a OOM while peforming iobuf_get2()
on a large enough packet side.

Such a scenario is observed when running vulnerability tests,
it would look like one of those tests perhaps based on DDOS (Denial
of Service) attacks hand-crafts a RPC packet which is of a large
enough size - since we do not verify the size and do not provide
any boundary checks, there are secenarios where it leads to OOM.

Reproducible consistently with those tests has revealed that we
should be ideally be adding a boundary check.

Limit such an allocation to a 1gigabyte, as it might be sufficient
to handle for all variety of RPC packets.

Change-Id: I5f1411dd96d6f167993d28a1718ffef2fb4e9923
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/8384
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Tested-by: Raghavendra G <rgowdapp@redhat.com>
gluster vol set nfs-ganesha.enable off fails to unexport volume.

 Due to a parameter type change in upstream nfs-ganesha
( v2.1 and above ), gluster vol set <volname> nfs-ganesha.enable
 off doesn't unexport the volume in question if it was already
exported.
A minor change in the hook script would result in the
expected behaviour.

Change-Id: I46f76f3cde6b6ebd7a94d5ee16f05a3648a2840f
BUG: 1124711
Signed-off-by: Meghana Madhusudhan <mmadhusu@redhat.com>
Reviewed-on: http://review.gluster.org/8390
Reviewed-by: soumya k <skoduri@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>