rpcsvc: Ignore INODELK/ENTRYLK/LK for throttling

Problem:
When iozone is in progress, number of blocking inodelks sometimes becomes greater
than the threshold number of rpc requests allowed for that client
(RPCSVC_DEFAULT_OUTSTANDING_RPC_LIMIT). Subsequent requests from that client
will not be read until all the outstanding requests are processed and replied
to. But because no more requests are read from that client, unlocks on the
already granted locks will never come thus the number of outstanding requests
would never come down. This leads to a ping-timeout on the client.

Fix:
Do not account INODELK/ENTRYLK/LK for throttling

BUG: 1089470
Change-Id: I9cc2c259d2462159cea913d95f98a565acb8e0c0
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7570
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
features/locks: Remove stale entrylk objects from 'blocked_locks' list

        Backport of http://review.gluster.org/7560

* In the event of a DISCONNECT from a client, as part of cleanup,
  entrylk objects are not removed from the blocked_locks list before
  being unref'd and freed, causing the brick process to crash at
  some point when the (now) stale object is accessed again in the list.

* Also during cleanup, it is pointless to try and grant lock to a
  previously blocked entrylk (say L1) as part of releasing another
  conflicting lock (L2), (which is a side-effect of L1 not being
  deleted from blocked_locks list before grant_blocked_entry_locks()
  in cleanup) if L1 is also associated with the DISCONNECTing client.
  This patch fixes the problem.

Change-Id: Ie077f8eeb61c5505f047a8fdaac67db32e5d4270
BUG: 1089470
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-on: http://review.gluster.org/7576
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
cluster/afr: Unable to self heal symbolic links

Problem:
Under the entry self heal, readlink is done at the
source and sink. When readlink is done at the sink,
because link is not present at the sink, afr expects
ENOENT. AFR translator takes decisions for new link
creation based on ENOENT but server translator is modified
to return ESTALE because of which afr xlator is not able
to heal.

Fix:
The check for inode absence at server includes ESTALE as
well.

Change-Id: I9218da214ed44f7219570ad9dae298d6b5cbded9
BUG: 1046624
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/6600
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
fuse: prevent READDIR(P) from writing to much data to /dev/fuse

In an environment with mixed architectures (32-bit servers, 64-bit
client), it is possible that the on-wire Reply on a READDIR(P) procedure
contains more direntries than the client can fit in the maximum size
that the fuse-request indicated.

A direntry is a dynamically sized structure, because the structure
contains the name of the entry. The client sends a maximum size in the
READDIR(P) Call to the server, and the server uses this size to limit
the number of direntries to return. In case the server can pack more
direntries in the requested maximum size (due to alignment differences
between the architectures), it can happen that the client unpacks the
list of direntries into a buffer that exceeds the maximum size that was
given in the initial fuse-request.

This change introduces a check for the maximum requested size of the
fuse-response in fuse_readdir_cbk() and fuse_readdirp_cbk(). When the
conversion from gluster-direntries to the fuse-direntry format takes
place, the maximum size is checked, and the 'extra' direntries are
discarded. The next readdir()/getdents() that is done, will fetch the
just discarded direntries again.

In addition to this bugfix, some extra logging in send_fuse_iov() and
send_fuse_data() has been added to help diagnosing similar issues.

This change combines two commits from master:
> Commit: 20e317011af7c0f075819bf0648b225f6dc42350
> Change-Id: If2eecfcdf9c248f3820035601446d2c89ff9d1a1
> BUG: 1074023
> Signed-off-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-on: http://review.gluster.org/7278
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>
> Reviewed-by: Anand Avati <avati@redhat.com>
>
> Commit: 76ab97169f63da78c9e83daf040d7b09766497cf
> Change-Id: Ic62d9d729a86a1a6a53ed1354fce153bac01d860
> BUG: 1074023
> Reported-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
> Signed-off-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-on: http://review.gluster.org/7547
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Anand Avati <avati@redhat.com>

Change-Id: Id847db26d861a9805041ff3367e848da7be05545
BUG: 1089934
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7522
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Xavier Hernandez <xhernandez@datalab.es>
write-behind: track filesize when doing extending writes

A program that calls mmap() on a newly created sparse file, may receive
a SIGBUS signal. If SIGBUS is not handled, a segmentation fault will
occur and the program will exit.

A bug in the write-behind translator can cause the creation of a sparse
file created with open(), seek(), write() to be cached. The last write()
may not be sent to the server, until write-behind deems this necessary.

* open(.., O_TRUNC, ...)/creat() the file, it is 0 bytes big
* seek() into the file, use offset 31
* write() 1 byte to the file
* the range from byte 0-30 are unwritten so called 'sparse'

The following illustration tries to capture this:

    Legend:
    [ = start of file
    _ = unallocated/unwritten bytes
    # = allocated bytes in the file
    ] = end of file

    [_______________#]
     |              |
     '- byte 0      '- byte 31

Without this change, reading from byte 0-30 will return an error, and
reading the same area through an mmap()'d pointer will trigger a SIGBUS.
Reading from this range did not trigger the outstanding write() to be
flushed. The brick that receives the read() (translated over the network
from mmap()) does not know that the file has been extended, and returns
-EINVAL. This error gets transported back from the brick to the
glusterfs-fuse client, and translated by the Linux kernel/VFS into
SIGBUS triggered by mmap().

In order to solve this, a new attribute to the wb_inode structure is
introduced; the current size of the file. All FOPs that can modify the
size, are expected to update wb_inode->size. This makes it possible for
extending writes with an offset bigger than EOF to mark the unwritten
area as modified/pending.

Cherry picked from commit b0515e2a4a08b657ef7e9715fb8c6222c700e78c:
> Change-Id: If5ba6646732e6be26568541ea9b12852a5d0b988
> BUG: 1058663
> Signed-off-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-on: http://review.gluster.org/6835
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
> Reviewed-by: Anand Avati <avati@redhat.com>

Change-Id: Ic73288202c6cb37d8c066628e9c489c6fade49bd
BUG: 1071191
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7166
Tested-by: Gluster Build System <jenkins@build.gluster.com>
posix: if brick-uid or brick-gid is not specified, do not set

Current code would set owner uid/gid explicitly to 0/0 on start
even if none was specified. Fix it.

Cherry picked from commit 8bdc329:
> Change-Id: I72dec9e79c51bd1eb3af5334c42b7c23b01d0258
> BUG: 1040275
> Signed-off-by: Anand Avati <avati@redhat.com>
> Reviewed-on: http://review.gluster.org/6476
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Tested-by: Lukáš Bezdička <lukas.bezdicka@gooddata.com>
> Reviewed-by: Krishnan Parthasarathi <kparthas@redhat.com>
> Reviewed-by: Vijay Bellur <vbellur@redhat.com>

Change-Id: Ie0396c1a4e6e0979ea9c855d33db963544a75c42
BUG: 1095971
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7720
Tested-by: Gluster Build System <jenkins@build.gluster.com>
gfapi: glfs_set_volfile_server() now entertains multiple calls

Backport from - http://review.gluster.org/7317

Previous API:

glfs_set_volfile_server (..., const char *host, ...) - single call

New API's:

glfs_set_volfile_server (..., const char *host1, ...)
glfs_set_volfile_server (..., const char *host2, ...)

Multiple calls to this function with different volfile servers,
port or transport-type would create a list of volfile servers
which would be polled during `volfile_fetch_attempts()`

glfs_unset_volfile_server (..., const char *host, ...) to remove
a server from the list (this is provided for future usage)

>> Change-Id: I313efbd3efbd0214e2a71465f33195788df406cc
>> BUG: 986429
>> Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
>> Reviewed-on: http://review.gluster.org/7317
>> Tested-by: Gluster Build System <jenkins@build.gluster.com>
>> Reviewed-by: Niels de Vos <ndevos@redhat.com>
>> Reviewed-by: Anand Avati <avati@redhat.com>

Change-Id: If0020f6b8775bdbe987563247c83d59a2d3744ad
BUG: 986429
Signed-off-by: Harshavardhana <harsha@harshavardhana.net>
Reviewed-on: http://review.gluster.org/7384
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
cluster/afr: Fix bugs in quorum implementation

- Have common place to perform quorum fop wind check
- Check if fop succeeded in a way that matches quorum
  to avoid marking changelog in split-brain.

Change-Id: I663072ece0e1de6e1ee9fccb03e1b6c968793bc5
BUG: 1066996
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/7513
Reviewed-by: Ravishankar N <ravishankar@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
features/marker-quota: more stringent error handling in rename.

If an error occurs and op_errno is not set to non-zero value, we can
end up in loosing a frame resulting in a hung syscall. This patch
adds code setting op_errno appropriately in storage/posix and
makes marker to set err to a default non-zero value in case of
op_errno being zero.

Change-Id: Idc2c3e843b932709a69b32ba67deb284547168f2
BUG: 833586
Signed-off-by: Raghavendra G <raghavendra@gluster.com>
Reviewed-on: http://review.gluster.org/6913
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
build: glusterfs.spec.in, minor/nit changes to sync with Fedora spec

sync with Fedora spec

Change-Id: I48a52522624e38d9df0ab98071d873533f1756a8
BUG: 1091392
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/7564
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Removed the old and unused non-Glupy Python bindings code

Cherry picked from commit 0a8abcc5d0b727746a2ac6c0d5729700bfea742b:
> BUG: 1064096
> Change-Id: Ifd34f46f0f0b45bbc27d35953e6ecbca36753886
> Signed-off-by: Justin Clift <jclift@redhat.com>
> Reviewed-on: http://review.gluster.org/6980
> Reviewed-by: Jeff Darcy <jdarcy@redhat.com>

Change-Id: I330dca1d53cb85ffc9ee823afa6bca056f34c824
BUG: 1064096
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7731
Tested-by: Gluster Build System <jenkins@build.gluster.com>
doc: Adding user documentation for WORM feature

Cherry picked from commit 03813a6ec08405c815b993551d16f07075efd467:
> Change-Id: Ic31c8305254dd00eae986fbb322cac3761b5d330
> BUG: 1086760
> Signed-off-by: Kasturi Narra <knarra@redhat.com>
> Reviewed-on: http://review.gluster.org/7530
> Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
> Reviewed-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-by: Lalatendu Mohanty <lmohanty@redhat.com>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>

Change-Id: I5897ad47ae995ec55418dca0308dd97f00e1a1e0
BUG: 1086760
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/7732
libgfapi:  Added support to fetch volume info from glusterd and store in glfs object.

Defined new APIs in the libgfapi module, given a glfs object,
 * to send handshake RPC call to glusterd process to fetch UUID of the volume
 * store it in the glusterfs_context linked to the glfs object.
 * to parse UUID from its cannonical string format into 16-byte array
   before sending it to the libgfapi users.

Defined a RPC call in glusterd which can be used to query volume related
info by other processes using 'clnt_handshake_procs'.

Note - Currently this RPC call to glusterd process is used only to fetch UUID.
But it can be extended to get other volume related structures as well.

In addition to the above, defined a new variable to keep track of such handshake
RPCs still in progress to make sure all the corresponding RPC callbacks have been
processed before libgfapi returns the glfs object initialized.

Also bumping up the GFAPI current version number since there is a new API
"glfs_get_volume_id" defined and exposed by libgfapi as part of these changes.

Cherry picked from commit 5adb10b9ac1c634334f29732e062b12d747ae8c5:
> Change-Id: I303f76d7177d32d25bdb301b1dbcf5cd73f42807
> BUG: 1095775
> Signed-off-by: Soumya Koduri <skoduri@redhat.com>
> Reviewed-on: http://review.gluster.org/7218
> Reviewed-by: Anand Avati <avati@redhat.com>
> Reviewed-by: Harshavardhana <harsha@harshavardhana.net>
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Vijay Bellur <vbellur@redhat.com>

This change differs a little from the patch in the master branch:
- libgfapi in 3.5 does not have glfs_get_volfile(), so there were some
  merge conflicts resolved,
- libgfapi in 3.5 is not versioned, the configure.ac changes related to
  the versioning have been skipped,
- in the master branch only the XDR .x files are available, release-3.5
  requires a manual re-generation of the relates .h and .c files.

Change-Id: I52c32d0e69a52a7f4285f74164bca6fd83c4f3b3
BUG: 1095775
Signed-off-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-on: http://review.gluster.org/7741
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Tested-by: Niels de Vos <ndevos@redhat.com>