cluster/dht: Don't do extra unref in dht-migration checks

Problem:
syncop_open used to perform a ref in syncop_open_cbk so the extra
unref was needed but now syncop_open_cbk does not take a ref so no
need to do extra unref.

Fix:
remove the extra fd_unref and let dht_local_wipe do the final unref.

Change-Id: Ibe8f9a678d456a0c7bff175306068b5cd297ecc4
BUG: 961615
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8029
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Tested-by: Joe Julian <joe@julianfamily.org>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
doc: Fix peer probe deficiencies

Added the command to probe the first server to assign it a hostname

Change-Id: I67f35553df74718f23cd8453dc7854a44c61be11
BUG: 1095596
Reviewed-on: http://review.gluster.org/7854
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
fuse: fix memory leak in fuse_getxattr()

The fuse_getxattr() function was not freeing fuse_state_t resulting in a
memory leak. As a result, when continuous writes (run dd command in a loop)
were done from a FUSE mount point, the OOM killer killed the client
process (glusterfs).

Manual backport of: http://review.gluster.org/#/c/5392/, provided
by Martin Svec <martin.svec@zoner.cz>.

BUG: 1112844
Change-Id: Ic723675c53384d48c79ad1b11b21c1b17fb56866
Reviewed-on: http://review.gluster.org/8164
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
performance/quick-read: Check for iobuf and iobref before unref.

In the absence of this check, iobuf_unref() can cause a log flood
for read operations.

Change-Id: Ib8beef6e55b69ecfb566bf61a96215ef0d0d74dc
BUG: 1116514
Signed-off-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8242
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Humble Devassy Chirammal <humble.devassy@gmail.com>
Reviewed-by: Prashanth Pai <ppai@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
nfs: prevent assertion error with MOUNT over UDP

see http://review.gluster.org/#/c/8258/, http://review.gluster.org/8241;
and BZs 1116997, 1116503, respectively.

The MOUNT service over UDP runs in a separate thread. This thread does
not have the correct *THIS xlator set. *THIS points to the global (base)
xlator structure, but GF_CALLOC() requires it to be the NFS-xlator so
that assertions can get validated correctly.

This is solved by passing the NFS-xlator to the pthread function, and
setting the *THIS pointer explicitly in the new thread.

It seems that on occasion (needs further investigation) MOUNT over UDP
does not unregister itself. There can also be issues when the kernel NLM
implementation has been registered at portmap/rpcbind, so adding some
unregister procedures in the cleanup of the test-cases.

Change-Id: Iea3bafd65bf480f2b12da91875c73571594d52d3
BUG: 1117256
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/8262
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
Reviewed-by: Santosh Pradhan <spradhan@redhat.com>
doc/release-nodes:  release notes for 3.4.5 GA

Change-Id: I359c280b67006473a836c90771b28598fe712c80
BUG: 1110456
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/8370
NFS: stripe-xlator should pass EOF at end of READDIR

NFS READDIR replies are made of a header, a sequence of
entries, and a EOF flag. When GlusterFS's NFS server is
used along with stripe xlator, it fails to set the EOF
flag, which violates NFS RFC and confuses some clients.

The bug is caused because nfs xlator sets EOF if it gets
op_errno set to ENOENT. That value is produced in storage
xlator and propagated through server, client, and other
xlators until stripe xlator handles it. stripe only passed
op_errno if op_ret < 0, which is not the case here. This
change set adds a special case for that situation to fix
the problem.

Cherry picked from commit 9b5231e5c98b8cfa116838287c7a14042702795f:
> Change-Id: Ie6db94b0515292387cfb04c1e4a9363f34fcd19a
> BUG: 1130969
> Reported-by: Emmanuel Dreyfus <manu@netbsd.org>
> Signed-off-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-on: http://review.gluster.org/8493
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Santosh Pradhan <spradhan@redhat.com>
> Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
> Reviewed-by: Emmanuel Dreyfus <manu@netbsd.org>
> Tested-by: Emmanuel Dreyfus <manu@netbsd.org>

Change-Id: Ie6db94b0515292387cfb04c1e4a9363f34fcd19a
BUG: 1132392
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/8510
Reviewed-by: Santosh Pradhan <spradhan@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
cluster/afr: Fix memory leak of file-path in self-heal-daemon

        Backport of http://review.gluster.org/4790

Note: Only the part which fixes the memory leak is backported

shd event has path which needs to be freed as part of circular buffer cleanup.
This patch introduces the functionality so that self-heal-daemon can use it.

Change-Id: I3f3823d5587eda2fcb278f0fdb89123a31c9d786
BUG: 1119894
Signed-off-by: Pranith Kumar K <pkarampu@redhat.com>
Reviewed-on: http://review.gluster.org/8541
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Ravishankar N <ravishankar@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
mount/fuse: Handle fd resolution failures

Backport from http://review.gluster.org/#/c/8402, BZ 1126048

Problem:
Even when the fd resolution failed, the fop is continuing on the
new graph which may not have valid inode. This lead to NULL layout
subvols in dht which lead to crash in fsync after graph migration.

Fix:
- Remove resolution error handling in FUSE_FOP as it was only added
  to handle fd migration failures.
- check in fuse_resolve_done for fd resolution failures and fail the
  fop right away.
- loc resolution failures are already handled in the corresponding
  fops.
- Return errno from state->resolve.op_errno in resume functions.
- Send error to fuse on frame allocation failures.
- Removed unused variable state->resolved
- Removed unused macro FUSE_FOP_COOKIE

Change-Id: I3f3129b9b4fc34ff1fe07218e6924a027346a95e
BUG: 1123289
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/8562
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
gNFS: Fix memory leak in setacl code path

Backport of http://review.gluster.org/#/c/8651 (BZ # 1139195) to
release-3.4

If ACL is set on a file in Gluster NFS mount (setfacl command),
and it succeed, then the NFS call state data is leaked. Though
all the failure code path frees up the memory.

Impact: There is a OOM kill i.e. vdsm invoked oom-killer during
rebalance and Killed process 4305, UID 0, (glusterfs nfs process)

FIX:
Make sure to deallocate the memory for call state in acl3_setacl_cbk()
using nfs3_call_state_wipe();

Change-Id: I6664a9151e9417e114c858bb2cdb136d9cb0b283
BUG: 1139250
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/8653
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Santosh Pradhan <spradhan@redhat.com>
Reviewed-by: Niels de Vos <ndevos@redhat.com>
cluster/dht: Fix dict_t leaks in rebalance process' execution path

        Backport of: http://review.gluster.org/8763

Two dict_t objects are leaked for every file migrated in success codepath.
It is the caller's responsibility to unref dict that it gets from calls to
syncop_getxattr(); and rebalance performs two syncop_getxattr()s per file
without freeing them.

Also, syncop_getxattr() on GF_XATTR_LINKINFO_KEY doesn't seem to be using
the response dict. Hence, NULL is now passed as opposed to @dict to
syncop_getxattr().

Change-Id: I59556ee6e135e7e65d4ddd31ba0f39e7eb50b02d
BUG: 1144792
Signed-off-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-on: http://review.gluster.org/8789
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
socket: Fixed parsing RPC records containing multi fragments

In __socket_proto_state_machine(), when parsing RPC records containing
multi fragments, just change the state of parsing process, had not
processed the memory to coalesce the multi fragments.

Cherry picked from commit fb6702b7f8ba19333b7ba4af543d908e3f5e1923:
> Change-Id: I5583e578603bd7290814a5d26885b31759c73115
> BUG: 1139598
> Signed-off-by: Gu Feng <flygoast@126.com>
> Reviewed-on: http://review.gluster.org/8662
> Tested-by: Gluster Build System <jenkins@build.gluster.com>
> Reviewed-by: Niels de Vos <ndevos@redhat.com>
> Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
> Tested-by: Raghavendra G <rgowdapp@redhat.com>

Change-Id: I5583e578603bd7290814a5d26885b31759c73115
BUG: 1146470
Signed-off-by: Niels de Vos <ndevos@redhat.com>
Reviewed-on: http://review.gluster.org/8849
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
glusterfs.spec.in: add psmisc to -server subpackage

apparently some minimalist installs omit psmisc
needed for the killall in various %pre and %post scriptlets

smarter logic for restarting glusterd in %post server

Change-Id: I1581d278c2be6a917c20785437ce78c502111d8d
BUG: 1146624
Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>
Reviewed-on: http://review.gluster.org/8859
Tested-by: Gluster Build System <jenkins@build.gluster.com>
dht/rebalance: Do not allow rebalance when gfid mismatch found

Due to race condition, it may so happen that, gfid obtained
in readdirp and gfid found by lookup are different for a given
name. in that case do no allow the rebalance.

Readdirp of an entry will bring the gfid, which will be stored
in the inode through inode_link, and when lookup is done and
gfid brought by lookup is different from the one stored in the
inode, client3_3_lookup_cbk will return ESATLE and error will be
captured by rebalance process.

Change-Id: Iad839177ef9b80c1dd0e87f3406bcf4cb018e6fa
BUG: 1139984
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Reviewed-on: http://review.gluster.org/7973
Reviewed-by: Shyamsundar Ranganathan <srangana@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Krutika Dhananjay <kdhananj@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8670
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
DHT/readdirp: Directory not shown/healed on mount point if exists on single brick(non first up subvolume).

Problem: If snapshot is taken, when mkdir has succeeded only on
hashed_subvolume, then after restoring snapshot the directory
is not shown on mount point.

Why:    dht_readdirp takes only those directory entries in to
account, which are present on first_up_subvolume. Hence, if the
"hashed subvolume" is not same as first_up_subvolume, it wont be listed
on mount point and also not healed.

Solution:
Case 1: (Rebalance not running)If hashed subvolume is NULL or down then
filter in first_up_subvolume. Other wise the corresponding hashed subvolume
will take care of the directory entry.

Case 2: If readdirp_optimize option is turned on then read from first_up_subvol

Change-Id: Idaad28f1c9f688dbfb1a8a3ab8b244510c02365e
BUG: 1139986
Signed-off-by: Susant Palai <spalai@redhat.com>
Reviewed-on: http://review.gluster.org/7599
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8671
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
Tested-by: Kaleb KEITHLEY <kkeithle@redhat.com>
dht: fix rename race

If two clients try to rename the same file at the same time, we
sometimes end up with *no file at all* in either the old or new
location.  That's kind of bad.  The culprit seems to be some overly
aggressive cleanup code.  AFAICT, based on today's study of the code,
the intent of the changed section is to remove any linkfile we might
have created before the actual rename.  However, what we're removing
might not be our extra link.  If we're racing with another client that's
also doing a rename, it might be the only remaining link to the user's
data.  The solution, which is good enough to pass this test but almost
certainly still not complete, is to be more selective about when we do
this unlink.  Now, we only do it if we know that, at some point, we did
in fact create the link without error (notably ENOENT on the source or
EEXIST on the destination) ourselves.

Change-Id: I8d8cce150b6f8b372c9fb813c90be58d69f8eb7b
BUG: 1139988
Signed-off-by: Jeff Darcy <jdarcy@redhat.com>
Reviewed-on: http://review.gluster.org/8269
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8672
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
DHT/Create : Failing to identify a linkto file in lookup_everywhere_cbk path

In case a file is not found in its cached subvol we proceed with
dht_lookup_everywhere. But as we dont add the linkto xattr to the
dictionary, we fail to identify any linkto file encountered.The
implication being we end up thinking the linkto file as a regular file
and proceed with the fop.

Change-Id: Iab02dc60e84bb1aeab49182f680c0631c33947e2
BUG: 1139992
Signed-off-by: Susant Palai <spalai@redhat.com>
Reviewed-on: http://review.gluster.org/8277
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-on: http://review.gluster.org/8673
Reviewed-by: N Balachandran <nbalacha@redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
cluster/dht: Fix races to avoid deletion of linkto
 file

Explanation of Race between rebalance processes:
https://bugzilla.redhat.com/show_bug.cgi?id=1110694#c4

scenario-1:
===========

STATE 1:                          BRICK-1
only one brick                   Cached File
in the system

STATE 2:
Add brick-2                       BRICK-1                BRICK-2

STATE 3:                                       Lookup of File on brick-2
                                               by this node's rebalance
                                               will fail because hashed
                                               file is not created yet.
                                               So dht_lookup_everywhere is
                                               about to get called.

STATE 4:                         As part of lookup
                                 link file at brick-2
                                 will be created.

STATE 5:                         getxattr to check that
                                 cached file belongs to
                                 this node is done

STATE 6:

                                            dht_lookup_everywhere_cbk detects
                                            the link created by rebalance-1.
                                            It will unlink it.

STATE 7:                        getxattr at the link
                                file with "pathinfo" key
                                will be called will fail
                                as the link file is deleted
                                by rebalance on node-2

Fix:
So in the STATE 6, we should avoid the deletion of link file. Every time
dht_lookup_everywhere gets called, lookup will be performed on all the nodes.
So to avoid STATE 6, if linkto file is found, it is not deleted until valid
case is found in dht_lookup_everywhere_done.

Case 1: if linkto file points to cached node, and cached file exists,
        uwind with success.

Case 2: if linkto does not point to current cached node, and cached file
        exists:
        a) Unlink stale link file
        b) Create new link file

Case 3: Only linkto file exists:
        Delete linkto file

Case 4: Only cached file
        Create link file (Handled event without patch)

Case 5: Neither cached nor hashed file is present
        Return with ENOENT (handled even without patch)

Reviewed-on: http://review.gluster.org/8231

******************************************************************************

scenario-2:
===========
cluster/dht: Modified logic of linkto file deletion on non-hashed

Currently whenever dht_lookup_everywhere gets called, if in
dht_lookup_everywhere_cbk, a linkto file is found on non-hashed
subvolume, file is unlinked. But there are cases when this file
is under migration. Under such condition, we should avoid deletion
of file.

When  some other rebalance process changes the layout of parent
such that dst_file (w.r.t. migration) falls on non-hashed node,
then may be lookup could have found it as linkto file but just
before unlink, file  is under migration or already migrated
In such cased unlink can be avoided.

Race:
-------
If we have two bricks (brick-1 and brick-2) with initial file "a"
under BaseDir which is hashed as well as cached on (brick-1).

Assume "a"  hashing gives 44.

                              Brick-1              Brick-2

Initial Setup:               BaseDir/a             BaseDir
                             [1-50]                [51-100]

Now add new-brick Brick-3.

1. Rebalance-1 on node Node-1 (Brick-1 node) will reset
the BaseDir Layout.

2. After that it will perform
a)  Create linkto file on  new-hashed (brick-2)
b)  Perform file migration.

1.Rebalance-1 Fixes the base-layout:
                 Brick-1             Brick-2           Brick-3
                 ---------         ----------         ------------
                 BaseDir/a            BaseDir           BaseDir
                  [1-33]              [34-66]           [67-100]

2. Only a) is     BaseDir/a          BaseDir/a(linkto)   BaseDir
   performed                         Create linktofile

Now rebalance 2 on node-2 jumped in and it will perform
step 1 and 2-a.

After (rebal-2, step-1), it changes the layout of the BaseDir.
                    BaseDir/a     BaseDir/a(link)    BaseDir
                    [67-100]           [1-33]        [34-66]

For  (rebale-2, step-2), It will perform lookup at Brick-3 as w.r.t new
layout 44 falls for brick-3. But lookup will fail.
So  dht_lookup_everywhere gets called.

NOTE: On brick-2 by rebalance-1, a linkto file was created.

Currently that linkto files gets deleted by rebalance-2 lookup as it
is considered as stale linkto file.  But  with patch if rebalance is
already in progress or rebalance is over,  linkto file will not be
unlinked. If rebalance is in progress fd will be  open and if rebalance
is over then linkto file wont be set.

Reviewed-on: http://review.gluster.org/8345

*******************************************************************************

scenario-3:
===========

cluster/dht: Added keys in dht_lookup_everywhere_done

Case where both cached  (C1)  and hashed file are found,
but hash does not point to above cached node (C1), then
dont unlink if either fd-is-open on hashed or
linkto-xattr is not found.

Reviewed-on: http://review.gluster.org/8429
BUG: 1139995
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Change-Id: I86d0a21d4c0501c45d837101ced4f96d6fedc5b9
Signed-off-by: Venkatesh Somyajulu <vsomyaju@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: susant palai <spalai@redhat.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8674
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
storage/posix: removing deleting entries in case of creation failures

The code is not atomic enough to not to delete a dentry created by a
prallel dentry creation operation.

Change-Id: I9bd6d2aa9e7a1c0688c0a937b02a4b4f56d7aa3d
BUG: 1139988
Signed-off-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-on: http://review.gluster.org/8327
Reviewed-by: Pranith Kumar Karampuri <pkarampu@redhat.com>
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8675
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>
dht: fix rename race

Additional check to check if we created the linkto
file before deleting it in the rename cleanup function

Change-Id: I919cd7cb24f948ba4917eb9cf50d5169bb730a67
BUG: 1139988
Signed-off-by: Nithya Balachandran <nbalacha@redhat.com>
Reviewed-on: http://review.gluster.org/8338
Tested-by: Gluster Build System <jenkins@build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp@redhat.com>
Reviewed-by: Vijay Bellur <vbellur@redhat.com>
Reviewed-on: http://review.gluster.org/8676
Reviewed-by: Kaleb KEITHLEY <kkeithle@redhat.com>