LU-2522 ptlrpc: add more request flags in log

Add request::rq_receiving_reply and ::rq_must_unlink status in its
log message to help debug the issue.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I5dee322231d15ad50466d7d1244b312549cd2b94
Patch-Set: 1, modified by Morrone
LU-3188 osc: shorten IO calling path

By using osc_io_unplug_aync() for osc_queue_sync_pages() to shorten
the IO calling path, to reduce the chance of stack overflow.

This is revive of git commit 83ae17df2bdce837e62473aec27c03d67312c8ea.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2ac32866f7adbc4701370704612c849a18a5d1ac
Reviewed-on: http://review.whamcloud.com/10292
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Patch-Set: 5
LU-5039 mdd: do orphan cleanup asynchronously

Orphan cleanup shouldn't block mount process, it can be done
asynchronously.

backport from: http://review.whamcloud.com/#/c/10584/

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I1f4e178206e5055fd77c6b884ab9859f421e74e9
Patch-Set: 1
LU-793 ptlrpc: allow client to reconnect with RPC in progress

Since not letting clients with pending RPCs to reconnect caused a lot
of problems, this patch allows such a reconnection and would only
refuse to process requests with the same xids that are already being
processed.

Bulk requests are aborted upon reconnection by comparing connection
count of request and export.

Lustre-commit: da6a28fc59a70226dd89bac95880f7791f1db075
Lustre-change: http://review.whamcloud.com/4960

also includes related change:
Lustre-commit: 50efebe70d503cf0b51be539447de71ad982787e
Lustre-change: http://review.whamcloud.com/8842

Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Reviewed-by: Alexander Boyko <alexander_boyko@xyratex.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I4242db9fee88f4adbe23a593c3331bf7433da9a6
Patch-Set: 3
LU-4349 target: count disconnected exports as stale

If export is disconnected during recovery then stale client
counter need to be increased.

Lustre-commit: dd9b9c821c52c6912157a9a6a3f31e354fed9c55
Lustre-change: http://review.whamcloud.com/8785

Test-Parameters: envdefinitions=SLOW=yes testlist=conf-sanity,conf-sanity,conf-sanity,conf-sanity,conf-sanity
Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Change-Id: I7aa448a10f21680cd899a07574c8f0d4b1641b63
LU-4345 osp: store valid bits in setattr record

Backport a series of 4 patches to resolve the random ID on OST
objects problem.

1: store valid bits in setattr record (LU-4345)

We'd store LA_UID/LA_GID bit along with the UID/GID in the setattr
llog record, otherwise, osp could set a random uid/gid to the OST
object.

Lustre-commit: 80f90fcde73e2faff8b7b0ffc7c19bc52982e027
Lustre-change: http://review.whamcloud.com/10223

2: Correctly check for invalid setattr record (LU-5188)

Patch for LU-4345 (commit 80f90fcde73e2faff8b7b0ffc7c19bc52982e027 )
has a correct comment about lsr_valid member being either 0 or
having UID and GID fields set, but the check has a typo causing
it to check for lsr_valid to be both 0 and have the bits set which
is impossible.

The osp_sync_new_setattr_job() should return 0 for invalid record,
so that sync thread can continue processing on other records.

Lustre-commit: 79dd530f1352e6b57fcb870a1e0f2c2a05a0648d
Lustre-change: http://review.whamcloud.com/10706

3: don't skip attr_set for osp objects (LU-5296)

The lsr_valid handling in osp_sync_add_rec() was got a problem in
commit 80f90fcde73e because it stored all of the passed attr flags
in struct llog_setattr64_rec, even though there are only fields for
storing the UID and GID.  Since the time stamps do not need to be
propagated to the OSTs (the MDT values are good enough), this is
not a problem.  Also, osp_sync_add_rec stored LA_UID/LA_GID flags
on disk, but they are not part of lustre_idl.h or lustre_disk.h and
are not guaranteed to be constant over time.  Instead, use the flags
OBD_MD_FLUID/OBD_MD_FLGID that are in lustre_idl.h.

Lustre-commit: 2d5a5e81660adbcd05f14a10bc6c246d6f108d10
Lustre-change: http://review.whamcloud.com/10989

4: return 1 if osp_sync_xxx_job issue RPC (LU-5188)

Return 1 if osp_sync_new_xxx_job() issue RPC, so
sp_sync_process_record() can decrease opd_syn_rpc_in_flight
and opd_syn_rpc_in_progress correctly if RPC is not
being sent, otherwise the opd_sync_thread will not be
stopped, caused LBUG (see LU-5244)

Lustre-commit: 73f47cdda42305df0be57e0ea6252eb6fd36db55
Lustre-change: http://review.whamcloud.com/10828

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Id05510b4d6fbf011fcdface8dba71aa3f2e963e6
Patch-Set: 7
LU-4584 mdt: ensure orig lock is found in hash upon resend

Look for 1st lock found in hash for remote_hdl but that's not
new_lock itself.
There should only be 2 upon resent, new_lock and the 1st/orig
one we want ...

Also add a ref on it again like when just created!

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: If2b9287ef2d919ac46d8bf3a237b7040947db96b
Patch-Set: 7
LU-4054 llog: don't leak llog handle in llog_cat_process_cb()

An early return from llog_cat_process_cb() was leaking the llog
handle. Fix this by not doing that.

Lustre-commit: 26c55b24d8745f3384370c8dcf1c1f302e19805d
Lustre-change: http://review.whamcloud.com/7847

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Ia060afa4c6e38187bd7d2f5dccadbe642f1c497d
LU-4778 llog: update catlog when plain log destroyed

When the empty plain log is destroyed while processing
catlog, we should call llog_cat_cleanup() to close the
plain log and update in catlog.

Lustre-commit: 6760f383f6cd0964e70250723b9d24bce13b5318
Lustre-change: http://review.whamcloud.com/9702

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: Id7b85df28f8b2da404e74d5074b866dab39c8ebe
LU-5038 llog: do not scan for empty llogs at initialization

in some cases the catalog can contain lots of non-empty llogs
(e.g. robinhood isn't running all the time and this lead to
many alive plain llogs) and scanning this in the context of
mount can take a lot. instead we can do this during normal
llog processing.

Lustre-commit: dc910ede715c4f13cde0c64d74e7badaa8dcb07b
Lustre-change: http://review.whamcloud.com/10308

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I0418659e14230c746559d61b411de17a2896e3b0
Revert "LU-5150 mdt: filter out the bad ACL"

This reverts commit 87e8fb828c2acd9fbba92be219896bd870cfdc53.
LU-3660 acl: support mount option "noacl" for zfs

zfs parse mount option in zpl layer, but osd-zfs calls dmu
directly, since ldiskfs parse "noacl" at mount time, osd-zfs
should do the same in osd_mount().

Similar to ldiskfs, osd-zfs should check whether ACL is enabled
for ACL operations, if not -EOPNOTSUPP is returned.

add sanity test 103b for "noacl" mount option.

Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I654db02facb64d51151ed0d7ee1cd7a0ba41904a
LU-5150 acl: 'cp -rp' set empty default ACL

'cp -rp' will set an empty default ACL on target, MDD should treat
this as removing specified ACL.

add sanity 103c to verify this.

Signed-off-by: Lai Siayo <lai.siyao@intel.com>
Change-Id: Ia0b10959d4f187c5f4c0669a9e2456e979eb6256
Patch-Set: 3
LU-5434 mdd: disregard empty POSIX ACLs

Some files may have a POSIX ACL consisting of only a struct
posix_acl_xattr_header, resulting in unexpected permission errors.
Update mdd_check_acl() to return -EAGAIN when such ACLs are
encountered so the caller falls back to standard UNIX
permissions.

These empty ACLs originate from tools like cp. In some cases (e.g.
'cp -rp') just POSIX_ACL_XATTR_VERSION is given as the value of the
system.posix_acl_default extended attribute.  Lustre accepts and
stores this value on disk, and the bad default ACL then propagates
from directories to newly created directory entries.  Separate
patches will prevent the storage of these bad ACLs.  This patch
mitigates problems caused by the ones already stored.

Other minor updates:
- Use the kernel function posix_acl_xattr_count() to count the
  number of ACL entries.
- Convert remainder of mdd_check_acl() to TAB indentation.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: Ib4f7b3874696b7645b9de3f1f10ef45dc4105646
Reviewed-on: http://review.whamcloud.com/11300
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>