LU-5428 libcfs: false alarm of libcfs watchdog

lc_watchdog_disable will not delete kernel timer, benefit of this
is no overhead of del_timer in each serivce loop, however, there
is a corner case that lc_watchdog_touch can race with lcw_cb when
service thread is actually idle:

1. service thread armed a timer before processing request, after
   it handled request, it called lc_watchdog_disable() and set
   lc_watchdog as disabled, but kernel timer is still outstanding.
2. service thread slept for many seconds because there is no
   request to handle
3. it's waken up by incoming request
4. it called lc_watchdog_touch and set watchdog status to enabled.
   now because timer is still alive, if timer is expired right
   before cfs_timer_arm(), then lcw_cb will set a disabled watchdog
   as "expired", which is wrong.
5. the next call of lc_watchdog_disable will complain for expired
   watchdog.

There are two options to fix this issue:
1. always del_timer in lc_watchdog_disable, however, it may increase
   system overhead when service threads are busy.
2. set watchdog as "enabled" after cfs_timer_arm, so there is no
   window for lcw_cb to see false status of watchdog.

This patch chooses the second approach to fix the problem

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: Iac24082e2d63de8330285cf243ed585da6524ab9
Reviewed-on: http://review.whamcloud.com/11415
Tested-by: Jenkins
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1279 kernel: Fix concurrent module loading deadlocks

Concurrently starting multiple OSTs on a single OSS frequently
triggers 30s deadlocks on module_mutex.  This RHEL 6 kernel bug
applies to any module that results in additional request_module()
calls in its init callback.  In Lustre, at least ptlrpc and libcfs are
affected.  (RHEL 7 should have enough fixes in this area, but testing
needs to be done.)  This patch adds a fix adapted from a number of
upstream commits to the RHEL 6 kernel patch series.

Change-Id: Ibdd384fd7622a0b4fcbf4cb5fdb864de87fcc25e
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/11229
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5299 obdclass: avoid race during Server device start

Handle concurrent starts for same device (multiple mounts, ...).
But allows for separate nosvc and nomgs case.
Also add a specific test of concurent MDT/OST start with an
artificial delay to verify.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I442819a5b865ed3e98477f9d2602efc4d09d7860
Reviewed-on: http://review.whamcloud.com/11139
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5195: hsm: delete HSM records not found on disk

After a MDS crash, it is possible the file containing an hsm record is
not present anymore. When the MDS restarts, we get traces like this:

(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000:
    failed to process HSM_ACTIONS llog (rc=-2)
(mdt_hsm_cdt_actions.c:104:cdt_llog_process())
    Skipped 600 previous similar messages
(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000:
    error opening log id 0x1c:1:0: rc = -2
(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages
(llog_cat.c:556:llog_cat_process_cb()) tas01-MDD0000:
    cannot find handle for llog 0x1c:1: -2
(llog_cat.c:556:llog_cat_process_cb()) Skipped 600 previous similar messages
(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000:
    failed to process HSM_ACTIONS llog (rc=-2)
(mdt_hsm_cdt_actions.c:104:cdt_llog_process())
    Skipped 600 previous similar messages
(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000:
    error opening log id 0x1c:1:0: rc = -2
(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages

No HSM operation can happen, and the only way to clean is to unmount
the MDT and delete hsm_actions.

If the record can't be found, let the MDS delete it instead.

This is a backport to b2_5.

Lustre-change: http://review.whamcloud.com/11419
Lustre-commit: 3a83b4b93373db46a4ee60a7388775fa0be2eb9a

Change-Id: I61f625d4c18750c8044909ff56d53042cf0b6d86
Signed-off-by: frank zago <fzago@cray.com>
Reviewed-on: http://review.whamcloud.com/11419
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
(cherry picked from commit 3a83b4b93373db46a4ee60a7388775fa0be2eb9a)
Reviewed-on: http://review.whamcloud.com/11619
Reviewed-by: Patrick Farrell <paf@cray.com>
LU-4086 tests: strengthen sanity-hsm/test_33

Instead of waiting+checking both Restore and Cancel HSM operations
sequentially, wait for both to be finished and then check each
results. This will allow to handle conditions where Restore
terminates before the Cancel is started.

Also added to wait_all_done() the capability to only work on
requests for a specific fid.

Lustre-change: http://review.whamcloud.com/9600
Lustre-commit: adc6a5e76348d9bb1c8355103f419405a14decca

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I480ce074513d6829ecb3078c273bac1fdffa6256
Reviewed-on: http://review.whamcloud.com/9600
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Aurelien Degremont <aurelien.degremont@cea.fr>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
(cherry picked from commit adc6a5e76348d9bb1c8355103f419405a14decca)
Reviewed-on: http://review.whamcloud.com/10712
LU-5385: HSM: do not call the JSON log function if no log is open

llapi_hsm_log_ct_registration() and llapi_hsm_log_ct_progress() are
very expensive (fid2path+allocations). Do not let them do anything
if llapi_hsm_write_json_event() is going to discard the JSON record.

Make llapi_hsm_log_ct_registration() and llapi_hsm_log_ct_progress()
static too.

This is a backport to b2_5.

Lustre-change: http://review.whamcloud.com/11164
Lustre-commit: 35d122fd99c961b0b9fd500b3a78bdc0f5334a97

Signed-off-by: frank zago <fzago@cray.com>
Change-Id: Ib10023968a1bd021694bca6338c0a962f58da19a
Reviewed-on: http://review.whamcloud.com/11164
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Reviewed-by: Patrick Farrell <paf@cray.com>
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
(cherry picked from commit 35d122fd99c961b0b9fd500b3a78bdc0f5334a97)
Reviewed-on: http://review.whamcloud.com/11617
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5051 hsm: do not reallocate hsm receive buffer

Instead of allocating 1MB for every call to
llapi_hsm_copytool_recv, allocate that buffer once in
llapi_hsm_copytool_register. The application must not call
llapi_hsm_copytool_recv until it has cleared the data from
the previous call, which lhsmtool_posix already does. This
also make llapi_hsm_action_list_free unnecessary.

Lustre-change: http://review.whamcloud.com/10299
Lustre-commit: 76dca841869d87df2b625cb9d186d531c406fc9b

Signed-off-by: Patrick Farrell <paf@cray.com>
Signed-off-by: Frank Zago <fzago@cray.com>
Change-Id: Ie9e15a5cb4e3ba6d3fe23d40ee2bb47330280abf
Reviewed-on: http://review.whamcloud.com/10299
Tested-by: Jenkins
Reviewed-by: Faccini Bruno <bruno.faccini@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
(cherry picked from commit 76dca841869d87df2b625cb9d186d531c406fc9b)
Reviewed-on: http://review.whamcloud.com/10716
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2827 ldlm: wrong handling of ldlm resends

If a successful reply is lost for an intent
lock request, MDS will not correctly recover
from this situation on resend. Instead, a new
lock is created.

Lustre-commit: I9a23c13313d5168d0c232fdc0acbdab330089356
Lustre-change: http://review.whamcloud.com/5978

This patch also adds handling of resend during layout-lock

Lustre-commit: Ideb551779a884d77ded8036c513f62557b2dfcde
Lustre-change: http://review.whamcloud.com/#/c/10378

Change-Id: I6745461b2a1370837be7bcbff12dc021cd1021cf
Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5266 ldlm: granting the same lock twice on recovery

ASSERTION( lock->l_granted_mode != lock->l_req_mode ) is hit
when resending LDLM_FL_REPLAY.

Don't add lock to waiting list as it is already added or granted.

Lustre-commit : 63851b5816bb30687fbf3750380d6b448e9400f1
Lustre-change: http://review.whamcloud.com/10903

Change-Id: Ib8e5d2c7588f6cacd1723529e70d29f63742caad
Xyratex-bug-id: MRP-1944
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Reviewed-on: http://review.whamcloud.com/10903
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
LU-5496 ldlm: reconstruct proper flags on enqueue resend

otherwise, waiting lock may get granted as no BLOCKED_GRANTED
flag is returned

Lustre-commit: 175ce732b881e99d77d3f5c11fd1d0d5f604218a
Lustre-change: http://review.whamcloud.com/11644

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-1944
Change-Id: I5e938ff0454d5e8694b09f9fff3c1f82d746360d
Reviewed-on: http://review.whamcloud.com/11644
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5530 mdt: Properly match open lock and unlock

It seems that when request resend with a corresponding lock match is
in play (thanks to large striping + llnl patch for the client to send
small requests only), after all the suffering and fixing coming from
LU-2827, here is another casualty in in open/lease locking.
mdt_reint_open() and mdt_open_by_fid_lock() might match the lock
on resend and not call mdt_object_open_lock(), yet call
mdt_object_open_unlock() before exit.
Since mdt_object_open_lock/unlock also plays with a semaphore,
hilarity ensues usually most visible as a rw_sem lockup
in mdt_object_open_lock.

This patch adds tracking whenever we actually called mdt_object_open_lock
or not and only calls mdt_object_open_unlock if we did.

Change-Id: I73c529229acec98cac4ad73f7b487e759ad9a763
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5579 ldlm: re-sent enqueue vs lock destroy race

upon lock enqueue re-send, lock is pinned by ldlm_handle_enqueue0,
however it may race with client eviction or even lcok cancel (if
a reply for the original RPC finally reached the client) and the
lock cann be found by cookie anymore:

 ASSERTION( lock != NULL ) failed: Invalid lock handle

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Change-Id: I9d8156bf78a1b83ac22ffaa1148feb43bef37b1a
Xyratex-bug-id: MRP-2094
LU-4349 target: count disconnected exports as stale

If export is disconnected during recovery then stale client
counter need to be increased.

Test-Parameters: envdefinitions=SLOW=yes testlist=conf-sanity,conf-sanity,conf-sanity,conf-sanity,conf-sanity
Signed-off-by: Mikhail Pershin <mike.pershin@intel.com>
Change-Id: Id4f2a668dbe8d5cd2fa3d392b7df7b4d67a4265a
Reviewed-on: http://review.whamcloud.com/8785
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5164 osd: Limit lu_object cache

For OSDs like ZFS to perform optimally it's import that they be allowed
to manage their own cache.  This maximizes the likelyhood that the ARC
will prefetch and cache the right buffers.  In the existing ZFS OSD
code a cached LU object pins buffers in the ARC preventing them from
being dropped.  As the LU cache grows it can consume the entire ARC
preventing buffers for other objects, such as the OIs, from being
cached and severely impacting the performance for FID lookups.

By default this patch will only limit the LU cache for ZFS OSDs.

NOTES:

* Setting LU_CACHE_NR_ZFS_LIMIT to 0 results in an LBUG on the MDS.
  This may be because an object is being used without a reference.
  Setting a minimum value of 256 was arbitrary, ideally we would
  set this value to 0.

* In order to be able to quickly determine the number of objects in
  the hash table the CFS_HASH_COUNTER flag is added.  This adds an
  atomic_inc/dec to the hash insert/remove paths but is not expected
  to have any measurable impact of performance.

This patch is back-ported from the following one:
Lustre-commit: 0123baecc4e2050447f8c4f48f5b33a6d3c524a8
Lustre-change: http://review.whamcloud.com/10237

Test-Parameters: alwaysuploadlogs \
mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs \
testgroup=review-zfs

Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Change-Id: If573f02175a02abeeb100ed74f3109e629d4c392
Reviewed-on: http://review.whamcloud.com/12001
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5240 obdclass: Increase minimum LU Cache for ZFS

If the lu cache is too small threads will thrash for data including
the OI tables and other important metadata.

This patch is back-ported from the following one:
Lustre-commit: a9c51c03eaecb5234ba754b738877323a35bed51
Lustre-change: http://review.whamcloud.com/10802

Test-Parameters: alwaysuploadlogs \
mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs \
testgroup=review-zfs

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ia6c85f523219706b5d41054c869e73ed5bf3c2f5
Reviewed-on: http://review.whamcloud.com/12002
Tested-by: Jenkins
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3188 osc: shorten IO calling path

By using osc_io_unplug_aync() for osc_queue_sync_pages() to shorten
the IO calling path, to reduce the chance of stack overflow.

This is revive of git commit 83ae17df2bdce837e62473aec27c03d67312c8ea.

Lustre-commit: 5bc6be85c7b5434917791f97a199b16f6c8bf533
Lustre-change: http://review.whamcloud.com/10292

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: I2ac32866f7adbc4701370704612c849a18a5d1ac
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/11612
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4624 llite: Avoid statahead thread start/stop deadlocks

The statahead and statahead agl threads blindly set their
thread state to SVC_RUNNING without checking the state first.  If, for
instance, another thread sets the state to SVC_STOPPING that
stop signal will now have been lost.  Deadlock ensues.

We also partly improve the sai reference counting, because a race exists
where the ll_stop_statahead thread can drop the default reference, and
the statahead thread can exit and drop its reference as well.  With no
references on the sai, the final put will poison and free the buffer.  The
original do_statahead_enter() function may then continue to access
the buffer after it is freed because it did not take a reference of its
own.  We add a local reference to address that.

Lustre-commit: 373d937086e5cede6fc1a3b3f859622c0173c1de
Lustre-change: http://review.whamcloud.com/9358

Change-Id: I531f17966d49ba1e6ebd99abe9cb8b128eeac4f9
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/12182
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>