LU-4627 llite: deed taking lli_trunc_sem during file write

File write before io loop will take lli_trun_sem read semaphore to
protect osc_extent, while after generic_file_aio_write() done, it
could possible need to kill suid or sgid, which will call
ll_setattr_raw() to change the inode's attribute, and it does not
involve size.

So the ll_truc_sem write semaphore should be constrained
around ll_setattr_ost() to not come across the lli_trunc_sem read
semaphore get from the normal file write path.

Backported from master:
Lustre-change: http://review.whamcloud.com/9267
Lustre-commit: 1601e8591c1334ee7879bb6daac09342caa4a542

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ic7f9cfb6d76dcf7ad08635efab154e7cdc392ce8
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/12268
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
LU-4810 utils: print messages when set tunables

When set scheduler and max_sectors_kb on mount, error messages should
be printed to notify user that these tunables are changed by lustre.

Lustre-commit: 9813961151e8f6e19eac9cf5a8db676f7aa70f50
Lustre-change: http://review.whamcloud.com/9865

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I6a3618b7e5eb0127e9aea18631397cd4dcbde546
Reviewed-on: http://review.whamcloud.com/9865
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Blake Caldwell <blakec@ornl.gov>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/12553
Reviewed-by: James Nunez <james.a.nunez@intel.com>
LU-5545 ptlrpc: false alarm in AT network latency measuring

If early reply of client RPC is lost and client RPC is expired and
resent, server will drop the resent RPC because it's already in
processing, server may also send reply or early reply to client,
which can still match reply buffer of the original request.
In this case, client is measuring time from resent time, but server
is reporting service time of original RPC, which is longer than
the time measured by client.

This patch is back-ported from the following one:
Lustre-commit: c238be0bd02367b634c1061a30bbe161dfd9d8e9
Lustre-change: http://review.whamcloud.com/12018

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I5bdcdee9dd6b439d5d650fa812ea65527458c590
Reviewed-on: http://review.whamcloud.com/12855
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
LU-5650 mgc: check the import stat for lprocfs

in lprocfs_mgc_rd_ir_state, the import state should be checked
the validity before doing further work.

This patch is back-ported from the following one:
Lustre-commit: 4295402585605e1ba210ad96c12e3820fa523f7b
Lustre-change: http://review.whamcloud.com/12327

Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Change-Id: I41a5f7547d24bf61ce83dd488aef699409be209b
Reviewed-on: http://review.whamcloud.com/12896
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5822 osd-ldiskfs: Correctly return OS_STATE_READONLY

Ensure that read-only devices correctly return their state so that
the health_check file will be accurate

Signed-off-by: Matt Ezell <ezellma@ornl.gov>
Change-Id: Iea38c924ca0d8cb543b77599019fc88fd8b58105
Reviewed-on: http://review.whamcloud.com/12463
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-4839 tests: Give copytool more time to start

Copytool can take some time to start, and if the HSM archive directory
is on a busy NFS server, it can take a bit of time for the initial
opens to occur.  This allows those actions more time to complete which
should give this test a better chance of passing correctly.

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes \
mdtfilesystemtype=zfs mdsfilesystemtype=zfs ostfilesystemtype=zfs \
testlist=sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes,ONLY=60 \
mdtfilesystemtype=ldiskfs mdsfilesystemtype=ldiskfs ostfilesystemtype=ldiskfs \
mdtcount=4 testlist=sanity-hsm,sanity-hsm,sanity-hsm,sanity-hsm

This patch is back-ported from the following one:
Lustre-commit: 6948e80bc149aa689e09334a70941340143fa2ce
Lustre-change: http://review.whamcloud.com/12682

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ib8828805f52c1f86bf01388606b78ad2c4085acb
Reviewed-on: http://review.whamcloud.com/12823
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4441 test: improve run-llog.sh to print more information

Improve run-llog.sh script so that it can print more useful
information when "Module llog_test is in use" happens again.

This is a backport to b2_5.
Lustre-change: http://review.whamcloud.com/9966
Lustre-commit: 5d8de1fabf740eda724ad2e9eb40c3b57322b9e4

Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I15594f34f26550204b525d8f532fd064f30d00a4
Reviewed-on: http://review.whamcloud.com/11573
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-4943 obdclass: detach MGC dev on error

lustre_start_mgc() creates MGC device, if error happens later on
ll_fill_super(), this device is still attached, and later mount
fails by keep complaining that the MGC device's already in the
client node.

It turns out that the device was referenced by mgc config llog data
which is arranged in the mgc lock requeue thread re-trying to get its
mgc lock, and in normal case, this llog reference only released in
mgc_blocking_ast() when the system is umount.

This patch make mgc_precleanup() to wake up requeue thread to handle
the config llog data.

This patch also makes mgc_setup() wait for mgc_requeue_thread() start
before moving on.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I83df8c68c1dbe4ef4ee879e04ab20df46fea9062
Reviewed-on: http://review.whamcloud.com/11765
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5142 tests: add version check codes to conf-sanity test 43

This patch adds Lustre version check codes into conf-sanity test
43 to make the test interoperate with servers that do not have
the following changes:

Lustre-commit: 3064318df90ce7019b83c8a7f80e93a9634c00fa
Lustre-change: http://review.whamcloud.com/5700
Lustre-commit: 917b0a17d4e50db9f966763925a2dc860d932a3b
Lustre-change: http://review.whamcloud.com/9221

Test-Parameters: alwaysuploadlogs envdefinitions=ONLY=43 \
ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73 \
testlist=conf-sanity

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I7e12038cc8f77d3a9dbe19806133c21580bfd484
Reviewed-on: http://review.whamcloud.com/12911
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5707 lfsck: recreate LFSCK trace file for load failure

The extended attribute XATTR_NAME_LFSCK_NAMESPACE for namespace
LFSCK trace file is enlarged since Lustre-2.7. When downgrade
from Lustre-2.7, the old MDS will get -ERANGE when loads the
XATTR_NAME_LFSCK_NAMESPACE extended attribute from the new
formatted on-disk namespace LFSCK trace file. Under such case,
the LFSCK should not return failure, instead, it can recreate the
XATTR_NAME_LFSCK_NAMESPACE EA as initialization case.

More further, if the LFSCK failed to load trace file, it is
unncessary to fail the whole LFSCK, instead, just recreate the
trace file.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Icc2c7087f9f4a20d57664eaaf5a92f6c6260c3aa
Reviewed-on: http://review.whamcloud.com/12278
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
LU-5727 ldlm: revert the changes for lock canceling policy

The changes for LRU lock policy was introduced by commit 0a6c6fcd4,
where I was trying to revise the policy to pick locks for canceling.

However, this caused two problems as mentioned in LU-5727. The first
problem is that the lock can only be picked for canceling only if
the number of LRU locks is over preset LRU number AND it's aged; the
second problem is that mdc_cancel_weight() tends to not cancel OPEN
locks, therefore open locks can be kept forever and finally exhausts
memory on the MDT side.

This patch will revert the changes related to LRU policy revision.

This is backported from master:
Lustre-change: http://review.whamcloud.com/12733

LU-5727 ldlm: revert changes to ldlm_cancel_aged_policy()

The changes to ldlm_cancel_aged_policy() intrdouced from LU-4300
(bfae5a) was incorrect. This patch revert this part of changes.

This is backported from master:
Lustre-change: http://review.whamcloud.com/12448
Lustre-commit: e8812867e8c6c9eb62174b370cb818985e3d2180

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ic04f894c8bf54dbbdfb0e3bada5f6e4a367711a5
Reviewed-on: http://review.whamcloud.com/12565
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5778 lod: Fix lod_qos_statfs_update()

When an OST is sick, or unactivate, lod cannot fetch its statfs
information. In lod_qos_statfs_update() this was preventing lod
to get information for other OST because refresh was stopped at
first error.
This patch fixes this behaviour.

Patch backported from master:
Lustre-change: http://review.whamcloud.com/12617
Lustre-commit: b50a6ccf9fe32c2fae1e6225f8e1a5b420b45d56

Signed-off-by: Aurelien Degremont <aurelien.degremont@cea.fr>
Change-Id: I26ea91e54275f9e1accf51b779edf1b6e6a796a0
Reviewed-on: http://review.whamcloud.com/12685
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5781 osc: osc_lock_weight endless loop fix

With huge number of pages to scan by osc_lock_weight() it is likely
CLP_GANG_RESCHED is returned from osc_page_gang_lookup() and the scan
will be repeated again from the start. To be sure that the scan is
progressing across those restarts, next scan should be started from
the last scanned page index plus one.

Xyratex-bug-id: MRP-2145
Change-Id: I60775f2d8f688029d97ad4fe64f9b2698ed278c8
Signed-off-by: Alexander Zarochentsev <alexander.zarochentsev@seagate.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-on: http://review.whamcloud.com/12859
Tested-by: Jenkins
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5807 qos: enable QOS_DEBUG()

Enable QOS_DEBUG() by default.

Lustre-commit: 24f49b88cfefcd748902a6056dfd8ce20bdd6b7f
Lustre-change: http://review.whamcloud.com/12434

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Bob Glossman <bob.glossman@intel.com>
Change-Id: If2b0e1388a5f12edd5b03ffc1081709b9efb1c13
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: http://review.whamcloud.com/12651
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5827 mdt: do not crash server upon unsupported opc

mdt_handler_find() needs to check opc as well as whether we've defined
its handler to decide that the operation is supported.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Iefc2629cd4ad2d9df1b19ada7aa620282c5dab19
Reviewed-on: http://review.whamcloud.com/12497
Tested-by: Jenkins
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5248 osd: NOT inject OBD_FAIL_FID_LOOKUP on dotdot

The namespace LFSCK may cannot rebuild the FID-in-dirent for dotdot
entry after the MDT restored from file-level backup, because there
is not enough space in the directory head area.

Lustre-change: http://review.whamcloud.com/9748
Lustre-commit: 7dec22d474faa50e1a7fb6b377ceb1310b76a8e8

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iedf3b140a1dcc9fd77b40dbdf3dee52a44b5814b
Reviewed-on: http://review.whamcloud.com/11006
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5241 tests: speed up sanity-lfsck and sanity-scrub tests

1) drop unnecessary devices reformat.
2) drop unnecessary system stop/re-start.
3) replace 'sleep' with wait_update_facet to avoid idle wait.
4) drop unnecessary "-p" option for some "mkdir" cases.
5) replace "touch" with "createmany -m".
6) other code style changes and cleanup.

This patch is back-ported from the following one:
Lustre-commit: 1dbba329174e6c7f7712f01fc4e44c44400fbc92
Lustre-change: http://review.whamcloud.com/9704

LU-4803 ofd: skip orphan cleanup when inject OI error

When inject OI error for simulating some failure cases, the object
may be mapped to invalid local target (inode for ldiskfs), so skip
the orphan cleanup in OSP to avoid unexpected warning or destroy.

Lustre-commit: 0bafbd7d8f652997d83b3cc2419894f48833f424
Lustre-change: http://review.whamcloud.com/9759

Test-Parameters: alwaysuploadlogs envdefinitions=SLOW=yes \
testlist=lfsck-performance,sanity-scrub,sanity-lfsck

Signed-off-by: Fan Yong <fan.yong@intel.com>
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: Iacae0047a502e3baa778b86536220d70c717f6f5
Reviewed-on: http://review.whamcloud.com/10818
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4604 obdclass: handle ldt_device_nr/ldt_linkage properly

It is partly back-ported the patch from master:
"LU-4604 lfsck: LFSCK async updates RPC flow control"
But this only contains the part related with lu_device fixes,
not the lfsck async RPC changes.

There was no protection when inc/dec lu_device_type::ldt_device_nr,
which may caused the ldt_device_nr to be wrong and trigger assert.
This patch redefine lu_device_type::ldt_device_nr as atomic type.

There was no protection when add/del lu_device_type::ldt_linkage
into/from the global lu_device_types list, which may caused bad
address accessing. This patch uses the existing obd_types_lock
to protect related operations.

We do NOT need lu_types_stop() any longer. Such function scans
the global lu_device_types list, and for each type item on it
which has zerod lu_device_type::ldt_device_nr, call its stop()
method. In fact, the lu_device_type::ldt_device_nr only will be
zero when the last lu_device_fini() is called, and at that time,
inside the lu_device_fini(), its stop() method will be called.
So it is unnecessary to call the stop() again via lu_types_stop().

Lustre-commit: caef708d4040fd13499a11a42507ba56f9454298
Lustre-change: http://review.whamcloud.com/8694

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ibea9e8f51191de76dbd2e199e01b1d2adff35c75
Reviewed-on: http://review.whamcloud.com/12531
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>