LU-4181 tests: cleanup lustre before starting lnet-selftest.sh

Cleanup lustre before starting lnet-selftest.sh to attempt and
identify why RPC requests are timing out.

The approach is to land this patch and monitor the failure rate
for LU-4181 to see if it's been reduced.  The current failure rate
is approximately 1 failure per day.

Added a debug message to print out rpc stats when an RPC
error occurs.

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I86cbc00af30b165918a57e6665ad725a9926c089
Reviewed-on: http://review.whamcloud.com/12469
Tested-by: Jenkins
Reviewed-by: Isaac Huang <he.huang@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3259 clio: cl_lock simplification

In this patch, the cl_lock cache is eliminated. cl_lock is turned
into a cacheless data container for the requirements of locks to
complete the IO. cl_lock is created before I/O starts and destroyed
when the I/O is complete.

cl_lock depends on LDLM lock to fulfill lock semantics. LDLM lock
is attached to cl_lock at OSC layer. LDLM lock is still cacheable.

Two major methods are supported for cl_lock: clo_enqueue and
clo_cancel.  A cl_lock is enqueued by cl_lock_request(), which will
call clo_enqueue() methods for each layer to enqueue the lock.
At the LOV layer, if a cl_lock consists of multiple sub cl_locks,
each sub locks will be enqueued correspondingly. At OSC layer, the
lock enqueue request will tend to reuse cached LDLM lock; otherwise
a new LDLM lock will have to be requested from OST side.

cl_lock_cancel() must be called to release a cl_lock after use.
clo_cancel() method will be called for each layer to release the
resource held by this lock. At OSC layer, the reference count of LDLM
lock, which is held at clo_enqueue time, is released.

LDLM lock can only be canceled if there is no cl_lock using it.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I6a61250549cfbc28070fe4bb7789ba7429eaf089
Reviewed-on: http://review.whamcloud.com/10858
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4810 utils: print messages when set tunables

When set scheduler and max_sectors_kb on mount, error messages should
be printed to notify user that these tunables are changed by lustre.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I6a3618b7e5eb0127e9aea18631397cd4dcbde546
Reviewed-on: http://review.whamcloud.com/9865
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Blake Caldwell <blakec@ornl.gov>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5380 at: net AT after connect

Once connected, the previously gathered AT statistics is not valid
anymore because may reflect other routing, etc. The connect by itself
could take a long time due to different reasons (e.g. server was not
ready) and net latency got very high (see import_select_connection())
what does not reflect the current situation.

Take into account only the current (re-)CONNECT rpc latency.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Signed-off-by: Alexander Boyko <alexander_boyko@xyratex.com>
Xyratex-bug-id: MRP-1285
Change-Id: I6edc0e232a92319e7c8535aced28fe1ad3436c54
Reviewed-on: http://review.whamcloud.com/11155
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
LU-4856 obdclass: check val in proc_max_dirty_pages_in_mb()

In proc_max_dirty_pages_in_mb(), assigning "__u64 val" to
"unsigned long obd_max_dirty_pages" will cause values over
2^32 to be truncated on a 32-bit client (where "unsigned long"
is 32 bits, and not 64 bits).

This patch fixes the above issue by checking "val" in an
acceptable range before assigning it to obd_max_dirty_pages.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: I838b5bab283a0068f72a86f5a990d909c892a9d9
Reviewed-on: http://review.whamcloud.com/12269
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
LU-5807 qos: enable QOS_DEBUG()

Enable QOS_DEBUG() by default.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: If2b0e1388a5f12edd5b03ffc1081709b9efb1c13
Reviewed-on: http://review.whamcloud.com/12434
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-4176 tests: re-enable sanity-hsm/test_31a

Due to frequent failures and despite first patch/fix for LU-4176 has
landed, sanity-hsm/test_31a has been disabled like other sub-tests
in a patch for LU-4178. Since both test_31[b,c], which also triggered
the same issue, are running fine now and with original fix for
LU-4176, it is time to re-enable it.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I5cc6c23b7cb48e190438eb3b84fa55ffdb198739
Reviewed-on: http://review.whamcloud.com/9577
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
LU-5577 mdc: fix comparison between signed and unsigned

Change type of client_obd->*_mds_*size from int to __u32 and
argumanets of related create/rename/setattr functions.
Change type of op_data->op_namelen to size_t.
Change type of argument size for all mdc_*_pack() to size_t.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I97f4fa6928c24fa416c334206c75f9885266b1ae
Reviewed-on: http://review.whamcloud.com/11379
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4167 tests: correct version check to enable ff_convert

In conf-sanity.sh test_32d, filter fid convertion can be made only
when both of the following conditions are satisfied:
- ost device img version < 2.3.64 (struct filter_fid_old has been
introduced since 2.3.64)
- ost server version >= 2.5

Also, this patch fix the ofd_iocontrol() message to print the ioctl()
CMD argument as hex instead of a signed integer.

Test-Parameters: testlist=conf-sanity envdefinitions=SLOW=yes,ONLY=32 ossjob=lustre-b2_4 mdsjob=lustre-b2_4 ossbuildno=73 mdsbuildno=73  ossdistro=el6 mdsdistro=el6
Signed-off-by: Emoly Liu <emoly.liu@intel.com>
Change-Id: I4045fe6b7504a3ed30436da5a1319c09331fc261
Reviewed-on: http://review.whamcloud.com/11556
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
LU-2675 obd: cleanup struct md_op_data and uses

Make the following changes in or around struct md_op_data:

* Move the definition of enum op_cli_flags from lclient.h to obd.h and
  rename it to enum md_cli_flags.

* Change to type of the op_flags member from __u32 to enum
  md_op_flags.

* Remove the used but never set member op_npages.

* Remove the set but never used member op_stripe_offset (an alias for
  op_ioepoch).

* Remove the op_max_pages alias for op_valid and add a op_max_pages
  member.

* Add a new member op_attr_flags.

* Remove the definition and all uses of struct ll_iattr. This
  structure was only used in expressions of the form
  ((struct ll_iattr *)&op_data->op_attr)->ia_attr_flags which can all
  be rewritten as op_data->op_attr_flags.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I17aabfcecdfd1a02dbee04362b033ef404a2cb27
Reviewed-on: http://review.whamcloud.com/11734
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5589 obdclass: fix NULL pointer dereference

NULL pointer dereferenced without check in lsi_prepare()

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I144c9bcd3739c68563c7460799efa4897489a1a7
Reviewed-on: http://review.whamcloud.com/11769
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5591 lod: fix Null pointer dereference in lod_ah_init()

Null pointer may be dereferenced in lod_ah_init()

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I76567d222ac7eb74440c74692aaa79f7078bca61
Reviewed-on: http://review.whamcloud.com/11770
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5651: ptlrpc: fix import state during replay

Client doesn't restore import state correctly
on reconnect during replay. It resends lock replay
when final ping was queued by server.
Server fails with "target_queue_recovery_request())
ASSERTION( req->rq_export->exp_lock_replay_needed ) failed"

Add imp_replay_state to store last replay state.
imp_state is restored from imp_replay_state
during reconnect.

Xyratex-bug-id: MRP-2022
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Change-Id: Iaa14fe968cc31f266b605785df4fa676083fbca4
Reviewed-on: http://review.whamcloud.com/12015
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5545 ptlrpc: false alarm in AT network latency measuring

If early reply of client RPC is lost and client RPC is expired and
resent, server will drop the resent RPC because it's already in
processing, server may also send reply or early reply to client,
which can still match reply buffer of the original request.
In this case, client is measuring time from resent time, but server
is reporting service time of original RPC, which is longer than
the time measured by client.

Signed-off-by: Liang Zhen <liang.zhen@intel.com>
Change-Id: I772fb054480a3212e28dc018a6f592f3da7a87b5
Reviewed-on: http://review.whamcloud.com/12018
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
LU-5443 lustre: replace direct HZ access with kernel APIs

On some customer's systems, kernel was compiled with HZ defined to
100, instead of 1000. This improves performance for HPC applications.
However, to use these systems with Lustre, customers have to re-build
Lustre for the kernel because Lustre directly uses the defined
constant HZ.

Since kernel 2.6.21, some non-HZ dependent timing APIs become non-
inline functions, which can be used in Lustre codes to replace the
direct HZ access.

These kernel APIs include:
 jiffies_to_msecs()
 jiffies_to_usecs()
 jiffies_to_timespec()
 msecs_to_jiffies()
 usecs_to_jiffies()
 timespec_to_jiffies()

And here are some samples of the replacement:
 HZ            -> msecs_to_jiffies(MSEC_PER_SEC)
 n * HZ        -> msecs_to_jiffies(n * MSEC_PER_SEC)
 HZ / n        -> msecs_to_jiffies(MSEC_PER_SEC / n)
 n / HZ        -> jiffies_to_msecs(n) / MSEC_PER_SEC
 n / HZ * 1000 -> jiffies_to_msecs(n)

This patch replaces the direct HZ access in lustre modules.

Signed-off-by: Jian Yu <jian.yu@intel.com>
Change-Id: Ib0ed9b5faf6ed311ff5423873d1c125b02ec4ab5
Reviewed-on: http://review.whamcloud.com/12052
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Tested-by: Jenkins
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5668 test: enable ior data consistency check

Check file content (-W) is consistent during read phase of IOR
and also change task ordering to n+1 ordering for readback (-C).

Test-Parameters: testlist=parallel-scale

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: Ic9b1c3cda48ebd08907a6251f62cf2c845d00476
Reviewed-on: http://review.whamcloud.com/12058
Tested-by: Jenkins
Reviewed-by: Li Wei <wei.g.li@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5387 test: Skip sanity test_239 if MDS version older than 2.5.60

Skip sanity test_239 if MDS version older than 2.5.60

Change-Id: Ic2c71235a0f05ecb4a8b111a6044efe51c5270c8
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-on: http://review.whamcloud.com/12241
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
LU-5577 ldlm: count of pools is unsigned long

Function ldlm_pools_count() return unsigned long but counter is int.
Use ldlm_pool_granted() everywhere.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I6ee7bb1ba7f9590a776465b00e63584aada5f7dc
Reviewed-on: http://review.whamcloud.com/12304
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5577 mdd: lu_dirent_calc_size() return type to size_t

Change the type accordant usage.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I2451940225dd30015928cf85a2e0cc0e6cc8dfeb
Reviewed-on: http://review.whamcloud.com/12383
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5577 obdclass: change lu_site->ls_purge_start to unsigned

Change the type accordant usage.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ic2d6906eff21ab1fe964899f0da9732e68c193f7
Reviewed-on: http://review.whamcloud.com/12384
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>