LU-5042 ldlm: delay filling resource's LVB upon replay

This patch is an attempt to delay unnecessary filling+resend of
resource's LVB upon replay after Server reboot.
This should avoid recovery to take a very long time when
replaying a huge number of locks and due to all associated LVBs
beeing read from disk. Now resource's LVB is only read upon need
to be sent to a new Client.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I20bd20bce328953c46accb4b41dcba776f3608a6
Reviewed-on: http://review.whamcloud.com/10845
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5299 obdclass: avoid race during Server device start

Handle concurrent starts for same device (multiple mounts, ...).
But allows for separate nosvc and nomgs case.
Also add a specific test of concurent MDT/OST start with an
artificial delay to verify.

Signed-off-by: Bruno Faccini <bruno.faccini@intel.com>
Change-Id: I442819a5b865ed3e98477f9d2602efc4d09d7860
Reviewed-on: http://review.whamcloud.com/11139
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5417 lfs: fix comparison between signed and unsigned

Expression if (size != (ssize_t)size) is always false.
Therefore no bounds check errors detected.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: Ib8fc68b2ddbb50ad50d80aa391c9bed6308ea575
Reviewed-on: http://review.whamcloud.com/11376
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3963 ldlm: convert to linux list api

Move from the cfs_[h]list api to the native linux api for
all the code related to ldlm.

Change-Id: Ibedd3870c530318dc4cba6a27dfb7005e1961ece
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/10945
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5502 lod: add a high-level description of the LOD layer

Add a high-level description of the LOD layer to lustre/lod/lod_dev.c.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I903394e96a2792683351d741614e2d326e77879d
Reviewed-on: http://review.whamcloud.com/11551
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Richard Henwood <richard.henwood@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4976 osp: add comments for lwp_dev.c functions

Add introductory comment block for the lwp_dev.c file.
Add function comment blocks to all functions in the lwp_dev.c file.

Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Change-Id: I8a672982964a742d56b1d64e7b8cceb31aa64a48
Reviewed-on: http://review.whamcloud.com/10335
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ned Bass <bass6@llnl.gov>
Reviewed-by: Richard Henwood <richard.henwood@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5208 tests: inject failure on the proper OST

The old test scripts assumed that the file created by the MDS2 will
be striped to the OST2 and OST1, but such assumption is wrong. When
the OSTs reside on different OSS nodes, the sanity-lfsck test_18c
will get failure.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I755370e6a70384b5a70e046b1acc806e63817f6e
Reviewed-on: http://review.whamcloud.com/11275
Tested-by: Jenkins
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-4970 lfsck: flush async updating before exit

Otherwise, the test scripts may get some internal status by race.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I4af07cf91f6b6c77d5cab67fc0df21b27174ee4c
Reviewed-on: http://review.whamcloud.com/11276
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
LU-5075 test: keep LFSCK fail_loc until recovery completed

In sanity-lfsck test_8, the old script may over wrote the fail_loc
before the recovery completed, that will cause the paused LFSCK to
be restarted automatically, then the subsequent manual command for
starting the LFSCK will get -EALREADY. That is unexpected.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I8500ecdccf354e83ec70be8dea88943cafa47d81
Reviewed-on: http://review.whamcloud.com/11288
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5466 lfsck: typo in lfsck_del_target

To handle MDT or OST target differently.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I8ede89935ca89774261efa466f753f78a09a50c2
Reviewed-on: http://review.whamcloud.com/11407
Tested-by: Jenkins
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5195: hsm: delete HSM records not found on disk

After a MDS crash, it is possible the file containing an hsm record is
not present anymore. When the MDS restarts, we get traces like this:

(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000:
    failed to process HSM_ACTIONS llog (rc=-2)
(mdt_hsm_cdt_actions.c:104:cdt_llog_process())
    Skipped 600 previous similar messages
(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000:
    error opening log id 0x1c:1:0: rc = -2
(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages
(llog_cat.c:556:llog_cat_process_cb()) tas01-MDD0000:
    cannot find handle for llog 0x1c:1: -2
(llog_cat.c:556:llog_cat_process_cb()) Skipped 600 previous similar messages
(mdt_hsm_cdt_actions.c:104:cdt_llog_process()) tas01-MDT0000:
    failed to process HSM_ACTIONS llog (rc=-2)
(mdt_hsm_cdt_actions.c:104:cdt_llog_process())
    Skipped 600 previous similar messages
(llog_cat.c:192:llog_cat_id2handle()) tas01-MDD0000:
    error opening log id 0x1c:1:0: rc = -2
(llog_cat.c:192:llog_cat_id2handle()) Skipped 600 previous similar messages

No HSM operation can happen, and the only way to clean is to unmount
the MDT and delete hsm_actions.

If the record can't be found, let the MDS delete it instead.

Change-Id: I61f625d4c18750c8044909ff56d53042cf0b6d86
Signed-off-by: frank zago <fzago@cray.com>
Reviewed-on: http://review.whamcloud.com/11419
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Ryan Haasken <haasken@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2182 llapi: implementation of new llapi_layout API

Add a new set of llapi routines for interacting with file layouts that
hide the details of the wire-protocol data structures from the user.
Define an opaque struct llapi_layout to abstract the layout of a
lustre file and generic accessor functions to read and write it.  The
following documented functions are added the liblustreapi public
interface with accompanying man pages:

  llapi_layout_alloc - allocate a new layout
  llapi_layout_free - free memory allocated for a layout
  llapi_layout_file_create - create new file with given layout
  llapi_layout_file_open - open or create a file with given layout
  llapi_layout_get_by_path - get file layout given a path
  llapi_layout_get_by_fd - get file layout given a file descriptor
  llapi_layout_get_by_fid - get file layout given a Lustre FID
  llapi_layout_ost_index_get - get OST index associated with a stripe
  llapi_layout_ost_index_set - set OST index associated with a stripe
  llapi_layout_pattern_get - get RAID pattern of a layout
  llapi_layout_pattern_set - set RAID pattern of a layout
  llapi_layout_pool_name_get - get pool name of a layout
  llapi_layout_pool_name_set - set pool name of a layout
  llapi_layout_stripe_count_get - get stripe count of a layout
  llapi_layout_stripe_count_set - set stripe count of a layout
  llapi_layout_stripe_size_get - get stripe size of a layout
  llapi_layout_stripe_size_set - set stripe size of a layout

The layouts are read and written using fgetxattr() and fsetxattr()
instead of ioctl() to make it easier for architectures like Blue Gene
to function-ship the system calls.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I35fb51055b6438ef3090f43c28a4083a66eaa907
Reviewed-on: http://review.whamcloud.com/5302
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-4808 tests: fix check for sanity tests prerequisites

Check of prerequisites for tests that manipulate Lustre servers.

Signed-off-by: Dmitry Eremin <dmitry.eremin@intel.com>
Change-Id: I17d0c90356a62976e1b5478588925862b588d7a2
Reviewed-on: http://review.whamcloud.com/10295
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3963 libcfs: remove invalid cfs_hlist_for_each_* checks

For newer kernels cfs_hlist_for_each_entry* changed the
number of arguments so in this case lustre still needs
wrappers. The HPDD script to test decrepit wrappers
incorrectly complains about cfs_hlist_for_* functions
that are required. Added some more wrappers to deal with
kernel api change that were missing before.

Change-Id: Idda94c147f2e3e9c138ee50c242604494041c244
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/11005
Tested-by: Jenkins
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: frank zago <fzago@cray.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3963 libcfs: convert lod, mdt, and gss to linux list api

Move from the cfs_[h]list api to the native linux api for
the lod, mdt, and gss part of the ptlrpc layers.

Change-Id: Ieff231f3220a850521713a6f1c997b7e09130a4c
Signed-off-by: James Simmons <uja.ornl@gmail.com>
Reviewed-on: http://review.whamcloud.com/10387
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Tested-by: Jenkins
Reviewed-by: frank zago <fzago@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5339 tests: Un-duplicate test_24 and skip if MDS is 2.5.2 or older

Un-duplicate test_24 and disable the test if MDS version
is 2.5.2 or older.

Change-Id: I3564c44a21bf149baf09294d42edd1529179736a
Signed-off-by: Wei Liu <wei3.liu@intel.com>
Reviewed-on: http://review.whamcloud.com/11262
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5455 ptlrpc: fix magic return value of ptlrpc_init_portals

Previously, when running 'modprobe lustre', it hit the following
error message which is becaue of network initialisation failure:
modprobe: ERROR: could not insert 'lustre': Input/output error

However, error code is there, just let it return to caller,
after this patch, error message will be something like:
modprobe: ERROR: could not insert 'lustre': Network is down

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Change-Id: Ib0c893052b4cd4f2e9b427fb4ce91da356065351
Reviewed-on: http://review.whamcloud.com/11337
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Dmitry Eremin <dmitry.eremin@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-5312 ptlrpc: update export_last_request_time after recovery

Update export_last_request_time after recovery, because client can
not send ping to the server during the recovery, so we need refresh
the last_request_time to avoid the export is being evicted earlier
than it should be.

Change-Id: I1d9bf5aaa3ca4215479ddc942c7052ad047c6989
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/11443
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>