LU-3160 clio: don't ignore layout on writeback

In some cases such as kernel writeback, we shouldn't ignore the
layout, otherwise, it could race with layout change undergoing.

Test-Parameters: envdefinitions=DURATION=7200  clientdistro=el6 serverdistro=el6 clientcount=4  osscount=2 mdscount=2 austeroptions=-R failover=true  useiscsi=true testlist=recovery-random-scale
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ib9d0aa581de90711c92db4c631c52f1950ad5b67
Reviewed-on: http://review.whamcloud.com/6154
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2154 osp: osd-zfs: remove 2.3.x debugging code

When LUSTRE_VERSION_CODE is increased beyond 2.3.90 the version
based debug code generates a compile warning and fails to build.
Originally added in commit 04e1d0cb95e1ad12 and 2acb75c36511aca9.

Since we have been running with these checks for months without
problem, and it will soon be inactivated anyway, we may as well
remove the code entirely instead of just fixing it.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I1c104ef388f112c619d4ff4b7f00d17383500c1e
Reviewed-on: http://review.whamcloud.com/6219
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3030 build: Update Copyright Script

Add pevious mass copyright update and cfs wrapper changes
to the list of excluded patches.

Many files have incorrect author date set, revert to
using the git committer date for the patch date.

--follow and --author were not working properly.
Screen for Author after the git log call.  Git log
call code came via Andreas Dilger.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I46cf74b48095b54ac58dc17b6b0f94c9cf23cb8a
Reviewed-on: http://review.whamcloud.com/6183
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
LU-3030 build: Update Master Copyrights pre 2.4 split

This is the output of the tool. Please note the tool
changed to using commit dates over author dates and
there are a few spots with erroneous 2011 copyrights
that have been removed by the tool.

Please see script update for further details.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I6600369df53c01c425f33f62d5d6c4f8f1b48498
Reviewed-on: http://review.whamcloud.com/5841
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-3239 ofd: refill env in ofd_get_info

Because ofd_get_info(KEY_FIEMAP) might be called from
ptlrpc_server_handle_req_in(see the stack below),
where env might not be initialized correctly(see LBUG below),
so it refill refill in ofd_get_info.

LutreError: 19182:0:(ofd_internal.h:518:ofd_info_init()) LBUG
Pid: 19182, comm: ll_ost_io00_001
Call Trace:
[<ffffffffa044e895>] libcfs_debug_dumpstack+0x55/0x80 [libcfs]
[<ffffffffa044ee97>] lbug_with_loc+0x47/0xb0 [libcfs]
[<ffffffffa0e03e62>] ofd_info_init+0x92/0x130 [ofd]
[<ffffffffa0e05835>] ofd_get_info+0x2e5/0xa90 [ofd]
[<ffffffff812805cd>] ? pointer+0x8d/0x830
[<ffffffffa029f7e5>] ? lprocfs_counter_add+0x125/0x182 [lvfs]
[<ffffffffa078528a>] nrs_orr_range_fill_physical+0x18a/0x540
[ptlrpc]
[<ffffffffa0762dd6>] ? __req_capsule_get+0x166/0x700 [ptlrpc]
[<ffffffffa073e630>] ? lustre_swab_ost_body+0x0/0x10 [ptlrpc]
[<ffffffffa07871d7>] nrs_orr_res_get+0x817/0xb80 [ptlrpc]
[<ffffffffa077d306>] nrs_resource_get+0x56/0x110 [ptlrpc]
[<ffffffffa077dccb>] nrs_resource_get_safe+0x8b/0x100 [ptlrpc]
[<ffffffffa0780248>] ptlrpc_nrs_req_initialize+0x38/0x90 [ptlrpc]
[<ffffffffa074cff0>] ptlrpc_main+0x1170/0x16f0 [ptlrpc]
[<ffffffffa074be80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff8100c0ca>] child_rip+0xa/0x20
[<ffffffffa074be80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffffa074be80>] ? ptlrpc_main+0x0/0x16f0 [ptlrpc]
[<ffffffff8100c0c0>] ? child_rip+0x0/0x20

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Iee4b68fe331a895c61e6ccd0a14d6c60f5f9215c
Reviewed-on: http://review.whamcloud.com/6204
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Nikitas Angelinas <nikitas_angelinas@xyratex.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2677 utils: fix swabbing of f_oid

In ll_recover_lost_found_objs it is using le64_to_cpu() to swab
the f_oid field, which is a 32-bit value.  Swab it correctly using
le32_to_cpu().

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I2679b49dc16351140dbd1170b17056489c3ebbe5
Reviewed-on: http://review.whamcloud.com/6159
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1199 build: Remove unused LB_LDISKFS_RELEASE macros

The products of the LB_LDISKFS_RELEASE macros are unused.  They
also make some bad assumtions that complicate building against
a future lustre-devel package.  So we remove them.

If Lustre needs to know ldiskfs's version in the future, we
should add explicit version defines to the new ldiskfs_config.h.in
header that is in the works (or possibly landed by now).

Change-Id: I40b96dc72f5076ba9a9dda3f41b31088fdfd4341
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-on: http://review.whamcloud.com/5881
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
LU-2950 lnet: Add a mechanism to configure routes from a file

Created a bash script lustre_rotues_config which takes in a file
with routes configured in the following format:
<network>: { gateway: <gateway>, [hop:<hop>], [priority: <prio>] }
The script shall parse the file and generate:
lctl --net <network> add_route <gateway> [hop [priority]]
for each route.
The script can be used to unconfigure routes as well by running it
as follows:

   lustre_routes_config --cleanup <file>

In this case it will remove all routes configures via lctl del_route.

Also added another script: lustre_routes_conversion, which will be
used to convert from legacy syntax for configuring routes to the
new syntax described above. The script can be run on a file which
contains the legacy syntax and will generate a new file with the
passed in name:

   lustre_routes_conversion <legacy file> <new file>

Added two man pages to describe the usages of these scripts

Added a test case in conf-sanity.sh to test the new scripts
lustre_routes_conversion and lustre_routes_config. The test case
takes in a sample routes file which has the old synatx, run the
lustre_routes_conversion script to convert to the new syntax, then
runs the lustre_routes_config script in --dry-run to ensure that
the script configures the routes

Signed-off-by: Amir Shehata <amir.shehata@intel.com>
Change-Id: I57f81176c2d926fedefa8ea3be34586aa1ac9d76
Reviewed-on: http://review.whamcloud.com/5757
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Doug Oucharek <doug.s.oucharek@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2826 tests: add useful text to error() calls

Many tests just call "error" without any arguments, but this does not
provide any information to Maloo about why the test fails.  This
in turn causes autovet to match the empty failure message to many
different and irrelevant bugs, making Maloo statistics inaccurate.

Change a number of error() calls to have some valid error text, and
print something more useful if error() is called without arguments.
Not all of the error() calls in sanity.sh and sanityn.sh have been
converted, but it is at least a good start at cleaning this up, and
catches the most frequent failure cases currently.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Signed-off-by: Liu Ying <emoly.liu@intel.com>
Change-Id: Ie148e3e26e1ecb888ac021a8df7cd995183ebbe5
Reviewed-on: http://review.whamcloud.com/5750
Tested-by: Hudson
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
LU-3023 build: fix fuzzy logic in get_root_path()

Thanks to the call to llapi_is_lustre_mnt(), we are sure that
mnt.mnt_fsname contains ":/".
And, if no other mountpoint is found (len ==0), there is no reason
to abort scanning, so the check and -EINVAL can just be removed.

Signed-off-by: Sebastien Buisson <sebastien.buisson@bull.net>
Change-Id: Id77c41fbf53782d657034f152f724a3bdc83408c
Reviewed-on: http://review.whamcloud.com/5832
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
LU-3172 ost: call lustre_msg_get_opc() once inside ost_handle()

lustre_msg_get_opc() is called a few times inside ost_handle();
there is no real benefit from this, and we can just make one call
when entering the function.

There may be other occurrences in the source where this applies,
apart from ost_handle(), but we can at least make this change here
for now, as this is a frequently-called function.

Signed-off-by: Nikitas Angelinas <nikitas_angelinas@xyratex.com>
Change-Id: I7a8badc30ca31cb6826463ae5390cef96dec345f
Xyratex-bug-id: MRP-698
Reviewed-by: Andrew Perepechko <andrew_perepechko@xyratex.com>
Reviewed-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Reviewed-on: http://review.whamcloud.com/6055
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3208 tests: Fix typo in replay-single/70b

Missing space in if statement, caused syntax error:

	replay-single.sh: line 1912 [: missing `]`

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I2543f9b8fe41153d6109a64d2619a8443e700d9d
Reviewed-on: http://review.whamcloud.com/6131
Reviewed-by: James Nunez <james.a.nunez@intel.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-2442 kernel: SLES11 performance fixes and updates

This patch completes the SLES11 SP2 kernel support. The first problem
solved is the metadata performance slow down due to the heavy dqptr_sem.
So we remove dqptr_sem (but kept in struct quota_info to keep kernel ABI
unchanged), and the functionality of this lock is implemented
by other locks:

  * i_dquot is protected by i_lock, however only this pointer, the
    content of this struct is by dq_data_lock.

  * Q_GETFMT is now protected with dqonoff_mutex instead of dqptr_sem.

The second are small optimizations block level tunable optimizations.
Added in support to simulate fail over for testing purposes. The bulk
of the changes are removal of obsolete SLES10 and syncing SLES11 SP1
kernel side support to 2.6.32.

Signed-off-by: James Simmons <uja.ornl@gmail.com>
Change-Id: I474bdd8cf8293d2918273afa074e442aafa28d4c
Reviewed-on: http://review.whamcloud.com/6168
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3179 fids: fix compilation error with gcc 4.7.2

initialize oi.oi.oi_id which gcc 4.7.2 is afraid of being
used later.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Iaeb6eac01340d80786463efe67ff74479017f074
Reviewed-on: http://review.whamcloud.com/6064
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Bob Glossman <bob.glossman@intel.com>
Reviewed-by: James Simmons <uja.ornl@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2677 utils: update e2fsprogs to 1.42.7.wc1

Update the required e2fsprogs version to 1.42.7.wc1 to handle
smaller on-disk LMA and filter_fid structure sizes.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Id5bcbe1cd1bc2c170e253dbcb0bd0beb1a46c36f
Reviewed-on: http://review.whamcloud.com/6233
Tested-by: Hudson
Reviewed-by: Yang Sheng <yang.sheng@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3245 mdd: Fixed OBF of the FS root

Added a check to prevent fid access to the filesystem root directory
from returning EINVAL.

Added a sanity check accordingly.

Signed-off-by: Henri Doreau <henri.doreau@cea.fr>
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Iffb8bdb71377b878d165e9e48049c6b2ebd14859
Reviewed-on: http://review.whamcloud.com/6209
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-3188 osc: shorten IO calling path

By using osc_io_unplug_aync() for osc_queue_sync_pages() to shorten
the IO calling path, to reduce the chance of stack overflow.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I06112d9fb7b069d68c7945641e65d6430d0bb380
Reviewed-on: http://review.whamcloud.com/6191
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Keith Mannthey <kemannthey@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2742 oi: handle IGIF lookup

For the MDT upgraded from 1.8 device, OI scrub will immobilize the
IGIF and insert the mapping for "IGIF <=> ino#" into the OI file.
There are three cases to be processed:

1) After the upgrading, all IGIFs have been processed and inserted
   into OI files. Then any IGIF object lookup should be processed
   as normal FID does, means via OI. If no entry in the OI file,
   then the IGIF object does not exist.

2) During the upgrading, some IGIFs may be already in OI files,
   others may be not yet. Under such case, lookup an IGIF in OI
   files may return -ENOENT, but it does not means the IGIF obj
   does not exist. Since the new immobilize IGIF is the same as
   the original one before backup/restore, OSD will generate
   local identifier from IGIF directly as old 2.x did.

3) The upgrading is paused, and backup/restore. Means upgrading
   from 1.8 may be partly processed, but some clients may hold
   some immobilized IGIFs, and use them to access related objects.
   Under such case, OSD does not know whether an given IGIF has
   been processed or to be processed, and it also cannot generate
   local ino#/gen# directly from the immobilized IGIF because of
   the backup/restore. Then force OSD to lookup the given IGIF
   in OI files, and if no entry, then ask the client to retry
   after upgrading completed. No better choice.

Test-Parameters: testlist=sanity-scrub

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I7fff7898e2fcdf6b075bb04817f3b825c9a911b0
Reviewed-on: http://review.whamcloud.com/6254
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-3267 utils: missing setting echo seq for getattr/setattr

It should set echo seq before do echo getattr/setattr, otherwise
echo_client will regard it as the object with MDT0 sequence.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I1ea6cf0c9ef1edc5a81ba6f50345916deea4f95c
Reviewed-on: http://review.whamcloud.com/6263
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
LU-2193 ofd: check object exists before orphan destroy

MDS replay object destroys after recovery could meet non-existing
objects, skip it before following futile actions.

Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ibf233a07fa73d3226fdde5e2c020e73f51428f74
Reviewed-on: http://review.whamcloud.com/6266
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>