LU-1866 misc: fix some issues found during LFSCK

1) Test scripts issues for sanity.sh and sanity-scrub.sh

2) Incorrectly use open flag 'MDS_OPEN_OWNEROVERRIDE" for
   permission check for set_attr.

3) Other code clean, such as mdt_body, md_op_spec, wiretest.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I52fbfe8481e35edba75e8fe7e0ab2fec094eabae
Reviewed-on: http://review.whamcloud.com/5046
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
LU-1866 fid: cleanup object visibility definition and check

1) The normal fid, IGIF fid, "ROOT" fid, ".lustre" fid are
   used for client-mdt visible objects.

2) Remove redundan osd_igif.{c,h}

3) Hide server-side local root for up layer dir readpage.

4) It is unnecessary to append FID-in-dient for dot name entry.
   Because we can get it from the directory object directly.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If98de0c15c0cd3a0e93b666a8d48366b0d8a2cb0
Reviewed-on: http://review.whamcloud.com/4901
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1431 ptlrpc: PTLRPC_BRW_MAX_SIZE usage cleanup.

Clean-up the layering in current code by eliminating direct
usage of PTLRPC_BRW_MAX_SIZE macro outside of the ptlrpc
module. This should help us acheive "floating" max brw size
value across the cluster, which in turn should help with
4MB IO task.

Signed-off-by: Sergii Glushchenko <sergii_glushchenko@xyratex.com>
Reviewed-by: Alexey Lyashkov <alexey_lyashkov@xyratex.com>
Reviewed-by: Alexander Zarochentsev <Alexander_Zarochentsev@xyratex.com>
Change-Id: Ib9f6265bd28f25c34b27eb6ad1a56a752b71a4ac
Xyratex-bug-id: MRP-687
Reviewed-on: http://review.whamcloud.com/4876
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2725 ost: ost_rw_hpreq_check should return 0 or 1

ost_rw_hpreq_check was returning number of matched locks which is wrong.

Change-Id: Ic4d3dd4a186b616c13ffd9f897254da3c19f84f8
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5233
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1866 osd: ancillary work for initial OI scrub

1) Abstract some general interfaces that can be shared by common
   OSD operations, by normal OI scrub, and by initial OI scrub.

2) Mark the object as "LDISKFS_STATE_LUSTRE_NO_OI" if it is not
   unnecessary to add OI mapping for the object, such as backend
   local root, OI directories/files, and others created before OI
   files created.

3) Do not check fld in OI scurb, which may be not ready yet.

4) Reorganize OI scrub code to be more readable.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Ife8168df2454437ad18550b73d26f7372c21776f
Reviewed-on: http://review.whamcloud.com/4902
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1866 scrub: initial OI scrub

We will extend OI mapping usage (in OI directories/files) to
all the objects, except for the objects which are marked as
"LDISKFS_STATE_LUSTRE_NO_OI" or OST object.

The OI mappings for the server-side local files may become
invalid because of server-side file-level back/restore, or
server crash. Those files are necessary for server mount,
so related OI mappings must be checked and rebuilt before
they are really used. That is the initial OI scrub work.

Initial OI scrub is used for quickly checking and rebuilding
OI mappings for server-side local files during server mount.

Another change is about MDT_LAST_RECV_OID/OFD_LAST_RECV_OID.
Currently, we can just use the same OID for the "last_rcvd"
files on different targets. So merge the two OIDs as single
OID: LAST_RECV_OID, which also simplifies OI scrub work.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: Iddca4812d3c82ca43f1e852fd74475b11b6e6dea
Reviewed-on: http://review.whamcloud.com/4903
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1187 osd: add agent/local inode for remote directory

A local inode will be created for remote directory entry, so
to make e2fsck check pass.

The remote directory on the remote MDT will be put under the
agent directory.

Remove LA_TYPE checking during osd_inode_set, because when
initialize the remote object, it needs to get TYPE information
from remote attr get, where we need type information anyway,
(osp_md_attr_get ->out_attr_get->osd_attr_get | LA_TYPE).
So osd_attr_get will always return LA_TYPE valid information
here, then if it do attr_get/modify/attr_set might hit this
ASSERT, So during osd_attr_set, we can just ignore type.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I7f177fa8ae2ab450cde78a8c2da9a2770ec6b0cb
Reviewed-on: http://review.whamcloud.com/4931
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2733 osd: not return "-ENOENT" for zfs osd_object_init

For zfs backend, the osd_object_init() should not return
"-ENOENT" if osd_fid_lookup() cannot find the object,
which is normal for new created cases.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I76e5d802d0caf8ac3a9992278ec2f7c6c6e2cf07
Reviewed-on: http://review.whamcloud.com/5242
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1866 osd: FID-in-LMA and OI files

Generate FID-in-LMA not only for normal objects, but also for
IGIF objects, server-side local files, and other non-OST/LLOG
objects.

LAST_ID has FID-in-LMA but no related mappings in the OI file.

At the same time, OI mapping usage is also extended to above
objects. Only the objects marked as "LDISKFS_STATE_LUSTRE_NO_OI"
or OST objects have no OI mappings the OI files.

It is part of 1.8 IGIF objects upgrage handling. OI scrub will
automatically generate IGIF-in-LMA and add related OI mappings
in the OI files.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: If1e1434c238f2ab737e35433bfad1abbb00fcb00
Reviewed-on: http://review.whamcloud.com/4904
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-1866 lfsck: enhance otable-based iteration

1) Use ::load() method to unplug OSD layer iteration.

2) Use ::put() method to wakeup the LFSCK main engine
   if it is blocked at low layer for ::next() call.

Test-Parameters: envdefinitions=ENABLE_QUOTA=yes testlist=sanity-scrub
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I28e5689ad3a4ee96a3f17a64e8d6dca553a86d77
Reviewed-on: http://review.whamcloud.com/4906
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Tested-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-1187 utils: add lfs setdirstripe/getdirstripe

1.Add lfs setdirstripe/mkdir/getdirstripe to create
remote directory.

2.Add lfs rm_entry to just remove the entry of remote
directory.

Change-Id: Idaecfbb9dd97e3ae9fbc70e5b902a91a7b2ed18e
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4341
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
LU-1187 mdt: unlink remote directory

Send unlink req to the slave MDT, so both open/close
and unlink will send request to the slave MDT, where
the remote object resides. Then it would be able to
check orphan (unlink open file) locally.

Add lock_object api for enqueue remote object.

Change-Id: I7483b0f023e4e3de6597da58d3d9f3e96c0d53b7
Signed-off-by: Wang Di <di.wang@intel.com>
Reviewed-on: http://review.whamcloud.com/4339
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1187 mdt: add sanity check for rename and link

Add sanity check for rename/link, so remote rename/link
will return EXDEV.

Signed-off-by: wang di <di.wang@whamcloud.com>
Change-Id: I841c8ffaa74b162fcca5cd1661c383215350cccc
Reviewed-on: http://review.whamcloud.com/4348
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2654 ldlm: ldlm_resource_dump should be called with resource locked

Fix ldlm_resource_complain to lock resource before calling
ldlm_resource_dump

Change-Id: I9dfbe84bd63de3093c99e7379daeca30d02fc875
Signed-off-by: Oleg Drokin <green@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/5254
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1187 mdd: a few missing stuff in MD stack for DNE.

1. Assign the index operation for the directory create
in declare phase, which is needed for creating the
object in OUT.

2. Declare dotdot insertion for remote directory creation,
so the insert update can be packed into RPC and send to
the remote MDT (out) to be executed.

3. Add hint to the object declare phase, by which OSP can
get some info during update RPC packing, like parent fid.

4. Separate lu_object_exists into lu_object_remote and
lu_object_exists, so it can check whether remote object
exists as well.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: Icb359443d9982ee8567af933c5def42dc51a3a7a
Reviewed-on: http://review.whamcloud.com/4930
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
LU-1187 mdt: directory remote open fix

Directory remote open continue as normal open, i.e. return the
remote fid to client, and client will send the open request to
the right MDT.

Set correct it disposition in mdt_open_by_fid.

Signed-off-by: wang di <di.wang@intel.com>
Change-Id: I32d5dee9cd1eac47787e2d85a5db1e164d2846d3
Reviewed-on: http://review.whamcloud.com/4934
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
LU-2194 tests: add debugging to recovery-small/19

Add error messages to 19a and 19b

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: Ief35c063734654c5a6f3dab0d283c4b7a32706e8
Reviewed-on: http://review.whamcloud.com/5141
Reviewed-by: Li Wei <wei.g.li@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-2723 clio: a client hangs on osc_extent_wait in dio

If dio overlaps with mapped region, first overlapped pages are sent
by generic_file_direct_write->filemap_write_and_wait_range->writepages
After that the dio pages are sent by write_begin/write_end, writepages
However, write_end->osc_page_cache_add does not release extent because
the IO is not considered as synchronous. as the result, the following
writepages->osc_cache_writeback_range() does not send pages as the
extent is still ACTIVE and osc_cache_wait_range() hangs forever.

The patch make directIO to be considered as sync by lustre so that
extent was released at write_end.

Signed-off-by: Vitaly Fertman <vitaly_fertman@xyratex.com>
Xyratex-bug-id: MRP-839
Change-Id: I0c0e1599ee180f467826e4b6715c3cbfd0c6eb5c
Reviewed-on: http://review.whamcloud.com/5224
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2467 protocol: Add OBD_CONNECT_PINGLESS

Reserve a bit for OBD_CONNECT_PINGLESS, which indicates a client is
capable of suppressing keep-alive OBD_PINGs.  If granted by a server,
it means the server does not require (but still allows) pings.

Signed-off-by: Li Wei <wei.g.li@intel.com>
Change-Id: I7575082244a84f2e633a1936fcf894da3c6ee7dd
Reviewed-on: http://review.whamcloud.com/5229
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2467 ptlrpc: Allow OBD_PINGs to be suppressed

This patch introduces a new ptlrpc module parameter, "suppress_pings",
to provide an option for reducing excessive OBD_PINGs in large
clusters.  The parameter affects all MDTs and OSTs on a node.  It is
off (zero) by default, giving a behavior identical to current
implementation.  If it is on (non-zero), all clients of the affected
targets who understand OBD_CONNECT_PINGLESS will know, at connect
time, that pings are not required.  When suppressing pings, there must
be an external mechanism to notify the targets of client deaths, via
the targets' "evict_client" procfs entries.  In addition, a highly
available standalone MGS is also recommended when suppressing pings,
so that clients are notified (through Imperative Recovery) of target
recoveries.

The changes do two basically independent things.  One is initializing
import and export states (i.e., imp_connect_data and
exp_obd_chain_timed) according to "suppress_pings", since whether to
ping or not is a property of each import-export pair.  MGC pings can
not be suppressed, because maintaining MGS connections is dictated by
the reliance on Imperative Recovery.

The other thing is changing pinger and import routines to respect the
import property set earlier.  (The export side does not need any
change at all.)  Pings are still needed to query last committed
transactions if there are uncommitted requests on an import, so that
resources pinned for replays can be released even when applications
become idle.  An early version of this patch removes imports that do
not need to be pinged from pinger_imports and add them back when last
committed transactions are needed or recoveries must be initiated.
This version does not do that because a) the overheads of iterating
through 10,000 imports are not prohibitively large---around 10 ms, b)
adding imports back to pinger_imports requires the global pinger_mutex
to be held, and c) the imp_lock contention added on each import is
small.

Change-Id: Iabc84d395c978c3f156c52aebfad83621facb4fe
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5009
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Tested-by: Hudson
Reviewed-by: Emoly Liu <emoly.liu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>