LU-2748 osd: allocate buffers on demand

instead of putting a lot of buffers statically within osd_thread_info,
we can allocate them on the first demand within this thread.
we also can allocate not the maximum, but some optimal amount and
reallocate if really needed. dr_created is not used, so removed.
the number of blocks is calculated using actual blocksize, not the
smallest one, so no need to multiply by 8 in 99.9% cases.

with PTLRPC_MAX_BRW_PAGES=1024 (as default in master branch) and
regular 1MB IO,
before: sizeof(struct osd_thread_info) = 82104
after:  sizeof(struct osd_thread_info) = 4328 + 4K (if IO thread)

should improve threads not doing IO: all MDS threads, LDLM threads,
MGS threads.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Ie07780537a4598c6a888ed9be4ef0bbb0d9b3d54
Reviewed-on: http://review.whamcloud.com/5444
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-by: Mike Pershin <mike.pershin@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2449 osd: osd-zfs to initialize parent attribute

to follow ZFS on-disk format osd_object_create() should
initialize regular attribute storing parent dnode.
parent dnode is taken from object passed to create as an
allocation hint.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I1b4b4b5d7c6989c39f6bdecd52af48f270ad5beb
Reviewed-on: http://review.whamcloud.com/5642
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1468 o2iblnd: Support OFED-3.5 for o2ib

OFED is having new structure based on linux kernel code + backports
and packaging. Here is detailed information.
http://lists.openfabrics.org/pipermail/ewg/2011-December/017156.html

This patches are the lustre build improvements to support OFED 3.5,
3.x whatever future OFED release.

Signed-off-by: Shuichi Ihara <sihara@ddn.com>
Change-Id: Id4ffc39bc7fc24cc591bf6fb47e9b0e662993bda
Reviewed-on: http://review.whamcloud.com/3011
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Brian J. Murrell <brian.murrell@intel.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
Reviewed-by: Isaac Huang <he.huang@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2523 mdt: handle -ENOENT && !MDS_OPEN_CREAT in reint open

If mdt_open_by_fid_lock() returns -ENOENT and MDS_OPEN_CREAT is not
set in create flags then bail out and return -ENOENT from
mdt_reint_open().  In mdt_open_by_fid_lock() if -ENOENT is returned
then ensure that DISP_IT_EXECD is set in the reply disposition.  Add
sanity test_27B to call the LL_IOC_LOV_SETSTRIPE ioctl() on an open
unlinked file, a situation which triggers the first case.  In sanityn
test 30, remote the requirement that opening an unlinked file via
/proc/PID/exe return -ESTALE.  In racer's file_create.sh, reinstate
the call to 'lfs setstripe' before dd is started, effectively
reverting commit 0be1c87fe2d4ffddaca9e568cc137518b7368b2d.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I74e2297c4210ff18acbf60efdf51049d9a88cbea
Reviewed-on: http://review.whamcloud.com/5417
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Hudson
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2190 utils: Use LDD_PREFIX in LDD_*_PROP macros

This patch replaces the hard coded "lustre:" string used in the
LDD_*_PROP macro definitions with the LDD_PREFIX macro. This is
functionally equivalent, but eases the burden if the prefix string
is ever changed (and subjectively makes the code easier to read).

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ief0e710b90cf922eb4d4cc8a162ee1b0d21317b4
Reviewed-on: http://review.whamcloud.com/5671
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
LU-2936 ptlrpc: Do not try to fetch hp request blindly

ptlrpc_svcpt_health_check tries to blindly fetch a hp requet from
any service it happens to be called on, but some services don't have
any hp policies registered resulting in an underlyign assertion
in nrs_svcpt2nrs.
Make sure there are in fact pending hp requests on a service before
attempting to fetch them.

Change-Id: Ia38dcce758db948a1e4c187d009da4a8d5f2cbc6
Signed-off-by: Oleg Drokin <oleg.drokin@intel.com>
Reviewed-on: http://review.whamcloud.com/5665
Tested-by: Hudson
Reviewed-by: Nikitas Angelinas <nikitas_angelinas@xyratex.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Liang Zhen <liang.zhen@intel.com>
LU-2713 hsm: limit HSM RPC count from client

Put HSM RPCs under the control of the max_rpcs_in_flight param which
limits the number of concurrent RPCs in flight between a single client
and an MDT.

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I760333c7391ffd5aafca396b5fd97ef139799076
Reviewed-on: http://review.whamcloud.com/5616
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2942 ptlrpc: Fix an unswabbed status check in after_reply()

The -EINPROGRESS handling in after_reply() checks reply status while
pb_status still contains unswabbed data.  This patch moves the block
below the unpack_reply() call.

Change-Id: I619b26d213708b8f5c250cdf085f359bba31ffae
Signed-off-by: Li Wei <wei.g.li@intel.com>
Reviewed-on: http://review.whamcloud.com/5667
Tested-by: Hudson
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2791 ldlm: release reference against failed lock

On client-side, when ldlm_cli_enqueue_fini() gets reply from
server, which contains unexpected LVB size, it will mark the
lock as failure, but it does not release one reference, then
the failed lock prevents the lock from being freed. And then
umount client will be blocked.

The ldlm_lvbo_fill() caller should handle failure cases.

Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I197759a0b964e028627ecb6025820db9517fad7e
Reviewed-on: http://review.whamcloud.com/5634
Reviewed-by: Niu Yawei <yawei.niu@intel.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2918 tests: sanity 184c to start copy before layout swap

The test script has to make sure dd already starts to copy file
before swapping layout, otherwise the attempt to open file1 would
fail.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: I8217c9a38f1d09830c0ab259f65c9716c06736d1
Reviewed-on: http://review.whamcloud.com/5617
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2775 fid: allow FID-on-OST in fid_seq_is_mdt()

The LASSERT_SEQ_IS_MDT() macro used fid_seq_is_mdt() in several
places to verify that a FID was "sane" for where it was being used.
However, there should never be LASSERTs for data from the network
or disk.

The use of LASSERT_SEQ_IS_MDT() is removed from clients, since this
is "validating" data from the network.  This macro is no longer used,
so remove it.  The old CMD objseq_to_mdsno() and mdt_to_obd_objseq()
helpers are also no longer needed for DNE, and can be removed.

Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: I74bc9198799045b8bd91510cb45e8f876012cab0
Reviewed-on: http://review.whamcloud.com/5456
Tested-by: Hudson
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
Reviewed-by: wangdi <di.wang@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2816 llite: Set RAS_INCREASE_STEP correctly

RAS_INCREASE_STEP is by pages instead of by bytes.

However, I found it caused performance loss by setting it to be 4MB so
I set it back to 1MB. After 4MB RPC is enabled by default, more work
should be done to pick up a right value of max_read_ahead_mb to
maximize performance gain of 4MB RPC.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ic000f266cfefc827e112e03f76cf467c73ba88ad
Reviewed-on: http://review.whamcloud.com/5691
Tested-by: Hudson
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2809 llite: Do not return layout_gen for getxattr

The problem is that layout_gen and stripe_offset are sharing the
same field in lov_user_md{}. Layout gen would be wrongly interpreted
as stripe_offset when a backup is restored.

Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: If2c120bf861eeffd2db5f92e1d23cb1b9a2f5c63
Reviewed-on: http://review.whamcloud.com/5664
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: John Hammond <johnlockwoodhammond@gmail.com>
LU-2926 ldiskfs: crash in is_bad_inode()

Fix error handling in ldiskfs_xattr_inode_iget

Xyratex-bug-id: MRP-883
Signed-off-by: Andriy Skulysh <Andriy_Skulysh@xyratex.com>
Change-Id: I9840b7bd32f2c96763cae402a7bcb51d5798ea6c
Reviewed-on: http://review.whamcloud.com/5631
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2829 tests: sanityn/33a cleanup messages for zfs

Remove ldiskfs specific checks from running on zfs.
Account for lvm disks for ldiskfs.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I15cdb33d095c465b117761d40d61579eb3fbf52a
Reviewed-on: http://review.whamcloud.com/5693
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>