LU-957 scrub: Proc interfaces and tests for OI scrub

1) Control/trace OI scrub running.

2) Verify whether the OI scrub basic functions works or not.

3) Test OI scrub performance.

For autotest:
Test-Parameters: testlist=sanity-scrub,scrub-performance

Signed-off-by: Fan Yong <yong.fan@whamcloud.com>
Change-Id: I5be3d1a521f5f7875f56e9455ff2010016e6a344
Reviewed-on: http://review.whamcloud.com/3168
Reviewed-by: Yu Jian <yujian@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-957 lfsck: LFSCK main engine

Implement the main engine for Lustre online fsck. The kernel
thread "lfsck" scans the system object table by low layer DT
iteration APIs, and will drives registered scrub component(s)
to check/repair Lustre system.

It is the controller for the whole LFSCK, and controls the
speed, including the low layer OI scrub (for osd-ldiskfs).

For urgent mode, like MDT is restored from file-level backup
against ldiskfs, the OI files are invalided. Under such case,
we need to rebuild the OI files ASAP, then low layer OI scrub
inside osd-ldiskfs ignores the main engine speed control, and
runs with full speed.

Signed-off-by: Fan Yong <yong.fan@whamcloud.com>
Change-Id: If1bd3cfac1f299e964c029e5e9c4cce6432edfa5
Reviewed-on: http://review.whamcloud.com/3169
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-169 lov: add lsm refcounting

This patch adds a reference counter to the lsm structure.
Each time a lsm is used, a reference must be taken with lsm_get/addref
(lsm_addref() can be used when we already own a reference on the lsm).
This reference can be released via obd_free_memmd(). The lsm is freed
when the last reference is dropped.

This patch also moves lov_stripe_md into a private data of
lov_object.

Signed-off-by: Jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Signed-off-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Change-Id: I156b4cc2dc82bb15ae8107cf9842f100048c03d4
Reviewed-on: http://review.whamcloud.com/1874
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-56 ksocklnd: CPT affinity socklnd

this patch covered a few things:
- implement percpt scheduler threads for socklnd
- decrease overall threads number for fat core machine
- create more threads only if there are more than one NIC
- remove IRQ affinity implementation from socklnd
  IRQ affinity is not very helpful because CPUs on modern computer
  are very powerful. Also, user can still setup IRQ affinity via
  /proc and cpu_pattern of libcfs

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: Idfa19037a529fe96cb1432cbd7f55a5dfac89d29
Reviewed-on: http://review.whamcloud.com/2718
Reviewed-by: Lai Siyao <laisiyao@whamcloud.com>
Reviewed-by: Doug Oucharek <doug@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1588 lnet: deadlock while shutting down router

Should release lock on exiting of lnet_prune_rc_data(), otherwise
we will get deadlock on later attempting on lnet_net_lock().
Also, there is a wrong condition check in lnet_prune_rc_data()
can prevent router checker from shutting down.

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: I5292075453e61f300384043e2346df714c530303
Reviewed-on: http://review.whamcloud.com/3250
Tested-by: Hudson
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-by: Doug Oucharek <doug@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1589 lnet: read peers of /proc could be endless

There is a chance that reading /proc/sys/lnet/peers could be endless
because we didn't set correct condition for ending. This bug is
introduced by commit a07e9d350b3e500c7be877f6dcf54380b86a9cbe of LU-56

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: I25f6dad4a926bb7c62a4b1b1d4c3a86c783e3f7a
Reviewed-on: http://review.whamcloud.com/3251
Tested-by: Hudson
Reviewed-by: Doug Oucharek <doug@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-56 lnet: re-finalize failed ACK or routed message

lnet_finalize should restart finalizing process for failed ACK
or failed forwarding, because message could be committed for
sending then failed before delivering to LND, i.e: ENOMEM,
in that case we can't just continue to call lnet_msg_decommit():

- The rule is message must decommit for sending first if
  the it's committed for both sending and receiving

- CPT for sending can be different with CPT for receiving,
  so we should return back to lnet_finalize() to make
  sure we are locking the correct partition.

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: I0b35434762225fcb0dccad7d23bcd63740484e0a
Reviewed-on: http://review.whamcloud.com/3252
Reviewed-by: Bobi Jam <bobijam@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Doug Oucharek <doug@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-56 o2iblnd: CPT affinity o2iblnd

this patch covered a few things:
- implement percpt scheduler threads for o2iblnd
- decrease overall threads number for fat core machine
- increase thread number only if there are more than one NIC

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: Ic4b72258f73baabed2e59746639e271cab4467fc
Reviewed-on: http://review.whamcloud.com/2725
Reviewed-by: Lai Siyao <laisiyao@whamcloud.com>
Reviewed-by: Doug Oucharek <doug@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1305 lprocfs: osd-ldiskfs to use common helpers

introduced with osd-zfs

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I504fa697f6502ddfbac1e0ebc524337a02a74328
Reviewed-on: http://review.whamcloud.com/3198
Tested-by: Hudson
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-1581 defs: zfs string

a string corresponding to ZFS type

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I491041e22680f051016e5f782843b88580bf841e
Reviewed-on: http://review.whamcloud.com/3214
Reviewed-by: Li Wei <liwei@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1581 defs: new param

identity_upcall param

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ie5ed6fea5de4f7dd6ed1f8e2a8333e158d0d147b
Reviewed-on: http://review.whamcloud.com/3215
Reviewed-by: Li Wei <liwei@whamcloud.com>
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-56 ptlrpc: partitioned ptlrpc service

Current ptlrpc service only have a per-service instance, all service
threads share locks and request queue of this instance, this causes
many perforamnce issues like heavy lock contention and data/threads
migration between CPUs/NUMA nodes.

This patch created per-partition data for ptlrpc service, each service
have locks/request-queues on each partition, also, service will have
CPT (CPU partition) affinity threads on each partition, threads
bond on a CPT will only access data on local partition, this feature
should decrease lock contention and data/thread migration and improve
server side performance.

Another change is we use cfs_hash to replace big array fo_iobuf_pool
in obdfilter, filter_iobuf can be found by from the hash by thread ID.
The reason we made this change is because we removed absolute limit
of ptlrpc threads number, which means we have no idea how big the
fo_iobuf_pool should be. Also, even we have obsolute limit of threads
number, it's still dangerous to use a pre-allocated array because
it's difficult to guarantee thread ID to be contiguous if we want
to shrink threads in the future.

Signed-off-by: Liang Zhen <liang@whamcloud.com>
Change-Id: I5f8dce7bcf389f9f076f5ce2d4685a03f910260b
Reviewed-on: http://review.whamcloud.com/3133
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: wangdi <di.wang@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>